Skip to Content

Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2026-01-19 14:35
From version 105.4
edited by sndueste
on 2026-01-09 15:22
Change comment: There is no comment for this version
To version 96.1
edited by sndueste
on 2025-05-19 17:06
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,3 +1,6 @@
1 +{{layout}}
2 +{{layout-section ac:type="single"}}
3 +{{layout-cell}}
1 1  == Contents ==
2 2  
3 3  
... ... @@ -5,24 +5,10 @@
5 5  {{toc/}}
6 6  
7 7  
8 -== General information about HDF5 and how to look on it ==
11 +== General information about HDF5 ==
9 9  
10 10  HDF5 is a data format maintained by the HDF group. For detailed general information see [[here>>url:https://portal.hdfgroup.org/display/support||shape="rect"]]
11 11  
12 -The easiest way to look at the HDF5 files from your beamtime is to use the Jupyter notebook interface to Maxwell:  [[https:~~/~~/max-jhub.desy.de/>>https://max-jhub.desy.de/]]
13 -
14 - The file content can be browsed by double clicking on the file.
15 -
16 -Typically the Jupyter notebook starts at your home directory and you have to create a (soft) link to your beamtime folder: open a terminal window (File->New-> Terminal) and create the link via :
17 -
18 -{{code language="none"}}> ln -sn beamtime path linkname
19 -e.g.
20 -> ln -sn /asap3/flash/gpfs/fl23/2025/data/11024321/ ~/beamtime_11024321 {{/code}}
21 -Then a “beamtime_11024321” folder should appear in your home directory.
22 -
23 -
24 -Alternativaly you can also use a conventional HDF viewer:
25 -
26 26  To download the HDF5 Viewer click [[here>>url:https://portal.hdfgroup.org/display/support/Download+HDFView||shape="rect"]]
27 27  
28 28  For use on the DESY Maxwell or PAL (max-fsc or pal) **hdfview** is available in the xray module :
... ... @@ -72,17 +72,25 @@
72 72  {{/expand}}
73 73  
74 74  === HDF examples: ===
64 +{{/layout-cell}}
65 +{{/layout-section}}
75 75  
67 +{{layout-section ac:type="three_equal"}}
68 +{{layout-cell}}
76 76  * ADC data as example for **fast** **data** (10 Hz):            
77 77  
78 78  [[image:attach:image2020-11-16_15-26-28.png||height="250"]]
72 +{{/layout-cell}}
79 79  
74 +{{layout-cell}}
80 80  * The //average// FEL pulse energy as example for the **slow** **data**. Here the data is not saved with 10 Hz  - thus not for every FEL pulse train.  The data is typically saved with 1 Hz if the values are changing (like the FEL energy) and every about 20 sec if there is no change for longer time ( e.g. motor set values). Thus to use the data together with the "fast" one, one has to interpolate the data as explained in the examples in the repos below show (e.g. {{code language="none"}}df['GMD_T_average'] = df['GMD_T_average'].interpolate(method='linear'){{/code}})
81 81  
82 82   ( as example only every 10th train ID is listed in the HDF group "index")
83 83  
84 84  [[image:attach:image2020-11-16_15-31-45.png||height="250"]]
80 +{{/layout-cell}}
85 85  
82 +{{layout-cell}}
86 86  * (((
87 87  //zraw// group contains the **original DAQ (DOOCS) names**
88 88  )))
... ... @@ -91,7 +91,11 @@
91 91  
92 92  
93 93  [[image:attach:image2020-11-16_16-26-3.png||height="400"]]
91 +{{/layout-cell}}
92 +{{/layout-section}}
94 94  
94 +{{layout-section ac:type="single"}}
95 +{{layout-cell}}
95 95  {{info}}
96 96  === There are different options that help you to work with the FLASH HDF5 data in Python ===
97 97  
... ... @@ -98,16 +98,20 @@
98 98  * The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
99 99  * and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
100 100  
101 -(% class="Object" %)See also the collection of Demo data and sample scripts : [[Collection of HDF5 sample data from different beamlines>>doc:FLASHUSER.Offline data analysis (DAQ).Collection of HDF5 sample data from different beamlines.WebHome]] and [[DEMO - Working with FLASH data>>doc:FLASHUSER.Offline data analysis (DAQ).DEMO - Working with FLASH data.WebHome]]
102 +(% class="Object" %)See also the collection of Demo data and sample scripts : [[doc:FS-FLASH USER tmp.Data Acquisition and controls 1.DAQ and controls overview.Offline data analysis (DAQ).Collection of HDF5 sample data from different beamlines.WebHome]] and [[doc:FS-FLASH USER tmp.Data Acquisition and controls 1.DAQ and controls overview.Offline data analysis (DAQ).DEMO - Working with FLASH data.WebHome]]
102 102  {{/info}}
104 +{{/layout-cell}}
105 +{{/layout-section}}
103 103  
107 +{{layout-section ac:type="single"}}
108 +{{layout-cell}}
109 +
104 104  
105 -
106 106  == Complete list of recordable parameters ==
107 107  
108 108  The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
109 109  
110 -== Most popular FLASH parameters and their names in HDF5, DOOCS and (.raw) DAQ ==
115 +== Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
111 111  
112 112  {{id name="DOOCSparameters"/}}
113 113  
... ... @@ -119,7 +119,7 @@
119 119  
120 120  The previously used naming scheme (2024 and before) can be found here:
121 121  
122 -{{expand expanded="false" title="FLASH1 naming scheme used until 2024"}}
127 +{{expand expanded="false" title="FLASH1 Naming scheme used until 2024"}}
123 123  ==== (% id="cke_bm_8853497S" style="display:none" %) (%%)Beamline info (FLASH1) ====
124 124  
125 125  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -746,6 +746,7 @@
746 746  ==== Beamline info (FLASH2) ====
747 747  
748 748  {{code language="none"}}/FL2/Beamlines/Attenuator/pressure  {{/code}}
754 +(% style="color:#000000" %)//always saved (PBD2)// (%%)
749 749  (% style="color:#000000" %)DOOCS prop : {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
750 750  (% style="color:#000000" %)DAQ channel:  {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
751 751  (% style="color:#000000" %)desc: set pressure in the gas attenuator (%%)
... ... @@ -752,6 +752,7 @@
752 752  (% style="color:#000000" %)units: mbar
753 753  
754 754  {{code language="none"}}/FL2/Beamlines/FL20/Shutter/open{{/code}}
761 +//always saved (PBD2)//
755 755  DOOCS prop : {{code language="none"}}FLASH.FEL/ADC.SIS.FL2FS/FL20.SHUTTER/CH00.TD{{/code}}
756 756  DAQ channel: {{code language="none"}}FLASH.FEL/ADC.SIS.FL2FS/FL20.SHUTTER{{/code}}
757 757  desc: BL Beamline Fast shutter state: 1 is open, 0 is closed ( for technical reasons there are 100 vales of this state saved ...)
... ... @@ -759,6 +759,7 @@
759 759  
760 760  
761 761  {{code language="none"}}/FL2/Beamlines/Filter wheel/position wheel 1{{/code}}
769 +//always saved (PBD2)//
762 762  DOOCS prop : {{code language="none"}}FLASH.FEL/FL20H.PH.MOTOR/MOTOR1.MOT3/FPOS{{/code}}
763 763  DAQ channel: FLASH.FEL/FL20H.PH.MOTOR/MOTOR1.MOT3/FPOS
764 764  desc: Position of the BL filter wheel 1 - to correlate with the filter material please look **[[here>>doc:FS-FLASH USER tmp.jddd-linked help pages.Filter-Units.Filter wheels in FLASH1 and FLASH2.WebHome]]**
... ... @@ -765,6 +765,7 @@
765 765  units: degree
766 766  
767 767  /FL2/Beamlines/Filter wheel/position wheel 2
776 +always saved (PBD2)
768 768  DOOCS prop : {{code language="none"}}FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS{{/code}}
769 769  DAQ channel: {{code language="none"}}FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS{{/code}}
770 770  desc: Position of the BL filter wheel 2 - to correlate with the filter material please look [[here>>doc:FS-FLASH USER tmp.jddd-linked help pages.Filter-Units.Filter wheels in FLASH1 and FLASH2.WebHome]]
... ... @@ -780,6 +780,7 @@
780 780  ==== Photon Diagnostics SASE ([[XGMD>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/gmd_intensity_and_position/index_eng.html||shape="rect"]] - FLASH2) ====
781 781  
782 782  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
792 +//always saved (PBD2)//
783 783  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ{{/code}}
784 784  DAQ channel:{{code language="none"}} FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ{{/code}}
785 785  desc : calibrated average ( ~~ 20 sec averaging time ) SASE Energy/pulse measured in the TUNNEL before the attenuator (ion current)
... ... @@ -787,6 +787,7 @@
787 787  
788 788  
789 789  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
800 +//always saved (PBD2)//
790 790  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
791 791  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
792 792  desc : Energy per pulse measured in the Tunnel  (in front of the gas attenuator and the apertures in the Hall) In addition measurement errors and beam position are included for EACH pulse in the pulse train  !! (see below)
... ... @@ -821,6 +821,7 @@
821 821  
822 822  
823 823  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Average Beam position/position tunnel horizontal{{/code}}
835 +//always saved (PBD2)//
824 824  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL2.TUNNEL/IX.POS{{/code}}
825 825  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.POSMON/FL2.TUNNEL/IX.POS{{/code}}
826 826  desc : the Average ( ~~ 20 sec averaging time ) Beam position of the photon Beam determined by the GMD (tunnel, x=horizontal)
... ... @@ -828,6 +828,7 @@
828 828  
829 829  
830 830  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Average Beam position/position tunnel vertical{{/code}}
843 +//always saved (PBD2)//
831 831  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL2.TUNNEL/IY.POS{{/code}}
832 832  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.POSMON/FL2.TUNNEL/IY.POS{{/code}}
833 833  desc : the Average ( ~~ 20 sec averaging time ) Beam position of the photon Beam determined by the GMD (tunnel, x=horizontal)
... ... @@ -835,6 +835,7 @@
835 835  
836 836  
837 837  /FL2/Photon Diagnostic/GMD/Pulse resolved beam position/position tunnel x
851 +always saved (PBD2)
838 838  DOOCS prop : FLASH.FEL/XGM.BPM/FL2.HALL/X.TD
839 839  DAQ channel: FLASH.FEL/XGM.BPM/FL2.HALL:2
840 840  desc: Besides the well calibrated averaged beam position information there is also the option to measure the beam position on a single bunch level. HOWEVER this methide needs a perfectly adjusted signal level (talk to your local contact !!) and also then the signal to noise is rather small and one needs some averaging ... BUT with this option one can determine if there was a spatial slope on a burst (say forst bunches were lower than the last ones or so ...)
... ... @@ -854,6 +854,7 @@
854 854  
855 855  
856 856  {{code language="none"}}/FL2/Photon Diagnostic/Wavelength/OPIS tunnel/Processed/mean photon energy{{/code}}
871 +// saved opon request (PBD2)//
857 857  DOOCS prop : {{code language="none"}}FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL042{{/code}}
858 858  DAQ channel:{{code language="none"}} FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL042{{/code}}
859 859  desc : mean photon energy ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
... ... @@ -861,6 +861,7 @@
861 861  
862 862  
863 863  {{code language="none"}}/FL2/Photon Diagnostic/Wavelength/OPIS tunnel/Processed/mean wavelength{{/code}}
879 +// saved opon request (PBD2)//
864 864  DOOCS prop : {{code language="none"}}FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
865 865  DAQ channel:{{code language="none"}} FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
866 866  desc : mean wavelength ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
... ... @@ -868,28 +868,12 @@
868 868  
869 869  
870 870  {{code language="none"}}/FL2/Photon Diagnostic/Wavelength/OPIS tunnel/Processed/number of analyzed bunch        (in older version this can be found in OPIS tunnel/Expert stuff/General operation parameters/){{/code}}
887 +// saved opon request (PBD2)//
871 871  DOOCS prop : {{code language="none"}}FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL060{{/code}}
872 872  DAQ channel:{{code language="none"}} FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL060{{/code}}
873 873  desc : The bunch number of the bunch used for the wavelength calculation
874 874  units :
875 875  
876 -{{expand expanded="false" title="More detailed info on OPIS properties"}}
877 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL370,'%','relative width of photoline in percent of the FEL photon energy ROI1, rising flank'
878 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL375,'%','relative width of photoline in percent of the FEL photon energy ROI1, tailing flank'
879 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL470,'%','relative width of photoline in percent of the FEL photon energy ROI1, mean'
880 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL371,'%','relative width of photoline in percent of the FEL photon energy ROI2, rising flank'
881 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL376,'%','relative width of photoline in percent of the FEL photon energy ROI2, tailing flank'
882 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL471,'%','relative width of photoline in percent of the FEL photon energy ROI2, mean'
883 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL372,'%','relative width of photoline in percent of the FEL photon energy ROI3, rising flank'
884 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL377,'%','relative width of photoline in percent of the FEL photon energy ROI3, tailing flank'
885 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL472,'%','relative width of photoline in percent of the FEL photon energy ROI3, mean'
886 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL373,'%','relative width of photoline in percent of the FEL photon energy ROI4, rising flank'
887 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL378,'%','relative width of photoline in percent of the FEL photon energy ROI4, tailing flank'
888 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL473,'%','relative width of photoline in percent of the FEL photon energy ROI4, mean'
889 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL374,'%','relative width of photoline in percent of the FEL photon energy ROI5, rising flank'
890 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL379,'%','relative width of photoline in percent of the FEL photon energy ROI5, tailing flank'
891 -FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL474,'%','relative width of photoline in percent of the FEL photon energy ROI5, mean'
892 -{{/expand}}
893 893  
894 894  If Opis is running typically on the the averaged data is saved. For several experiments it may make sense to save the information for each single bunch. This is up to now done by saving the complete ADC trace of the TOF setup. This is a huge amount of data and needs processing. This has to be performed after the beamtime in close contact to [[Markus Braune>>mailto:markus.braune@desy.de||shape="rect"]] ( responsible for [[OPIS>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/opis_spectrometer/index_eng.html||shape="rect"]])
895 895  
... ... @@ -903,6 +903,7 @@
903 903  ===== bunch charge =====
904 904  
905 905  {{code language="none"}}/FL2/Electron Diagnostic/Bunch charge/after undulator{{/code}}
906 +//always saved (PBD2)//
906 906  DOOCS prop : {{code language="none"}}FLASH.DIAG/PBD2.TOROID.ML/9FL2BURN/CHARGE.FLASH2{{/code}}
907 907  DAQ channel: {{code language="none"}}FLASH.DIAG/PBD2.TOROID.ML/9FL2BURN/CHARGE.FLASH2{{/code}}
908 908  desc: electron bunch charge FLASH2 (average value for each bunchtrain).
... ... @@ -912,6 +912,7 @@
912 912  ===== electron bunch energy =====
913 913  
914 914  {{code language="none"}}/FL2/Electron Diagnostic/Electron energy/energy of first bunch/behind undulators{{/code}}
916 +//always saved (PBD2)//
915 915  DOOCS prop : {{code language="none"}}FLASH.DIAG/BEAM_ENERGY_MEASUREMENT/FL2XTDS/ENERGY.FLASH2{{/code}}
916 916  DAQ channel: (% style="color:#5e6c84" %)TTF2.DAQ/PBD2.BEAM.ENERGY.MEAS.ML.COPY/FL2XTDS.ENERGY.FLASH2{{code language="none"}}{{/code}}(%%)
917 917  desc: electron bunch energy measured behind the undulator. Data is saved with 10 Hz - BUT (for computation reasons) only the energy of the FIRST bunch is recorded. The data is also available for (% style="color:#5e6c84" %)extraction and septum  in the beginning of FLASH2
... ... @@ -922,19 +922,21 @@
922 922  (% style="color: rgb(94, 108, 132); font-weight: 600; letter-spacing: 0px;" %)
923 923  ===== undulator settings =====
924 924  
925 -{{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/set wavelength 1{{/code}}
926 -DOOCS prop : {{code language="none"}}FLASH.FEL/FL2.WAVELENGTHCONTROL/FLASH2.COLOR1/WAVELENGTH{{/code}}
927 -DAQ channel:  {{code language="none"}}FLASH.FEL/FL2.WAVELENGTHCONTROL/FLASH2.COLOR1/WAVELENGTH{{/code}}
928 -desc: Set value for the anticipated wavelength 1 . This parameter is used to set the undulator gap. It may however deviate from the actual wavelength by several % ... For 2 color operation there is also  the same parameter for COLOR 2
927 +{{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/set wavelength{{/code}}
928 +//always saved (PBD2)//
929 +DOOCS prop : {{code language="none"}}TTF2.FEEDBACK/FL2.WAVELENGTHCONTROL/FLASH2/WAVELENGTH{{/code}}
930 +DAQ channel:  {{code language="none"}}TTF2.FEEDBACK/FL2.WAVELENGTHCONTROL/FLASH2/WAVELENGTH{{/code}}
931 +desc: Set value for the anticipated wavelength . This parameter is used to set the undulator gap. It may however deviate from the actual wavelength by several % ...
929 929  units: nm
930 930  
931 -{{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/SASE13 gap{{/code}}
932 -DOOCS prop : {{code language="none"}}FLASH.UTIL/FL2.UND.MOTOR/FL2SASE13/GAP{{/code}}
933 -DAQ channel:  {{code language="none"}}FLASH.UTIL/FL2.UND.MOTOR/FL2SASE13/GAP{{/code}}
934 -desc: gap value of the undulators. This can be used to follow up how many undulators were closed and if there was a taper. 
934 +{{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/SASE14 gap{{/code}}
935 +//always saved (PBD2)//
936 +DOOCS prop : {{code language="none"}}FLASH.UTIL/FL2.UND.MOTOR/FL2SASE14/GAP{{/code}}
937 +DAQ channel:  {{code language="none"}}FLASH.UTIL/FL2.UND.MOTOR/FL2SASE14/GAP{{/code}}
938 +desc: gap value of the undulators. This can be used to follow up how many undulators were closed and if there was a taper
935 935  units: mm
936 936  
937 -The gap values are saved for all 12 undulators (Nr 2 to 13). Undulator 13 is the one closest to the experimental hall.
941 +The gap values are saved for all 12 undulators (Nr 3 to 14). Undulator 14 is the one closest to the experimental hall.
938 938  
939 939  =====
940 940  arrival time (BAM) =====
... ... @@ -1218,7 +1218,7 @@
1218 1218  
1219 1219  {{view-file att--filename="FSLADAQ-DAQParameterlistFL24-2.pdf"}}{{/view-file}}
1220 1220  
1221 -internal link to parameter list: [[FS-LA DAQ - XWiki>>url:https://xwiki.desy.de/xwiki/bin/view/FSLADAQ/]]
1225 +
1222 1222  {{/info}}
1223 1223  
1224 1224  {{expand title="Laser parameters used until 2023"}}
... ... @@ -1693,3 +1693,8 @@
1693 1693  )))
1694 1694  
1695 1695  [[Contents>>doc:||anchor="Contents"]]
1700 +
1701 +
1702 +{{/layout-cell}}
1703 +{{/layout-section}}
1704 +{{/layout}}