Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 39.1
edited by sndueste
on 2021/02/09 11:53
Change comment: There is no comment for this version
To version 34.1
edited by sndueste
on 2020/10/28 16:14
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -8,9 +8,9 @@
8 8  
9 9  == General information about HDF5 ==
10 10  
11 -HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support||shape="rect"]]
11 +HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://www.hdfgroup.org/HDF5/||shape="rect"]]
12 12  
13 -To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support/Download+HDFView||shape="rect"]]
13 +To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://support.hdfgroup.org/products/java/hdfview/index.html||shape="rect"]]
14 14  
15 15  For use on the DESY Maxwell or PAL (max-fsc or pal) **hdfview** is available in the xray module :
16 16  {{code language="none"}}> module load xray{{/code}}
... ... @@ -18,9 +18,9 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== The current FLASH HDF5 structure ==
21 +== FLASH HDF5 structure ==
22 22  
23 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
23 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organised according to train IDs. The general structure is:
24 24  
25 25  * Electron Diagnostic
26 26  * Photon Diagnostics
... ... @@ -27,16 +27,16 @@
27 27  * Beamlines
28 28  * Experiment
29 29  * Timing
30 -[[image:attach:HDF5_structure.jpg||height="400"]]
30 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure.jpg~|~|alt="HDF5_structure.jpg" width="700" height="500"~]~]>>attach:HDF5_structure.jpg]]
31 31  
32 -A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
33 -[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
32 +A detailled description of (most) channels can be found in the lower part of the hdf5 viewer:
33 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure_desc.jpg~|~|alt="HDF5_structure_desc.jpg" width="341" height="413"~]~]>>attach:HDF5_structure_desc.jpg]]
34 34  
35 35  [[Contents>>doc:||anchor="Contents"]]
36 36  
37 37  \\
38 38  
39 -=== HDF5 example files ===
39 +== HDF5 example files ==
40 40  
41 41  Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
42 42  
... ... @@ -54,14 +54,8 @@
54 54  
55 55  \\
56 56  
57 -== The new (starting 2021) HDF5 format ==
58 -
59 -Here is [[some documentation on the changes of the HDF5 format>>doc:FLASHUSER.HDF5 format]] that well be available in 2021 (work in progress)
60 -
61 61  \\
62 62  
63 -\\
64 -
65 65  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
66 66  
67 67  {{id name="DOOCSparameters"/}}
... ... @@ -151,14 +151,12 @@
151 151  
152 152  \\
153 153  
154 -==== Photon Diagnostics SASE ([[GMD>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/gmd_intensity_and_position/index_eng.html||shape="rect"]]) ====
148 +==== Photon Diagnostics SASE ([[GMD>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/gmd_intensity_and_position/index_eng.html||shape="rect"]]) / Spectrometer (FLASH1) ====
155 155  
156 -\\
157 -
158 -(% style="color: rgb(0,0,0);" %)**Discontinued GMD data recording / evaluation  (VME + PhotonFlux ML server)**
159 -
160 160  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
161 -//always saved (PBD)//\\
151 +//always saved (PBD)//
152 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}} (%%)
153 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}}
162 162  
163 163  (% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}} (%%)
164 164  (% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}(%%)
... ... @@ -168,6 +168,9 @@
168 168  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
169 169  //always saved (PBD)//
170 170  
163 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
164 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}
165 +
171 171  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT34/VAL{{/code}}
172 172  DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENERGYPULSE.USER{{/code}}
173 173  desc :Energy per pulse Tunnel (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -183,6 +183,9 @@
183 183  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy BDA{{/code}}
184 184  //always saved (PBD)//
185 185  
181 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}} (%%)
182 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}}
183 +
186 186  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT05/VAL{{/code}}
187 187  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENPULSEIC{{/code}}
188 188  desc : calibrated average SASE Energy/pulse measured in the BDA (in the experimental hall) after the attenuator (ion current)
... ... @@ -191,6 +191,8 @@
191 191  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA{{/code}}
192 192  //always saved (PBD)//
193 193  
192 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
193 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}(%%)
194 194  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT35/VAL{{/code}}
195 195  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.USER{{/code}}
196 196  desc :Energy per pulse BDA (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -205,55 +205,6 @@
205 205  
206 206  \\
207 207  
208 -(% style="color: rgb(0,0,0);" %)**NEW (2021) GMD data recording / evaluation  (MTCA, analog to FLASH2 and XFEL)**
209 -
210 -{{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
211 -//always saved (PBD)//
212 -(% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}} (%%)
213 -(% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}}
214 -
215 -desc : calibrated average SASE Energy/pulse measured in the TUNNEL upstream the gas attenuator
216 -units : microJ
217 -
218 -\\
219 -
220 -{{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
221 -//always saved (PBD)//
222 -
223 -(% style="color: rgb(0,0,0);" %)DOOCS prop : 
224 -{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}
225 -
226 -desc : Energy per pulse measured in the Tunnel.  In addition measurement errors and beam position are included for EACH pulse in the pulse train  !! (see below)
227 -units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
228 -
229 -{{info title="GMD pulse resolved data structure"}}
230 -For every pulse in the pulse train the information is saved:
231 -
232 -
233 -1. **Intensity per pulse** (a.u. (more or less µJ ))
234 -1. Intensity per pulse (auxillary GMD) - not used
235 -1. Position horizontal (mm, for a single pulse the position information may be very noisy  - talk to your local contact)
236 -1. Position vertical (mm, for a single pulse the position information may be very noisy  - talk to your local contact)
237 -1. **Intensity per pulse sigma** (a.u. (more or less µJ ), (% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%))
238 -1. Position horizontal sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
239 -1. Position vertical sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
240 -1. Combined warning and error flags
241 -
242 -The pulse energy and the error are plotted for the first bunch of the pulse trains saved in this 
243 -file
244 -
245 -
246 -[[image:attach:image2021-2-9_10-51-6.png||height="250"]]
247 -
248 -\\
249 -{{/info}}
250 -
251 -All values for the GMD are also available for the** BDA GMD **which is located in the experimental hall down stream the gas attenuator. If the attenuator is on the ratio between BDA and Tunnel signal shows the attenuation. **BUT NOTE** that the filter units  are downstream the GMD. So if filters  are used this influence is NOT measured by the GMD BDA !
252 -
253 -\\
254 -
255 -Besides  pulse energy the GMD also provides information about the beam position
256 -
257 257  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Beam position/position BDA horizontal{{/code}}
258 258  //always saved (PBD)//
259 259  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL1.BDA/IX.POS{{/code}}
... ... @@ -282,10 +282,6 @@
282 282  desc :Beam position of the photon Beam determined by the GMD (TUNNEL, y=vertical)
283 283  units : mm
284 284  
285 -\\
286 -
287 -==== Spectrometer (FLASH1) ====
288 -
289 289  {{code language="none"}}/FL1/Photon Diagnostic/Wavelength/Tunnelspectrometer/wavelength{{/code}}
290 290  _always saved (when Spectrum is measured !!) (PBD) _
291 291  DOOCS prop : {{code language="none"}}TTF2.EXP/PHOTONWL.ML/WAVE_LENGTH/VAL.TD{{/code}}
... ... @@ -677,37 +677,40 @@
677 677  
678 678  \\
679 679  
627 +(% style="color: rgb(255,204,0);" %)average Sigma    to be included in
628 +
629 +(% style="color: rgb(193,199,208);" %){{code language="none"}}/FL2/Photon Diagnostic/GMD/Average energy/energy tunnel uncertainty{{/code}} (%%)
630 +(% style="color: rgb(193,199,208);" %)//always saved (PBD2)// (%%)
631 +(% style="color: rgb(193,199,208);" %)DOOCS prop : FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
632 +(% style="color: rgb(193,199,208);" %)DAQ channel: FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
633 +(% style="color: rgb(193,199,208);" %)desc :  This parameter gives an indication of the error of the measurement of the average pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
634 +(% style="color: rgb(193,199,208);" %)units : microJ
635 +
636 +\\
637 +
638 +\\
639 +
680 680  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
681 681  //always saved (PBD2)//
682 682  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
683 683  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
684 -desc : Energy per pulse measured in the Tunnel  (in front of the gas attenuator and the apertures in the Hall) In addition measurement errors and beam position are included for EACH pulse in the pulse train  !! (see below)
644 +desc : Energy per pulse measured in the Tunnel  (in fromt of the gas attenuator and the apertures in the Hall)
685 685  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
686 686  
687 -{{info title="GMD pulse resolved data structure"}}
688 -For every pulse in the pulse train the information is saved:
689 -
647 +\\
690 690  
691 -1. **Intensity per pulse** (a.u. (more or less µJ ))
692 -1. Intensity per pulse (auxillary GMD) - not used
693 -1. Position horizontal (mm, for a single pulse the position information may be very noisy  - talk to your local contact)
694 -1. Position vertical (mm, for a single pulse the position information may be very noisy  - talk to your local contact)
695 -1. **Intensity per pulse sigma** (a.u. (more or less µJ ), (% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%))
696 -1. Position horizontal sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
697 -1. Position vertical sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
698 -1. Combined warning and error flags
649 +\\
699 699  
700 -The pulse energy and the error are plotted for the first bunch of the pulse trains saved in this 
701 -file
651 +{{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel uncertainty (sigma){{/code}}
652 +//always saved (PBD2)//
653 +DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.SIGMA.TD{{/code}}
654 +DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL:4{{/code}}
655 +desc :(% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
656 +units : a.u. (more or less µJ - related to the value of the pulse energy )
702 702  
703 -
704 -[[image:attach:image2021-2-9_10-51-6.png||height="250"]]
705 705  
706 -\\
707 -{{/info}}
659 +{{code language="none"}}{{/code}}
708 708  
709 -\\
710 -
711 711  All values for the GMD are also available for the** HALL GMD **which is located in the experimental hall down stream the gas attenuator. If the attenuator is on the ratio between Hall and Tunnel signal shows the attenuation. **BUT NOTE** that the filter unit and the Aperture 4 are downstream the GMD. So if filters and aperture are used this influence is NOT measured by the GMD hall !
712 712  
713 713  \\
... ... @@ -1194,6 +1194,10 @@
1194 1194  
1195 1195  [[Contents>>doc:||anchor="Contents"]]
1196 1196  
1147 +=== How to read HDF5 with Python via FLASHH5 ===
1148 +
1149 +With [[FlashH5>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]] we provide h5py based examples to access and evaluate FLASH's HDF5 files.
1150 +
1197 1197  \\
1198 1198  
1199 1199  == HDF5 and DOOCS ==