Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 38.1
edited by sndueste
on 2021/02/09 10:51
Change comment: There is no comment for this version
To version 30.1
edited by erkben
on 2020/06/16 10:51
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.erkben
Content
... ... @@ -8,9 +8,9 @@
8 8  
9 9  == General information about HDF5 ==
10 10  
11 -HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support||shape="rect"]]
11 +HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://www.hdfgroup.org/HDF5/||shape="rect"]]
12 12  
13 -To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support/Download+HDFView||shape="rect"]]
13 +To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://support.hdfgroup.org/products/java/hdfview/index.html||shape="rect"]]
14 14  
15 15  For use on the DESY Maxwell or PAL (max-fsc or pal) **hdfview** is available in the xray module :
16 16  {{code language="none"}}> module load xray{{/code}}
... ... @@ -18,9 +18,9 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== The current FLASH HDF5 structure ==
21 +== FLASH HDF5 structure ==
22 22  
23 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
23 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organised according to train IDs. The general structure is:
24 24  
25 25  * Electron Diagnostic
26 26  * Photon Diagnostics
... ... @@ -27,16 +27,16 @@
27 27  * Beamlines
28 28  * Experiment
29 29  * Timing
30 -[[image:attach:HDF5_structure.jpg||height="400"]]
30 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure.jpg~|~|alt="HDF5_structure.jpg" width="700" height="500"~]~]>>attach:HDF5_structure.jpg]]
31 31  
32 -A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
33 -[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
32 +A detailled description of (most) channels can be found in the lower part of the hdf5 viewer:
33 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure_desc.jpg~|~|alt="HDF5_structure_desc.jpg" width="341" height="413"~]~]>>attach:HDF5_structure_desc.jpg]]
34 34  
35 35  [[Contents>>doc:||anchor="Contents"]]
36 36  
37 37  \\
38 38  
39 -=== HDF5 example files ===
39 +== HDF5 example files ==
40 40  
41 41  Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
42 42  
... ... @@ -54,14 +54,8 @@
54 54  
55 55  \\
56 56  
57 -== The new (starting 2021) HDF5 format ==
58 -
59 -Here is [[some documentation on the changes of the HDF5 format>>doc:FLASHUSER.HDF5 format]] that well be available in 2021 (work in progress)
60 -
61 61  \\
62 62  
63 -\\
64 -
65 65  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
66 66  
67 67  {{id name="DOOCSparameters"/}}
... ... @@ -145,7 +145,7 @@
145 145  \\
146 146  
147 147  
148 -NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. For more Info contact your local contact
142 +NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. for more Info contact your local contact
149 149  
150 150  [[Contents>>doc:||anchor="Contents"]]
151 151  
... ... @@ -155,20 +155,13 @@
155 155  
156 156  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
157 157  //always saved (PBD)//
158 -(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}} (%%)
159 -(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}}
160 -
161 -(% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}} (%%)
162 -(% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}(%%)
163 -desc : calibrated average SASE Energy/pulse measured in the TUNNEL upstream the gas attenuator (ion current)
152 +DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}}
153 +DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}
154 +desc : calibrated average SASE Energy/pulse measured in the TUNNEL before the attenuator (ion current)
164 164  units : microJ
165 165  
166 166  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
167 -//always saved (PBD)//
168 -
169 -(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
170 -{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}
171 -
158 +//always saved (PBD)//
172 172  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT34/VAL{{/code}}
173 173  DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENERGYPULSE.USER{{/code}}
174 174  desc :Energy per pulse Tunnel (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -182,11 +182,7 @@
182 182  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision) **[[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
183 183  
184 184  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy BDA{{/code}}
185 -//always saved (PBD)//
186 -
187 -(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}} (%%)
188 -(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}}
189 -
172 +//always saved (PBD)//
190 190  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT05/VAL{{/code}}
191 191  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENPULSEIC{{/code}}
192 192  desc : calibrated average SASE Energy/pulse measured in the BDA (in the experimental hall) after the attenuator (ion current)
... ... @@ -193,10 +193,7 @@
193 193  units : microJ
194 194  
195 195  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA{{/code}}
196 -//always saved (PBD)//
197 -
198 -(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
199 -{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}(%%)
179 +//always saved (PBD)//
200 200  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT35/VAL{{/code}}
201 201  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.USER{{/code}}
202 202  desc :Energy per pulse BDA (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -209,8 +209,6 @@
209 209  desc :Energy per pulse BDA (from e-) - uncorrected values. There are also values saved if there was no beam ... just background noise
210 210  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
211 211  
212 -\\
213 -
214 214  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Beam position/position BDA horizontal{{/code}}
215 215  //always saved (PBD)//
216 216  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL1.BDA/IX.POS{{/code}}
... ... @@ -322,25 +322,10 @@
322 322  //always saved (PBD)//
323 323  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/4DBC3/LOW_CHARGE_ARRIVAL_TIME{{/code}}
324 324  DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/4DBC3.LOW_CHARGE_ARRIVAL_TIME{{/code}}
325 -desc: Electron bunch arrival time measured with the BAM inside the accelerator - however shows a very good correlation to the arrivaltime  of the XUV pulses in the experiment (pulse resolved data).
326 -units: ps (bigger numbers indicate later arrivaltime of the electrons)
303 +desc: Electron bunch arrival time measured with the BAM (more or less) before the undulator (pulse resolved data)
304 +units: ps (bigger numbers indicate later arrivaltime of the electrons)
305 +note: besides the arivaltime from FLASH1 there is also the FLASH2/3 electron arrival time saved. [[LINK to detailled infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]] and to a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
327 327  
328 -{{code language="none"}}/FL1/Electron Diagnostic/BAM/1SFELC/electron bunch arrival time (low charge){{/code}}
329 -//always saved (PBD)//
330 -DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/1SFELC/LOW_CHARGE_ARRIVAL_TIME{{/code}}
331 -DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/1SFELC.LOW_CHARGE_ARRIVAL_TIME{{/code}}
332 -desc: Electron bunch arrival time measured with the BAM  before the undulator (pulse resolved data). This one was newly installed in 2020.
333 -units: ps (bigger numbers indicate later arrivaltime of the electrons)
334 -
335 -{{info title="BAM hints"}}
336 -* besides the arrivaltime from FLASH1 there is also the FLASH2/3 electron arrival time saved.In case of doubt ask your local contact
337 -* [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments >>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
338 -* [[LINK to detailed infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]]
339 -* a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
340 -{{/info}}
341 -
342 -\\
343 -
344 344  ===== electron beam profile =====
345 345  
346 346  {{code language="none"}}/FL1/Electron Diagnostic/Electron bunch profile/TDS profile{{/code}}
... ... @@ -630,32 +630,38 @@
630 630  
631 631  \\
632 632  
596 +(% style="color: rgb(255,204,0);" %)average Sigma    to be included in
597 +
598 +(% style="color: rgb(193,199,208);" %){{code language="none"}}/FL2/Photon Diagnostic/GMD/Average energy/energy tunnel uncertainty{{/code}} (%%)
599 +(% style="color: rgb(193,199,208);" %)//always saved (PBD2)// (%%)
600 +(% style="color: rgb(193,199,208);" %)DOOCS prop : FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
601 +(% style="color: rgb(193,199,208);" %)DAQ channel: FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
602 +(% style="color: rgb(193,199,208);" %)desc :  This parameter gives an indication of the error of the measurement of the average pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
603 +(% style="color: rgb(193,199,208);" %)units : microJ
604 +
605 +\\
606 +
607 +\\
608 +
633 633  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
634 634  //always saved (PBD2)//
635 635  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
636 636  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
637 -desc : Energy per pulse measured in the Tunnel  (in front of the gas attenuator and the apertures in the Hall) In addition measurement errors and beam position are included for EACH pulse in the pulse train  !! (see below)
613 +desc : Energy per pulse measured in the Tunnel  (in fromt of the gas attenuator and the apertures in the Hall)
638 638  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
639 639  
640 -For every pulse in the pulse train the information is saved:
641 -
616 +\\
642 642  
643 -1. **Intensity per pulse** (a.u. (more or less µJ ))
644 -1. Intensity per pulse (auxillary GMD) - not used
645 -1. Position horizontal (mm, for a single pulse the position informytion may be very noisy  - talk to your local contact)
646 -1. Position vertical (mm, for a single pulse the position informytion may be very noisy  - talk to your local contact)
647 -1. **Intensity per pulse sigma** (a.u. (more or less µJ ), (% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%))
648 -1. Position horizontal sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
649 -1. Position vertical sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
650 -1. Combined warning and error flags
618 +\\
651 651  
652 -The pulse energy and the error are plotted for the first bunch of the pulse trains saved in this 
653 -file
620 +{{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel uncertainty (sigma){{/code}}
621 +//always saved (PBD2)//
622 +DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.SIGMA.TD{{/code}}
623 +DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL:4{{/code}}
624 +desc :(% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
625 +units : a.u. (more or less µJ - related to the value of the pulse energy )
654 654  
655 -
656 -[[image:attach:image2021-2-9_10-51-6.png||height="250"]]
657 657  
658 -
659 659  {{code language="none"}}{{/code}}
660 660  
661 661  All values for the GMD are also available for the** HALL GMD **which is located in the experimental hall down stream the gas attenuator. If the attenuator is on the ratio between Hall and Tunnel signal shows the attenuation. **BUT NOTE** that the filter unit and the Aperture 4 are downstream the GMD. So if filters and aperture are used this influence is NOT measured by the GMD hall !
... ... @@ -776,30 +776,17 @@
776 776  =====
777 777  arrival time =====
778 778  
779 -{{code language="none"}}/FL2/Electron Diagnostic/BAM/8FL2XTDS/electron bunch arrival time (low charge){{/code}}
780 -//always saved (PBD2)//
781 -DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/8FL2XTDS/LOW_CHARGE_ARRIVAL_TIME{{/code}}
782 -DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/8FL2XTDS.LOW_CHARGE_ARRIVAL_TIME{{/code}}
783 -desc: Electron bunch arrival time measured with the BAM after the FLASH2 undulator (pulse resolved data)
784 -units: ps (bigger numbers indicate later arrivaltime of the electrons)
785 -
786 786  **FL1**{{code language="none"}}//Electron Diagnostic/BAM/4DBC3/electron bunch arrival time (low charge){{/code}}
787 787  //always saved (PBD2)//
788 788  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/4DBC3/LOW_CHARGE_ARRIVAL_TIME{{/code}}
789 789  DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/4DBC3.LOW_CHARGE_ARRIVAL_TIME{{/code}}
790 -desc: Electron bunch arrival time measured with the BAM  in the accelerator  (pulse resolved data)
752 +desc: Electron bunch arrival time measured with the BAM (more or less) before the undulator (pulse resolved data)
791 791  units: ps (bigger numbers indicate later arrivaltime of the electrons)
792 792  
793 -\\
755 +The BAM measures the arrivaltime of** FLASH 1 and FLASH2** in the same data set (thus also sorted in at /FL1/ !). The first values are for FLASH1 bunches.
756 +After a gap of about 70 micros ( 70 colums)  with  as entry for the switching between FLASH 1 and 2 the values for the electrons used in FLASH2 start.  The start time of FLASH2 is also recorded in the DAQ. FLASH1 start time is for historic reasons 700. thus if e.g. the start time of FLASH2 ( property name see below) is 1200 it means that FLASH starts at column 500 (1200-700) ...
794 794  
795 -{{info title="BAM hints"}}
796 -* The BAM 4DBC3 measures the arrivaltime of** FLASH 1 and FLASH2** in the same data set (thus also sorted in at /FL1/ !).
797 -* The BAM 8FL2XTDS measures only for FLASH2 BUT has the same data structure as the other BAMS ... thus there are also (random) values in in the  FLASH1 time slot 
798 -** Structure: The first values are for FLASH1 bunches. After a gap of about 70 micros ( 70 colums)  with  as entry for the switching between FLASH 1 and 2 the values for the electrons used in FLASH2 start.  The start time of FLASH2 is also recorded in the DAQ. FLASH1 start time is for historic reasons 700. thus if e.g. the start time of FLASH2 ( property name see below) is 1200 it means that FLASH starts at column 500 (1200-700) ... .In case of doubt ask your local contact
799 -* [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments >>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
800 -* [[LINK to detailed infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]]
801 -* a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
802 -{{/info}}
758 +Docu for BAM:  [[LINK to detailled infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]] and to a recent [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
803 803  
804 804  
805 805  [[Contents>>doc:||anchor="Contents"]]
... ... @@ -1144,6 +1144,10 @@
1144 1144  
1145 1145  [[Contents>>doc:||anchor="Contents"]]
1146 1146  
1103 +=== How to read HDF5 with Python via FLASHH5 ===
1104 +
1105 +With [[FlashH5>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]] we provide h5py based examples to access and evaluate FLASH's HDF5 files.
1106 +
1147 1147  \\
1148 1148  
1149 1149  == HDF5 and DOOCS ==