Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 33.1
edited by sndueste
on 2020/10/28 11:26
Change comment: There is no comment for this version
To version 38.1
edited by sndueste
on 2021/02/09 10:51
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -8,9 +8,9 @@
8 8  
9 9  == General information about HDF5 ==
10 10  
11 -HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://www.hdfgroup.org/HDF5/||shape="rect"]]
11 +HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support||shape="rect"]]
12 12  
13 -To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://support.hdfgroup.org/products/java/hdfview/index.html||shape="rect"]]
13 +To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support/Download+HDFView||shape="rect"]]
14 14  
15 15  For use on the DESY Maxwell or PAL (max-fsc or pal) **hdfview** is available in the xray module :
16 16  {{code language="none"}}> module load xray{{/code}}
... ... @@ -18,9 +18,9 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== FLASH HDF5 structure ==
21 +== The current FLASH HDF5 structure ==
22 22  
23 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organised according to train IDs. The general structure is:
23 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
24 24  
25 25  * Electron Diagnostic
26 26  * Photon Diagnostics
... ... @@ -27,16 +27,16 @@
27 27  * Beamlines
28 28  * Experiment
29 29  * Timing
30 -[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure.jpg~|~|alt="HDF5_structure.jpg" width="700" height="500"~]~]>>attach:HDF5_structure.jpg]]
30 +[[image:attach:HDF5_structure.jpg||height="400"]]
31 31  
32 -A detailled description of (most) channels can be found in the lower part of the hdf5 viewer:
33 -[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure_desc.jpg~|~|alt="HDF5_structure_desc.jpg" width="341" height="413"~]~]>>attach:HDF5_structure_desc.jpg]]
32 +A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
33 +[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
34 34  
35 35  [[Contents>>doc:||anchor="Contents"]]
36 36  
37 37  \\
38 38  
39 -== HDF5 example files ==
39 +=== HDF5 example files ===
40 40  
41 41  Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
42 42  
... ... @@ -54,8 +54,14 @@
54 54  
55 55  \\
56 56  
57 +== The new (starting 2021) HDF5 format ==
58 +
59 +Here is [[some documentation on the changes of the HDF5 format>>doc:FLASHUSER.HDF5 format]] that well be available in 2021 (work in progress)
60 +
57 57  \\
58 58  
63 +\\
64 +
59 59  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
60 60  
61 61  {{id name="DOOCSparameters"/}}
... ... @@ -139,7 +139,7 @@
139 139  \\
140 140  
141 141  
142 -NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. for more Info contact your local contact
148 +NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. For more Info contact your local contact
143 143  
144 144  [[Contents>>doc:||anchor="Contents"]]
145 145  
... ... @@ -149,13 +149,20 @@
149 149  
150 150  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
151 151  //always saved (PBD)//
152 -DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}}
153 -DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}
154 -desc : calibrated average SASE Energy/pulse measured in the TUNNEL before the attenuator (ion current)
158 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}} (%%)
159 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}}
160 +
161 +(% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}} (%%)
162 +(% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}(%%)
163 +desc : calibrated average SASE Energy/pulse measured in the TUNNEL upstream the gas attenuator (ion current)
155 155  units : microJ
156 156  
157 157  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
158 -//always saved (PBD)//
167 +//always saved (PBD)//
168 +
169 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
170 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}
171 +
159 159  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT34/VAL{{/code}}
160 160  DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENERGYPULSE.USER{{/code}}
161 161  desc :Energy per pulse Tunnel (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -169,7 +169,11 @@
169 169  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision) **[[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
170 170  
171 171  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy BDA{{/code}}
172 -//always saved (PBD)//
185 +//always saved (PBD)//
186 +
187 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}} (%%)
188 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}}
189 +
173 173  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT05/VAL{{/code}}
174 174  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENPULSEIC{{/code}}
175 175  desc : calibrated average SASE Energy/pulse measured in the BDA (in the experimental hall) after the attenuator (ion current)
... ... @@ -176,7 +176,10 @@
176 176  units : microJ
177 177  
178 178  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA{{/code}}
179 -//always saved (PBD)//
196 +//always saved (PBD)//
197 +
198 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
199 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}(%%)
180 180  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT35/VAL{{/code}}
181 181  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.USER{{/code}}
182 182  desc :Energy per pulse BDA (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -189,6 +189,8 @@
189 189  desc :Energy per pulse BDA (from e-) - uncorrected values. There are also values saved if there was no beam ... just background noise
190 190  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
191 191  
212 +\\
213 +
192 192  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Beam position/position BDA horizontal{{/code}}
193 193  //always saved (PBD)//
194 194  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL1.BDA/IX.POS{{/code}}
... ... @@ -608,38 +608,32 @@
608 608  
609 609  \\
610 610  
611 -(% style="color: rgb(255,204,0);" %)average Sigma    to be included in
612 -
613 -(% style="color: rgb(193,199,208);" %){{code language="none"}}/FL2/Photon Diagnostic/GMD/Average energy/energy tunnel uncertainty{{/code}} (%%)
614 -(% style="color: rgb(193,199,208);" %)//always saved (PBD2)// (%%)
615 -(% style="color: rgb(193,199,208);" %)DOOCS prop : FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
616 -(% style="color: rgb(193,199,208);" %)DAQ channel: FLASH.FEL/XGM.PHOTONFLUX/FL2.TUNNEL/PHOTONFLUX.UJ.SIGMA (%%)
617 -(% style="color: rgb(193,199,208);" %)desc :  This parameter gives an indication of the error of the measurement of the average pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
618 -(% style="color: rgb(193,199,208);" %)units : microJ
619 -
620 -\\
621 -
622 -\\
623 -
624 624  {{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
625 625  //always saved (PBD2)//
626 626  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
627 627  DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.TD{{/code}}
628 -desc : Energy per pulse measured in the Tunnel  (in fromt of the gas attenuator and the apertures in the Hall)
637 +desc : Energy per pulse measured in the Tunnel  (in front of the gas attenuator and the apertures in the Hall) In addition measurement errors and beam position are included for EACH pulse in the pulse train  !! (see below)
629 629  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
630 630  
631 -\\
640 +For every pulse in the pulse train the information is saved:
641 +
632 632  
633 -\\
643 +1. **Intensity per pulse** (a.u. (more or less µJ ))
644 +1. Intensity per pulse (auxillary GMD) - not used
645 +1. Position horizontal (mm, for a single pulse the position informytion may be very noisy  - talk to your local contact)
646 +1. Position vertical (mm, for a single pulse the position informytion may be very noisy  - talk to your local contact)
647 +1. **Intensity per pulse sigma** (a.u. (more or less µJ ), (% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%))
648 +1. Position horizontal sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
649 +1. Position vertical sigma  (mm, indicates the error (RMS, sigma) of the measurement according to known uncertainties and signal to noise)
650 +1. Combined warning and error flags
634 634  
635 -{{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel uncertainty (sigma){{/code}}
636 -//always saved (PBD2)//
637 -DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL/INTENSITY.SIGMA.TD{{/code}}
638 -DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL2.TUNNEL:4{{/code}}
639 -desc :(% style="color: rgb(0,0,0);" %)This parameter gives an indication of the error of the measurement of the pulse energy. This takes signal to noise,  detector resolution, uncertainties in crossection etc into account. (it is NOT the measurement of the statistical fluctuation of the SASE pulses)(%%)
640 -units : a.u. (more or less µJ - related to the value of the pulse energy )
652 +The pulse energy and the error are plotted for the first bunch of the pulse trains saved in this 
653 +file
641 641  
655 +
656 +[[image:attach:image2021-2-9_10-51-6.png||height="250"]]
642 642  
658 +
643 643  {{code language="none"}}{{/code}}
644 644  
645 645  All values for the GMD are also available for the** HALL GMD **which is located in the experimental hall down stream the gas attenuator. If the attenuator is on the ratio between Hall and Tunnel signal shows the attenuation. **BUT NOTE** that the filter unit and the Aperture 4 are downstream the GMD. So if filters and aperture are used this influence is NOT measured by the GMD hall !
... ... @@ -1128,10 +1128,6 @@
1128 1128  
1129 1129  [[Contents>>doc:||anchor="Contents"]]
1130 1130  
1131 -=== How to read HDF5 with Python via FLASHH5 ===
1132 -
1133 -With [[FlashH5>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]] we provide h5py based examples to access and evaluate FLASH's HDF5 files.
1134 -
1135 1135  \\
1136 1136  
1137 1137  == HDF5 and DOOCS ==