Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 24.1
edited by sndueste
on 2019/09/18 16:21
Change comment: There is no comment for this version
To version 37.1
edited by cpassow
on 2020/11/23 16:26
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.cpassow
Content
... ... @@ -8,9 +8,9 @@
8 8  
9 9  == General information about HDF5 ==
10 10  
11 -HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://www.hdfgroup.org/HDF5/||shape="rect"]]
11 +HDF5 is a data format maintained by the HDF group. For detailed general information see [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support||shape="rect"]]
12 12  
13 -To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://support.hdfgroup.org/products/java/hdfview/index.html||shape="rect"]]
13 +To download the HDF5 Viewer click [[here~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://portal.hdfgroup.org/display/support/Download+HDFView||shape="rect"]]
14 14  
15 15  For use on the DESY Maxwell or PAL (max-fsc or pal) **hdfview** is available in the xray module :
16 16  {{code language="none"}}> module load xray{{/code}}
... ... @@ -18,9 +18,9 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== FLASH HDF5 structure ==
21 +== The current FLASH HDF5 structure ==
22 22  
23 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organised according to train IDs. The general structure is:
23 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
24 24  
25 25  * Electron Diagnostic
26 26  * Photon Diagnostics
... ... @@ -27,17 +27,45 @@
27 27  * Beamlines
28 28  * Experiment
29 29  * Timing
30 -[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure.jpg~|~|alt="HDF5_structure.jpg" width="700" height="500"~]~]>>attach:HDF5_structure.jpg]]
30 +[[image:attach:HDF5_structure.jpg||height="400"]]
31 31  
32 -A detailled description of (most) channels can be found in the lower part of the hdf5 viewer:
33 -[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/HDF5main/HDF5_structure_desc.jpg~|~|alt="HDF5_structure_desc.jpg" width="341" height="413"~]~]>>attach:HDF5_structure_desc.jpg]]
32 +A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
33 +[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
34 34  
35 35  [[Contents>>doc:||anchor="Contents"]]
36 36  
37 37  \\
38 38  
39 -== Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
39 +=== HDF5 example files ===
40 40  
41 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
42 +
43 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
44 +
45 +\\
46 +
47 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
48 +
49 +\\
50 +
51 +\\
52 +
53 +[[Contents>>doc:||anchor="Contents"]]
54 +
55 +\\
56 +
57 +== The new (starting 2021) HDF5 format ==
58 +
59 +Here is [[some documentation on the changes of the HDF5 format>>doc:FLASHUSER.HDF5 format]] that well be available in 2021 (work in progress)
60 +
61 +\\
62 +
63 +\\
64 +
65 +== Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
66 +
67 +{{id name="DOOCSparameters"/}}
68 +
41 41  The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
42 42  The most common and often used ones are summarized below:
43 43  
... ... @@ -117,7 +117,7 @@
117 117  \\
118 118  
119 119  
120 -NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. for more Info contact your local contact
148 +NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. For more Info contact your local contact
121 121  
122 122  [[Contents>>doc:||anchor="Contents"]]
123 123  
... ... @@ -127,13 +127,20 @@
127 127  
128 128  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
129 129  //always saved (PBD)//
130 -DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}}
131 -DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}
132 -desc : calibrated average SASE Energy/pulse measured in the TUNNEL before the attenuator (ion current)
158 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}} (%%)
159 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.TUNNEL/PHOTONFLUX.UJ{{/code}}
160 +
161 +(% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}} (%%)
162 +(% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}(%%)
163 +desc : calibrated average SASE Energy/pulse measured in the TUNNEL upstream the gas attenuator (ion current)
133 133  units : microJ
134 134  
135 135  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy tunnel{{/code}}
136 -//always saved (PBD)//
167 +//always saved (PBD)//
168 +
169 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
170 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.TUNNEL/INTENSITY.TD{{/code}}
171 +
137 137  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT34/VAL{{/code}}
138 138  DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENERGYPULSE.USER{{/code}}
139 139  desc :Energy per pulse Tunnel (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -147,7 +147,11 @@
147 147  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision) **[[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
148 148  
149 149  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy BDA{{/code}}
150 -//always saved (PBD)//
185 +//always saved (PBD)//
186 +
187 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}} (%%)
188 +(% style="color: rgb(165,173,186);" %)New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.PHOTONFLUX/FL1.BDA/PHOTONFLUX.UJ{{/code}}
189 +
151 151  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT05/VAL{{/code}}
152 152  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENPULSEIC{{/code}}
153 153  desc : calibrated average SASE Energy/pulse measured in the BDA (in the experimental hall) after the attenuator (ion current)
... ... @@ -154,7 +154,10 @@
154 154  units : microJ
155 155  
156 156  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA{{/code}}
157 -//always saved (PBD)//
196 +//always saved (PBD)//
197 +
198 +(% style="color: rgb(193,199,208);" %)New MTCA Version: DOOCS prop : 
199 +{{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}New MTCA Version: DAQ channel: {{code language="none"}}FLASH.FEL/XGM.INTENSITY/FL1.BDA/INTENSITY.TD{{/code}}(%%)
158 158  DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT35/VAL{{/code}}
159 159  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.USER{{/code}}
160 160  desc :Energy per pulse BDA (from e-) - the values are set to "0" if there was no SASE beam in the FEL
... ... @@ -167,6 +167,8 @@
167 167  desc :Energy per pulse BDA (from e-) - uncorrected values. There are also values saved if there was no beam ... just background noise
168 168  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
169 169  
212 +\\
213 +
170 170  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Beam position/position BDA horizontal{{/code}}
171 171  //always saved (PBD)//
172 172  DOOCS prop : {{code language="none"}}FLASH.FEL/XGM.POSMON/FL1.BDA/IX.POS{{/code}}
... ... @@ -278,10 +278,25 @@
278 278  //always saved (PBD)//
279 279  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/4DBC3/LOW_CHARGE_ARRIVAL_TIME{{/code}}
280 280  DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/4DBC3.LOW_CHARGE_ARRIVAL_TIME{{/code}}
281 -desc: Electron bunch arrival time measured with the BAM (more or less) before the undulator (pulse resolved data)
282 -units: ps (bigger numbers indicate later arrivaltime of the electrons)
283 -note: besides the arivaltime from FLASH1 there is also the FLASH2/3 electron arrival time saved. [[LINK to detailled infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]] and to a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
325 +desc: Electron bunch arrival time measured with the BAM inside the accelerator - however shows a very good correlation to the arrivaltime  of the XUV pulses in the experiment (pulse resolved data).
326 +units: ps (bigger numbers indicate later arrivaltime of the electrons)
284 284  
328 +{{code language="none"}}/FL1/Electron Diagnostic/BAM/1SFELC/electron bunch arrival time (low charge){{/code}}
329 +//always saved (PBD)//
330 +DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/1SFELC/LOW_CHARGE_ARRIVAL_TIME{{/code}}
331 +DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/1SFELC.LOW_CHARGE_ARRIVAL_TIME{{/code}}
332 +desc: Electron bunch arrival time measured with the BAM  before the undulator (pulse resolved data). This one was newly installed in 2020.
333 +units: ps (bigger numbers indicate later arrivaltime of the electrons)
334 +
335 +{{info title="BAM hints"}}
336 +* besides the arrivaltime from FLASH1 there is also the FLASH2/3 electron arrival time saved.In case of doubt ask your local contact
337 +* [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments >>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
338 +* [[LINK to detailed infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]]
339 +* a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
340 +{{/info}}
341 +
342 +\\
343 +
285 285  ===== electron beam profile =====
286 286  
287 287  {{code language="none"}}/FL1/Electron Diagnostic/Electron bunch profile/TDS profile{{/code}}
... ... @@ -723,17 +723,30 @@
723 723  =====
724 724  arrival time =====
725 725  
785 +{{code language="none"}}/FL2/Electron Diagnostic/BAM/8FL2XTDS/electron bunch arrival time (low charge){{/code}}
786 +//always saved (PBD2)//
787 +DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/8FL2XTDS/LOW_CHARGE_ARRIVAL_TIME{{/code}}
788 +DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/8FL2XTDS.LOW_CHARGE_ARRIVAL_TIME{{/code}}
789 +desc: Electron bunch arrival time measured with the BAM after the FLASH2 undulator (pulse resolved data)
790 +units: ps (bigger numbers indicate later arrivaltime of the electrons)
791 +
726 726  **FL1**{{code language="none"}}//Electron Diagnostic/BAM/4DBC3/electron bunch arrival time (low charge){{/code}}
727 727  //always saved (PBD2)//
728 728  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/4DBC3/LOW_CHARGE_ARRIVAL_TIME{{/code}}
729 729  DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/4DBC3.LOW_CHARGE_ARRIVAL_TIME{{/code}}
730 -desc: Electron bunch arrival time measured with the BAM (more or less) before the undulator (pulse resolved data)
796 +desc: Electron bunch arrival time measured with the BAM  in the accelerator  (pulse resolved data)
731 731  units: ps (bigger numbers indicate later arrivaltime of the electrons)
732 732  
733 -The BAM measures the arrivaltime of** FLASH 1 and FLASH2** in the same data set (thus also sorted in at /FL1/ !). The first values are for FLASH1 bunches.
734 -After a gap of about 70 micros ( 70 colums)  with  as entry for the switching between FLASH 1 and 2 the values for the electrons used in FLASH2 start.  The start time of FLASH2 is also recorded in the DAQ. FLASH1 start time is for historic reasons 700. thus if e.g. the start time of FLASH2 ( property name see below) is 1200 it means that FLASH starts at column 500 (1200-700) ...
799 +\\
735 735  
736 -Docu for BAM:  [[LINK to detailled infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]] and to a recent [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
801 +{{info title="BAM hints"}}
802 +* The BAM 4DBC3 measures the arrivaltime of** FLASH 1 and FLASH2** in the same data set (thus also sorted in at /FL1/ !).
803 +* The BAM 8FL2XTDS measures only for FLASH2 BUT has the same data structure as the other BAMS ... thus there are also (random) values in in the  FLASH1 time slot 
804 +** Structure: The first values are for FLASH1 bunches. After a gap of about 70 micros ( 70 colums)  with  as entry for the switching between FLASH 1 and 2 the values for the electrons used in FLASH2 start.  The start time of FLASH2 is also recorded in the DAQ. FLASH1 start time is for historic reasons 700. thus if e.g. the start time of FLASH2 ( property name see below) is 1200 it means that FLASH starts at column 500 (1200-700) ... .In case of doubt ask your local contact
805 +* [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments >>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
806 +* [[LINK to detailed infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]]
807 +* a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
808 +{{/info}}
737 737  
738 738  
739 739  [[Contents>>doc:||anchor="Contents"]]
... ... @@ -1078,10 +1078,6 @@
1078 1078  
1079 1079  [[Contents>>doc:||anchor="Contents"]]
1080 1080  
1081 -=== How to read HDF5 with Python via FLASHH5 ===
1082 -
1083 -With [[FlashH5>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]] we provide h5py based examples to access and evaluate FLASH's HDF5 files.
1084 -
1085 1085  \\
1086 1086  
1087 1087  == HDF5 and DOOCS ==