Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 40.1
edited by sndueste
on 2021/02/09 14:35
Change comment: There is no comment for this version
To version 46.1
edited by sndueste
on 2021/02/10 10:13
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,3 +1,6 @@
1 +{{layout}}
2 +{{layout-section ac:type="single"}}
3 +{{layout-cell}}
1 1  == Contents ==
2 2  
3 3  
... ... @@ -18,12 +18,10 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== The new (starting 2021) HDF5 format ==
24 +== The FLASH HDF5 format ==
22 22  
23 -\\
26 +FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
24 24  
25 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
26 -
27 27  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
28 28  
29 29  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
... ... @@ -40,66 +40,77 @@
40 40  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
41 41  
42 42  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
43 -{{/expand}}
44 44  
45 -== HDF excerpt: ==
46 -
47 47  \\
48 48  
49 -ADC data as example for **fast** **data** (10 Hz):
47 +**HDF5 example files (old format)**
50 50  
51 -\\
49 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
52 52  
53 -[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
51 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
54 54  
55 55  \\
56 56  
57 -{{info title="Sample scripts in python"}}
58 -== Reference implementation (Python) ==
55 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
59 59  
60 -[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
61 -
62 -(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
63 -{{/info}}
64 -
65 65  \\
66 66  
67 67  \\
60 +{{/expand}}
68 68  
69 -== The discontinued (till 2021) FLASH HDF5 structure ==
62 +=== HDF examples: ===
63 +{{/layout-cell}}
64 +{{/layout-section}}
70 70  
71 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
66 +{{layout-section ac:type="three_equal"}}
67 +{{layout-cell}}
68 +* ADC data as example for **fast** **data** (10 Hz):           
72 72  
73 -* Electron Diagnostic
74 -* Photon Diagnostics
75 -* Beamlines
76 -* Experiment
77 -* Timing
78 -[[image:attach:HDF5_structure.jpg||height="400"]]
70 +\\
79 79  
80 -A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
81 -[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
72 +[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
73 +{{/layout-cell}}
82 82  
83 -[[Contents>>doc:||anchor="Contents"]]
75 +{{layout-cell}}
76 +* The //average// FEL pulse energy as example for the **slow** **data** (different e.g. 1 Hz):
84 84  
78 + ( every 10th train ID is listed in the HDF group "index")
79 +
85 85  \\
86 86  
87 -=== HDF5 example files ===
82 +[[image:attach:image2020-11-16_15-31-45.png||height="250"]]
83 +{{/layout-cell}}
88 88  
89 -Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
85 +{{layout-cell}}
86 +* (((
87 +//zraw// group contains the **original DAQ (DOOCS) names**
88 +)))
90 90  
91 -[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
90 + of the properties saved in the DESY internal raw format. (For experts)
92 92  
93 93  \\
94 94  
95 -[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
94 +[[image:attach:image2020-11-16_16-26-3.png||height="400"]]
95 +{{/layout-cell}}
96 +{{/layout-section}}
96 96  
97 -\\
98 +{{layout-section ac:type="single"}}
99 +{{layout-cell}}
100 +{{info title="Sample scripts in python"}}
101 +=== Reference implementation (Python) ===
98 98  
99 -\\
103 +[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
100 100  
101 -[[Contents>>doc:||anchor="Contents"]]
105 +(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
106 +{{/info}}
107 +{{/layout-cell}}
108 +{{/layout-section}}
102 102  
110 +{{layout-section ac:type="single"}}
111 +{{layout-cell}}
112 +== Complete list of recordable parameters ==
113 +
114 +The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
103 103  \\
104 104  
105 105  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
... ... @@ -106,9 +106,6 @@
106 106  
107 107  {{id name="DOOCSparameters"/}}
108 108  
109 -The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
110 -The most common and often used ones are summarized below:
111 -
112 112  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
113 113  
114 114  \\
... ... @@ -115,8 +115,6 @@
115 115  
116 116  === FLASH1 ===
117 117  
118 -\\
119 -
120 120  ==== Beamline info (FLASH1) ====
121 121  
122 122  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -1180,13 +1180,6 @@
1180 1180  [[Contents>>doc:||anchor="Contents"]]
1181 1181  
1182 1182  \\
1183 -
1184 -== Example code showing how to access HDF5 files ==
1185 -
1186 -\\
1187 -
1188 -== HDF5 and DOOCS ==
1189 -
1190 -Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1191 -
1192 -[[Contents>>doc:||anchor="Contents"]]
1190 +{{/layout-cell}}
1191 +{{/layout-section}}
1192 +{{/layout}}