Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 44.1
edited by cpassow
on 2021/02/10 09:30
Change comment: There is no comment for this version
To version 48.1
edited by sndueste
on 2021/02/10 10:15
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.cpassow
1 +XWiki.sndueste
Content
... ... @@ -21,12 +21,10 @@
21 21  
22 22  [[Contents>>doc:||anchor="Contents"]]
23 23  
24 -== The new (starting 2021) HDF5 format ==
24 +== The FLASH HDF5 format ==
25 25  
26 -\\
26 +FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
27 27  
28 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
29 -
30 30  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
31 31  
32 32  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
... ... @@ -34,7 +34,7 @@
34 34  \\
35 35  
36 36  {{expand title="Discontinued HDF formats"}}
37 -== Comparison to FLASH's deprecated HDF formats ==
35 +=== Comparison to FLASH's deprecated HDF formats ===
38 38  
39 39  Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
40 40  
... ... @@ -43,9 +43,25 @@
43 43  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
44 44  
45 45  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
44 +
45 +\\
46 +
47 +**HDF5 example files (old format)**
48 +
49 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
50 +
51 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
52 +
53 +\\
54 +
55 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
56 +
57 +\\
58 +
59 +\\
46 46  {{/expand}}
47 47  
48 -=== HDF excerpt: ===
62 +=== HDF examples: ===
49 49  {{/layout-cell}}
50 50  {{/layout-section}}
51 51  
... ... @@ -84,7 +84,7 @@
84 84  {{layout-section ac:type="single"}}
85 85  {{layout-cell}}
86 86  {{info title="Sample scripts in python"}}
87 -=== Reference implementation (Python) ===
101 +=== Sample scripts / Reference implementation (Python) ===
88 88  
89 89  [[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
90 90  
... ... @@ -95,49 +95,15 @@
95 95  
96 96  {{layout-section ac:type="single"}}
97 97  {{layout-cell}}
98 -== The discontinued (till 2021) FLASH HDF5 structure ==
112 +== Complete list of recordable parameters ==
99 99  
100 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
101 -
102 -* Electron Diagnostic
103 -* Photon Diagnostics
104 -* Beamlines
105 -* Experiment
106 -* Timing
107 -[[image:attach:HDF5_structure.jpg||height="400"]]
108 -
109 -A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
110 -[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
111 -
112 -[[Contents>>doc:||anchor="Contents"]]
113 -
114 +The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
114 114  \\
115 115  
116 -=== HDF5 example files (old format) ===
117 -
118 -Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
119 -
120 -[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
121 -
122 -\\
123 -
124 -[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
125 -
126 -\\
127 -
128 -\\
129 -
130 -[[Contents>>doc:||anchor="Contents"]]
131 -
132 -\\
133 -
134 134  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
135 135  
136 136  {{id name="DOOCSparameters"/}}
137 137  
138 -The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
139 -The most common and often used ones are summarized below:
140 -
141 141  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
142 142  
143 143  \\
... ... @@ -144,8 +144,6 @@
144 144  
145 145  === FLASH1 ===
146 146  
147 -\\
148 -
149 149  ==== Beamline info (FLASH1) ====
150 150  
151 151  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -1107,6 +1107,7 @@
1107 1107  
1108 1108  \\
1109 1109  
1088 +(% class="wrapped" %)
1110 1110  |=(((
1111 1111  earlier HDF path (vers. 0.2)
1112 1112  )))|=(((
... ... @@ -1208,14 +1208,6 @@
1208 1208  [[Contents>>doc:||anchor="Contents"]]
1209 1209  
1210 1210  \\
1211 -
1212 -== HDF5 and DOOCS ==
1213 -
1214 -Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1215 -
1216 -[[Contents>>doc:||anchor="Contents"]]
1217 -
1218 -\\
1219 1219  {{/layout-cell}}
1220 1220  {{/layout-section}}
1221 1221  {{/layout}}