Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 49.1
edited by sndueste
on 2021/02/17 16:29
Change comment: There is no comment for this version
To version 44.1
edited by cpassow
on 2021/02/10 09:30
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.cpassow
Content
... ... @@ -19,18 +19,14 @@
19 19  {{code language="none"}}> module load xray{{/code}}
20 20  {{code language="none"}}> hdfview{{/code}}
21 21  
22 -or you can use
22 +[[Contents>>doc:||anchor="Contents"]]
23 23  
24 -{{code language="none"}}> silix{{/code}}
24 +== The new (starting 2021) HDF5 format ==
25 25  
26 26  \\
27 27  
28 -[[Contents>>doc:||anchor="Contents"]]
28 +FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
29 29  
30 -== The FLASH HDF5 format ==
31 -
32 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
33 -
34 34  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
35 35  
36 36  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
... ... @@ -38,7 +38,7 @@
38 38  \\
39 39  
40 40  {{expand title="Discontinued HDF formats"}}
41 -=== Comparison to FLASH's deprecated HDF formats ===
37 +== Comparison to FLASH's deprecated HDF formats ==
42 42  
43 43  Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
44 44  
... ... @@ -47,25 +47,9 @@
47 47  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
48 48  
49 49  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
50 -
51 -\\
52 -
53 -**HDF5 example files (old format)**
54 -
55 -Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
56 -
57 -[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
58 -
59 -\\
60 -
61 -[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
62 -
63 -\\
64 -
65 -\\
66 66  {{/expand}}
67 67  
68 -=== HDF examples: ===
48 +=== HDF excerpt: ===
69 69  {{/layout-cell}}
70 70  {{/layout-section}}
71 71  
... ... @@ -104,7 +104,7 @@
104 104  {{layout-section ac:type="single"}}
105 105  {{layout-cell}}
106 106  {{info title="Sample scripts in python"}}
107 -=== Sample scripts / Reference implementation (Python) ===
87 +=== Reference implementation (Python) ===
108 108  
109 109  [[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
110 110  
... ... @@ -115,15 +115,49 @@
115 115  
116 116  {{layout-section ac:type="single"}}
117 117  {{layout-cell}}
118 -== Complete list of recordable parameters ==
98 +== The discontinued (till 2021) FLASH HDF5 structure ==
119 119  
120 -The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
100 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
101 +
102 +* Electron Diagnostic
103 +* Photon Diagnostics
104 +* Beamlines
105 +* Experiment
106 +* Timing
107 +[[image:attach:HDF5_structure.jpg||height="400"]]
108 +
109 +A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
110 +[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
111 +
112 +[[Contents>>doc:||anchor="Contents"]]
113 +
121 121  \\
122 122  
116 +=== HDF5 example files (old format) ===
117 +
118 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
119 +
120 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
121 +
122 +\\
123 +
124 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
125 +
126 +\\
127 +
128 +\\
129 +
130 +[[Contents>>doc:||anchor="Contents"]]
131 +
132 +\\
133 +
123 123  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
124 124  
125 125  {{id name="DOOCSparameters"/}}
126 126  
138 +The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
139 +The most common and often used ones are summarized below:
140 +
127 127  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
128 128  
129 129  \\
... ... @@ -130,6 +130,8 @@
130 130  
131 131  === FLASH1 ===
132 132  
147 +\\
148 +
133 133  ==== Beamline info (FLASH1) ====
134 134  
135 135  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -1091,7 +1091,6 @@
1091 1091  
1092 1092  \\
1093 1093  
1094 -(% class="wrapped" %)
1095 1095  |=(((
1096 1096  earlier HDF path (vers. 0.2)
1097 1097  )))|=(((
... ... @@ -1193,6 +1193,14 @@
1193 1193  [[Contents>>doc:||anchor="Contents"]]
1194 1194  
1195 1195  \\
1211 +
1212 +== HDF5 and DOOCS ==
1213 +
1214 +Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1215 +
1216 +[[Contents>>doc:||anchor="Contents"]]
1217 +
1218 +\\
1196 1196  {{/layout-cell}}
1197 1197  {{/layout-section}}
1198 1198  {{/layout}}