Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 48.1
edited by sndueste
on 2021/02/10 10:15
Change comment: There is no comment for this version
To version 44.1
edited by cpassow
on 2021/02/10 09:30
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.cpassow
Content
... ... @@ -21,10 +21,12 @@
21 21  
22 22  [[Contents>>doc:||anchor="Contents"]]
23 23  
24 -== The FLASH HDF5 format ==
24 +== The new (starting 2021) HDF5 format ==
25 25  
26 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
26 +\\
27 27  
28 +FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
29 +
28 28  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
29 29  
30 30  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
... ... @@ -32,7 +32,7 @@
32 32  \\
33 33  
34 34  {{expand title="Discontinued HDF formats"}}
35 -=== Comparison to FLASH's deprecated HDF formats ===
37 +== Comparison to FLASH's deprecated HDF formats ==
36 36  
37 37  Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
38 38  
... ... @@ -41,25 +41,9 @@
41 41  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
42 42  
43 43  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
44 -
45 -\\
46 -
47 -**HDF5 example files (old format)**
48 -
49 -Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
50 -
51 -[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
52 -
53 -\\
54 -
55 -[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
56 -
57 -\\
58 -
59 -\\
60 60  {{/expand}}
61 61  
62 -=== HDF examples: ===
48 +=== HDF excerpt: ===
63 63  {{/layout-cell}}
64 64  {{/layout-section}}
65 65  
... ... @@ -98,7 +98,7 @@
98 98  {{layout-section ac:type="single"}}
99 99  {{layout-cell}}
100 100  {{info title="Sample scripts in python"}}
101 -=== Sample scripts / Reference implementation (Python) ===
87 +=== Reference implementation (Python) ===
102 102  
103 103  [[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
104 104  
... ... @@ -109,15 +109,49 @@
109 109  
110 110  {{layout-section ac:type="single"}}
111 111  {{layout-cell}}
112 -== Complete list of recordable parameters ==
98 +== The discontinued (till 2021) FLASH HDF5 structure ==
113 113  
114 -The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
100 +The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
101 +
102 +* Electron Diagnostic
103 +* Photon Diagnostics
104 +* Beamlines
105 +* Experiment
106 +* Timing
107 +[[image:attach:HDF5_structure.jpg||height="400"]]
108 +
109 +A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
110 +[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
111 +
112 +[[Contents>>doc:||anchor="Contents"]]
113 +
115 115  \\
116 116  
116 +=== HDF5 example files (old format) ===
117 +
118 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
119 +
120 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
121 +
122 +\\
123 +
124 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
125 +
126 +\\
127 +
128 +\\
129 +
130 +[[Contents>>doc:||anchor="Contents"]]
131 +
132 +\\
133 +
117 117  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
118 118  
119 119  {{id name="DOOCSparameters"/}}
120 120  
138 +The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
139 +The most common and often used ones are summarized below:
140 +
121 121  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
122 122  
123 123  \\
... ... @@ -124,6 +124,8 @@
124 124  
125 125  === FLASH1 ===
126 126  
147 +\\
148 +
127 127  ==== Beamline info (FLASH1) ====
128 128  
129 129  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -1085,7 +1085,6 @@
1085 1085  
1086 1086  \\
1087 1087  
1088 -(% class="wrapped" %)
1089 1089  |=(((
1090 1090  earlier HDF path (vers. 0.2)
1091 1091  )))|=(((
... ... @@ -1187,6 +1187,14 @@
1187 1187  [[Contents>>doc:||anchor="Contents"]]
1188 1188  
1189 1189  \\
1211 +
1212 +== HDF5 and DOOCS ==
1213 +
1214 +Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1215 +
1216 +[[Contents>>doc:||anchor="Contents"]]
1217 +
1218 +\\
1190 1190  {{/layout-cell}}
1191 1191  {{/layout-section}}
1192 1192  {{/layout}}