Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 42.1
edited by sndueste
on 2021/02/09 14:42
Change comment: There is no comment for this version
To version 47.1
edited by sndueste
on 2021/02/10 10:14
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,3 +1,6 @@
1 +{{layout}}
2 +{{layout-section ac:type="single"}}
3 +{{layout-cell}}
1 1  == Contents ==
2 2  
3 3  
... ... @@ -18,12 +18,10 @@
18 18  
19 19  [[Contents>>doc:||anchor="Contents"]]
20 20  
21 -== The new (starting 2021) HDF5 format ==
24 +== The FLASH HDF5 format ==
22 22  
23 -\\
26 +FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
24 24  
25 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
26 -
27 27  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
28 28  
29 29  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
... ... @@ -31,7 +31,7 @@
31 31  \\
32 32  
33 33  {{expand title="Discontinued HDF formats"}}
34 -== Comparison to FLASH's deprecated HDF formats ==
35 +=== Comparison to FLASH's deprecated HDF formats ===
35 35  
36 36  Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
37 37  
... ... @@ -40,71 +40,77 @@
40 40  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
41 41  
42 42  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
43 -{{/expand}}
44 44  
45 -=== HDF excerpt: ===
45 +\\
46 46  
47 -* ADC data as example for **fast** **data** (10 Hz):            [[image:attach:image2020-11-16_15-26-28.png||height="250"]]
48 -* The //average// FEL pulse energy as example for the **slow** **data** (different e.g. 1 Hz):[[image:attach:image2020-11-16_15-31-45.png||height="250"]]
47 +**HDF5 example files (old format)**
49 49  
50 - ( every 10th train ID is listed in the HDF group "index")
49 +Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
51 51  
52 -* (((
53 -//zraw// group contains the **original DAQ (DOOCS) names[[image:attach:image2020-11-16_16-26-3.png||thumbnail="true" height="250"]]**
54 -)))
51 +[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
55 55  
56 - of the properties saved in the DESY internal raw format. (For experts)
57 -
58 58  \\
59 59  
60 -\\
55 +[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
61 61  
62 -{{info title="Sample scripts in python"}}
63 -=== Reference implementation (Python) ===
64 -
65 -[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
66 -
67 -(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
68 -{{/info}}
69 -
70 70  \\
71 71  
72 72  \\
60 +{{/expand}}
73 73  
74 -== The discontinued (till 2021) FLASH HDF5 structure ==
62 +=== HDF examples: ===
63 +{{/layout-cell}}
64 +{{/layout-section}}
75 75  
76 -The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
66 +{{layout-section ac:type="three_equal"}}
67 +{{layout-cell}}
68 +* ADC data as example for **fast** **data** (10 Hz):           
77 77  
78 -* Electron Diagnostic
79 -* Photon Diagnostics
80 -* Beamlines
81 -* Experiment
82 -* Timing
83 -[[image:attach:HDF5_structure.jpg||height="400"]]
70 +\\
84 84  
85 -A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
86 -[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
72 +[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
73 +{{/layout-cell}}
87 87  
88 -[[Contents>>doc:||anchor="Contents"]]
75 +{{layout-cell}}
76 +* The //average// FEL pulse energy as example for the **slow** **data** (different e.g. 1 Hz):
89 89  
78 + ( every 10th train ID is listed in the HDF group "index")
79 +
90 90  \\
91 91  
92 -=== HDF5 example files (old format) ===
82 +[[image:attach:image2020-11-16_15-31-45.png||height="250"]]
83 +{{/layout-cell}}
93 93  
94 -Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
85 +{{layout-cell}}
86 +* (((
87 +//zraw// group contains the **original DAQ (DOOCS) names**
88 +)))
95 95  
96 -[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]] [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
90 + of the properties saved in the DESY internal raw format. (For experts)
97 97  
98 98  \\
99 99  
100 -[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
94 +[[image:attach:image2020-11-16_16-26-3.png||height="400"]]
95 +{{/layout-cell}}
96 +{{/layout-section}}
101 101  
102 -\\
98 +{{layout-section ac:type="single"}}
99 +{{layout-cell}}
100 +{{info title="Sample scripts in python"}}
101 +=== Reference implementation (Python) ===
103 103  
104 -\\
103 +[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
105 105  
106 -[[Contents>>doc:||anchor="Contents"]]
105 +(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
106 +{{/info}}
107 +{{/layout-cell}}
108 +{{/layout-section}}
107 107  
110 +{{layout-section ac:type="single"}}
111 +{{layout-cell}}
112 +== Complete list of recordable parameters ==
113 +
114 +The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
108 108  \\
109 109  
110 110  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
... ... @@ -111,9 +111,6 @@
111 111  
112 112  {{id name="DOOCSparameters"/}}
113 113  
114 -The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
115 -The most common and often used ones are summarized below:
116 -
117 117  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
118 118  
119 119  \\
... ... @@ -120,8 +120,6 @@
120 120  
121 121  === FLASH1 ===
122 122  
123 -\\
124 -
125 125  ==== Beamline info (FLASH1) ====
126 126  
127 127  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
... ... @@ -1185,19 +1185,6 @@
1185 1185  [[Contents>>doc:||anchor="Contents"]]
1186 1186  
1187 1187  \\
1188 -
1189 -== Example code showing how to access HDF5 files ==
1190 -
1191 -\\
1192 -
1193 -== HDF5 and DOOCS ==
1194 -
1195 -Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1196 -
1197 -[[Contents>>doc:||anchor="Contents"]]
1198 -
1199 -\\
1200 -
1201 -\\
1202 -
1203 -\\
1190 +{{/layout-cell}}
1191 +{{/layout-section}}
1192 +{{/layout}}