Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 44.1
edited by cpassow
on 2021/02/10 09:30
Change comment: There is no comment for this version
To version 39.1
edited by sndueste
on 2021/02/09 11:53
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.cpassow
1 +XWiki.sndueste
Content
... ... @@ -1,6 +4,3 @@
1 -{{layout}}
2 -{{layout-section ac:type="single"}}
3 -{{layout-cell}}
4 4  == Contents ==
5 5  
6 6  
... ... @@ -21,82 +21,8 @@
21 21  
22 22  [[Contents>>doc:||anchor="Contents"]]
23 23  
24 -== The new (starting 2021) HDF5 format ==
21 +== The current FLASH HDF5 structure ==
25 25  
26 -\\
27 -
28 -FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
29 -
30 -The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
31 -
32 -Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
33 -
34 -\\
35 -
36 -{{expand title="Discontinued HDF formats"}}
37 -== Comparison to FLASH's deprecated HDF formats ==
38 -
39 -Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
40 -
41 -All data sets were aligned by the same global index by filling in missing data. Fast DAQ channels  (pulse synchronous data - update 10 Hz) are filled in by NaN or 0, slow channels (not pulse synchronous data - update e.g. 1 Hz ) are interpolated by the DAQs libraries, by keeping the value constant up to the next change.
42 -
43 -While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
44 -
45 -The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
46 -{{/expand}}
47 -
48 -=== HDF excerpt: ===
49 -{{/layout-cell}}
50 -{{/layout-section}}
51 -
52 -{{layout-section ac:type="three_equal"}}
53 -{{layout-cell}}
54 -* ADC data as example for **fast** **data** (10 Hz):           
55 -
56 -\\
57 -
58 -[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
59 -{{/layout-cell}}
60 -
61 -{{layout-cell}}
62 -* The //average// FEL pulse energy as example for the **slow** **data** (different e.g. 1 Hz):
63 -
64 - ( every 10th train ID is listed in the HDF group "index")
65 -
66 -\\
67 -
68 -[[image:attach:image2020-11-16_15-31-45.png||height="250"]]
69 -{{/layout-cell}}
70 -
71 -{{layout-cell}}
72 -* (((
73 -//zraw// group contains the **original DAQ (DOOCS) names**
74 -)))
75 -
76 - of the properties saved in the DESY internal raw format. (For experts)
77 -
78 -\\
79 -
80 -[[image:attach:image2020-11-16_16-26-3.png||height="400"]]
81 -{{/layout-cell}}
82 -{{/layout-section}}
83 -
84 -{{layout-section ac:type="single"}}
85 -{{layout-cell}}
86 -{{info title="Sample scripts in python"}}
87 -=== Reference implementation (Python) ===
88 -
89 -[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
90 -
91 -(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
92 -{{/info}}
93 -{{/layout-cell}}
94 -{{/layout-section}}
95 -
96 -{{layout-section ac:type="single"}}
97 -{{layout-cell}}
98 -== The discontinued (till 2021) FLASH HDF5 structure ==
99 -
100 100  The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
101 101  
102 102  * Electron Diagnostic
... ... @@ -113,7 +113,7 @@
113 113  
114 114  \\
115 115  
116 -=== HDF5 example files (old format) ===
39 +=== HDF5 example files ===
117 117  
118 118  Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
119 119  
... ... @@ -131,6 +131,14 @@
131 131  
132 132  \\
133 133  
57 +== The new (starting 2021) HDF5 format ==
58 +
59 +Here is [[some documentation on the changes of the HDF5 format>>doc:FLASHUSER.HDF5 format]] that well be available in 2021 (work in progress)
60 +
61 +\\
62 +
63 +\\
64 +
134 134  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ ==
135 135  
136 136  {{id name="DOOCSparameters"/}}
... ... @@ -227,7 +227,7 @@
227 227  (% style="color: rgb(0,0,0);" %)**Discontinued GMD data recording / evaluation  (VME + PhotonFlux ML server)**
228 228  
229 229  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
230 -//always saved (PBD)//
161 +//always saved (PBD)//\\
231 231  
232 232  (% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}TTF2.DAQ/PHFLUX/OUT04/VAL{{/code}} (%%)
233 233  (% style="color: rgb(0,0,0);" %)DAQ channel: {{code language="none"}}PBD.PHFLUX/TUNNEL.ENPULSEIC{{/code}}(%%)
... ... @@ -485,7 +485,7 @@
485 485  ===== electron bunch energy =====
486 486  
487 487  {{code language="none"}}/FL1/Electron Diagnostic/Electron energy/average electron energy{{/code}}
488 -//always saved (PBD)//
419 +_always saved (PBD)
489 489  DOOCS prop : {{code language="none"}}TTF2.DAQ/ENERGY.DOGLEG/E_INTRA_MEAN/VAL{{/code}}
490 490  DAQ channel: {{code language="none"}}PBD.ENERGY.DOGLEG/E_MEAN{{/code}}
491 491  desc: electron bunch energy (average over the bunch train)
... ... @@ -493,7 +493,7 @@
493 493  \\
494 494  
495 495  {{code language="none"}}/FL1/Electron Diagnostic/Electron energy/pulse resolved energy{{/code}}
496 -//always saved (PBD)//
427 +_always saved (PBD)
497 497  DOOCS prop : {{code language="none"}}TTF2.DAQ/ENERGY.DOGLEG/E_SPECT/VAL.TD{{/code}}
498 498  DAQ channel: {{code language="none"}}PBD.ENERGY.DOGLEG/E_SPECT{{/code}}
499 499  desc: electron bunch energy bunch resolved
... ... @@ -502,7 +502,7 @@
502 502  \\
503 503  
504 504  {{code language="none"}}/FL1/Electron Diagnostic/Electron energy/wavelength bunch train average{{/code}}
505 -//always saved (PBD)//
436 +_always saved (PBD)
506 506  DOOCS prop : {{code language="none"}}TTF2.DAQ/ENERGY.DOGLEG/LAMBDA_MEAN/VAL{{/code}}
507 507  DAQ channel: {{code language="none"}}PBD.ENERGY.DOGLEG/LAMBDA_MEAN{{/code}}
508 508  desc: Wavelength calculated by the electron bunch energy (average over the bunch train) (FLASH1)
... ... @@ -696,12 +696,12 @@
696 696  
697 697  ==== Beamline info (FLASH2) ====
698 698  
699 -{{code language="none"}}/FL2/Beamlines/Attenuator/pressure  {{/code}}
700 -(% style="color: rgb(0,0,0);" %)//always saved (PBD2)// (%%)
701 -(% style="color: rgb(0,0,0);" %)DOOCS prop : {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
702 -(% style="color: rgb(0,0,0);" %)DAQ channel:  {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
703 -(% style="color: rgb(0,0,0);" %)desc: set pressure in the gas attenuator (%%)
704 -(% style="color: rgb(0,0,0);" %)units: mbar(%%)
630 +(% style="color: rgb(255,153,0);" %)(not yet available){{code language="none"}}/FL2/Beamlines/Attenuator/pressure  {{/code}}(%%)
631 +(% style="color: rgb(193,199,208);" %)//always saved (PBD2)// (%%)
632 +(% style="color: rgb(193,199,208);" %)DOOCS prop : {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
633 +(% style="color: rgb(193,199,208);" %)DAQ channel:  {{code language="none"}}FLASH.FEL/ATT.GAS_DOSING/FL2.HALL/PRESSURE{{/code}} (%%)
634 +(% style="color: rgb(193,199,208);" %)desc: set pressure in the gas attenuator (%%)
635 +(% style="color: rgb(193,199,208);" %)units: mbar(%%)
705 705  \\
706 706  
707 707  {{code language="none"}}/FL2/Beamlines/FL20/Shutter/open{{/code}}
... ... @@ -720,13 +720,11 @@
720 720  units: degree
721 721  \\
722 722  
723 -/FL2/Beamlines/Filter wheel/position wheel 2
724 -always saved (PBD2)
725 -DOOCS prop : {{code language="none"}}FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS{{/code}}
726 -DAQ channel: {{code language="none"}}FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS{{/code}}
727 -desc: Position of the BL filter wheel 2 - to correlate with the filter material please look [[here>>doc:FLASHUSER.jddd-linked help pages.Filter-Units.Filter wheels in FLASH1 and FLASH2.WebHome]]
728 -units: degree
729 729  
655 +//always saved (PBD2)//
656 +\\\\**[[here>>doc:FLASHUSER.jddd-linked help pages.Filter-Units.Filter wheels in FLASH1 and FLASH2.WebHome]]**
657 +{{code language="none"}}/FL2/Beamlines/Filter wheel/position wheel 2 DOOCS prop : FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS DAQ channel: FLASH.FEL/FL20H.PH.MOTOR/MOTOR2.MOT3/FPOS desc: Position of the BL filter wheel 2 - to correlate with the filter material please look units: degree{{/code}}
658 +
730 730  \\
731 731  
732 732  NOTE: Aperture positions in the beamline as well as the positions of the beam steering mirrors are also saved. for more Info contact your local contact
... ... @@ -805,15 +805,13 @@
805 805  
806 806  \\
807 807  
808 -/FL2/Photon Diagnostic/GMD/Pulse resolved beam position/position tunnel x
809 -always saved (PBD2)
810 -DOOCS prop : FLASH.FEL/XGM.BPM/FL2.HALL/X.TD
811 -DAQ channel: FLASH.FEL/XGM.BPM/FL2.HALL:2
812 -desc: Besides the well calibrated averaged beam position information there is also the option to measure the beam position on a single bunch level. HOWEVER this methide needs a perfectly adjusted signal level (talk to your local contact !!) and also then the signal to noise is rather small and one needs some averaging ... BUT with this option one can determine if there was a spatial slope on a burst (say forst bunches were lower than the last ones or so ...)
813 -units : mm   
814 814  
815 -(x=horizontal, y = vertial)
738 +//always saved (PBD2)//
739 +\\\\{{code language="none"}}/FL2/Photon Diagnostic/GMD/Pulse resolved beam position/position tunnel x DOOCS prop : FLASH.FEL/XGM.BPM/FL2.HALL/X.TDDAQ channel: FLASH.FEL/XGM.BPM/FL2.HALL:2 desc: {{/code}}Besides the well calibrated averaged beam position information there is also the option
740 +{{code language="none"}}to measure the beam position on a single bunch level. HOWEVER this methide needs a perfectly adjusted signal level (talk to your local contact !!) and also then the signal to noise is rather small and one needs some averaging ... BUT with this option one can determine if there was a spatial slope on a burst (say forst bunches were lower than the last ones or so ...)units : mm    {{/code}}
816 816  
742 +{{code language="none"}}(x=horizontal, y = vertial){{/code}}
743 +
817 817  again the same parameter set is available for the **HALL GMD**
818 818  
819 819  \\
... ... @@ -1107,6 +1107,7 @@
1107 1107  
1108 1108  \\
1109 1109  
1037 +(% class="wrapped" %)
1110 1110  |=(((
1111 1111  earlier HDF path (vers. 0.2)
1112 1112  )))|=(((
... ... @@ -1209,13 +1209,67 @@
1209 1209  
1210 1210  \\
1211 1211  
1212 -== HDF5 and DOOCS ==
1140 +== Example code showing how to access HDF5 files ==
1213 1213  
1214 -Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1142 +\\
1215 1215  
1144 +=== Samples how to read HDF5 with Matlab ===
1145 +
1146 +The examples apply to HDF files with HDF tree version before vers. 0.3.0.
1147 +
1148 +(% style="color: rgb(0,128,0);" %)%% read in the needed data p=path; path(p,'D:\mess-daten\DAQ-data\hdf5\Gotthard') % add the actual folder to the path hdf5file='FLASH1_EXP-2016-03-16T1420.h5'; % data with Gotthard and VLS
1149 +
1150 +(% style="color: rgb(0,128,0);" %)% h5disp(hdf5file,'/','min') % to get an idea what is in the file % h5info(hdf5file,'/Experiment/Gotthard1/BL.0') % get info about the individual channel
1151 +
1152 +(% style="color: rgb(0,128,0);" %)% read a Number per 10 Hz pulse train:
1153 +
1154 +(% class="code" %)
1155 +(((
1156 +FEL_Wavelength_energy_server=h5read(hdf5file,'/Photon Diagnostic/Wavelength/Calculated by energy/wavelength' );
1157 +)))
1158 +
1159 +(% style="color: rgb(0,128,0);" %)% read in a 1D array (spectrum) %GMD data GMD_Spectrum=h5read(hdf5file,'/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA');
1160 +
1161 +(% style="color: rgb(0,128,0);" %)% This reads in all the data . One can also limit the amount of data read to a subset of the data stored in the file. e.g. for image Data (Gotthard)
1162 +
1163 +\\
1164 +
1165 +(% class="code" %)
1166 +(((
1167 +Start_event= 500 ; % define the first 10 Hz event
1168 + Number_of_events = 20; % how many 10 Hz events to load
1169 +)))
1170 +
1171 +(% style="color: rgb(0,128,0);" %)% read only part of the data:
1172 +
1173 +(% class="code" %)
1174 +(((
1175 +Gotthard_data=h5read(hdf5file,'/Experiment/Gotthard1/BL.0',[2 650 Start_event],[50 85 Number_of_events]);
1176 +)))
1177 +
1178 +(% style="color: rgb(0,128,0);" %)% start stop increment "manual" for 2 D data : [start bunch number in Gotthard data start pixelin spectrum start sample in 10 Hz trains], % [number of bunches in the Gotthard data number of points in the spectrum number of spectra ]
1179 +
1180 +(% class="code" %)
1181 +(((
1182 +VLS_Spectrum=h5read(hdf5file,'/Photon Diagnostic/Wavelength/VLS online spectrometer/PCO.ROI.X',[400 Start_event],[200 Number_of_events] );
1183 +
1184 +)))
1185 +
1186 +(% style="color: rgb(0,128,0);" %)% start stop increment "manual" for 1D data : [start sample in the spectrum start sample in 10 Hz trains],[number of points in the spectrum number of spectra ]
1187 +
1188 +(% style="color: rgb(0,128,0);" %)%GMD data
1189 +
1190 +(% class="code" %)
1191 +(((
1192 +GMD_Spectrum=h5read(hdf5file,'/Photon Diagnostic/GMD/Pulse resolved energy/energy BDA',[1 Start_event],[40 Number_of_events] );
1193 +)))
1194 +
1216 1216  [[Contents>>doc:||anchor="Contents"]]
1217 1217  
1218 1218  \\
1219 -{{/layout-cell}}
1220 -{{/layout-section}}
1221 -{{/layout}}
1198 +
1199 +== HDF5 and DOOCS ==
1200 +
1201 +Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
1202 +
1203 +[[Contents>>doc:||anchor="Contents"]]