Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55

From 40.1 to 39.1 From 55.1 to 54.1

From version 54.1

edited by sndueste
on 2022/01/14 09:44

Change comment: There is no comment for this version

To version 40.1

edited by sndueste
on 2021/02/09 14:35

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -1,6 +4,3 @@
--{{layout}}
--{{layout-section ac:type="single"}}
--{{layout-cell}}
  == Contents ==
@@ -19,18 +19,14 @@
  {{code language="none"}}> module load xray{{/code}}
  {{code language="none"}}> hdfview{{/code}}
--or you can use
++[[Contents>>doc:||anchor="Contents"]]
--{{code language="none"}}> silx view{{/code}}
++== The new (starting 2021) HDF5 format ==
  \\
--[[Contents>>doc:||anchor="Contents"]]
++FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate.If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
--== The FLASH HDF5 format ==
--
--FLASH provides a conversion of its data acquisition (DAQ) to the commonly used [[HDF5>>url:https://www.hdfgroup.org/solutions/hdf5/||shape="rect"]] format. Correlated  data are mapped by a primary index called **train ID**. Every data set has an individual index of train IDs to identify the data even when data are missing or deviate in update rate. If the data set of choice contains gaps,  users have to decide how to treat missing values. DAQ channels are spread across various files with one file name pattern for each DAQ. This means users have to assemble data from different files if necessary.
--
  The hierarchy is realized using a human readable named HDF tree with each DAQ channel containing the data sets "//value//" and "//index//". Additionally, the HDF group //zraw// contains a tree of the original DAQ channel names.
  Reference implementation, which follows the concept of Python libraries like Pandas, Xarray, or Dask, is given below.
@@ -38,7 +38,7 @@
  \\
  {{expand title="Discontinued HDF formats"}}
--=== Comparison to FLASH's deprecated HDF formats ===
++== Comparison to FLASH's deprecated HDF formats ==
  Before 2021, FLASH provided two different HDF formats formally known as //near-online// and //offline// HDF files.
@@ -47,77 +47,66 @@
  While the near-online HDF files were converted live during the beamtime, the offline HDF files were manually compiled weeks/months later. Near-online HDF files were used by a provided API [[BeamtimeDaqAccess>>url:https://confluence.desy.de/display/FLASHUSER/Near-Online+data+analysis#Near-Onlinedataanalysis-BriefUserGuidetoBeamtimeDAQAccess||shape="rect"]], which also searches for DAQ channels in files. With assembled files "by run", all DAQ channels were existing in the same HDF file. While shorter runs usually fitted into one file, longer runs still had to be aggregated over several files. The creation of the assembled type of files still involves the use of fairly unstable DAQ Mex-functions As the environment required for using the Mex-functions is already deprecated, its continued existence is uncertain.
  The HDF hierarchy is almost identical in all formats. While in the the recent format each DAQ channel contains the data sets "value" and "index", the deprecated format had one data set for each DAQ channel. The deprecated formats supplies no time axis parameters for spectra data types.
++{{/expand}}
++== HDF excerpt: ==
++
  \\
--**HDF5 example files (old format)**
++ADC data as example for **fast** **data** (10 Hz):
--Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
++\\
--[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]]  [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
++[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
  \\
--[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
++{{info title="Sample scripts in python"}}
++== Reference implementation (Python) ==
++[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
++
++(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
++{{/info}}
++
  \\
  \\
--{{/expand}}
--=== HDF examples: ===
--{{/layout-cell}}
--{{/layout-section}}
++== The discontinued (till 2021) FLASH HDF5 structure ==
--{{layout-section ac:type="three_equal"}}
--{{layout-cell}}
--*  ADC data as example for **fast** **data** (10 Hz):
++The photon diagnostic, electron diagnostic and beamline information as well as the information about the pump-probe laser and the infrastructure offered for users (GHz/MHz ADCs) can be included in one HDF5 file which is organized according to train IDs. The general structure is:
--\\
++* Electron Diagnostic
++* Photon Diagnostics
++* Beamlines
++* Experiment
++* Timing
++[[image:attach:HDF5_structure.jpg||height="400"]]
--[[image:attach:image2020-11-16_15-26-28.png||height="250"]]
--{{/layout-cell}}
++A detailed description of (most) channels can be found in the lower part of the hdf5 viewer:
++[[image:attach:HDF5_structure_desc.jpg||thumbnail="true" height="250"]]
--{{layout-cell}}
--* The //average// FEL pulse energy as example for the **slow** **data** (different e.g. 1 Hz):
++[[Contents>>doc:||anchor="Contents"]]
--           ( every 10th train ID is listed in the HDF group "index")
--
  \\
--[[image:attach:image2020-11-16_15-31-45.png||height="250"]]
--{{/layout-cell}}
++=== HDF5 example files ===
--{{layout-cell}}
--* (((
--//zraw// group contains the **original DAQ (DOOCS) names**
--)))
++Here we have a few HDF5 samples (User data combined with Photon diagnostics data) from a few beamtimes showing the different kind options.
--           of the properties saved in the DESY internal raw format. (For experts)
++[[image:attach:image2019-10-21_17-2-50.png||thumbnail="true" width="300"]]  [[download HDF5 (Images @ FL2)>>url:https://desycloud.desy.de/index.php/s/nyEgeCWJFC4gao2||shape="rect"]]
  \\
--[[image:attach:image2020-11-16_16-26-3.png||height="400"]]
--{{/layout-cell}}
--{{/layout-section}}
++[[~[~[image:attach:image2019-10-22_10-52-27.png~|~|thumbnail="true" width="300"~]~]download HDF5 (GHz ADC and OPIS @ FL2)>>url:https://desycloud.desy.de/index.php/s/AeA2kPNNnZgX95A||shape="rect"]]
--{{layout-section ac:type="single"}}
--{{layout-cell}}
--{{info}}
--=== Reference implementation (Python) ===
++\\
--(imperative)  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
++\\
--(% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
--{{/info}}
--{{/layout-cell}}
--{{/layout-section}}
++[[Contents>>doc:||anchor="Contents"]]
--{{layout-section ac:type="single"}}
--{{layout-cell}}
--== Complete list of recordable parameters ==
--
--The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
  \\
  == Most popular FLASH parameters and their names in HDF5, DOOCS and (raw) DAQ  ==
@@ -124,6 +124,9 @@
  {{id name="DOOCSparameters"/}}
++The complete list for the relation between DOOCS names and HDF5 names for the recordable parameters can be found in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
++The most common and often used ones are summarized below:
++
  Note, the HDF group and data set names apply to our HDF tree version since vers. 0.3.0.
  \\
@@ -130,6 +130,8 @@
  === FLASH1 ===
++\\
++
  ==== Beamline info (FLASH1) ====
  {{code language="none"}}/FL1/Beamlines/Attenuator/pressure{{/code}}
@@ -208,7 +208,6 @@
  \\
--{{expand title="GMD format untill 2021"}}
  (% style="color: rgb(0,0,0);" %)**Discontinued GMD data recording / evaluation  (VME + PhotonFlux ML server)**
  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
@@ -256,12 +256,9 @@
  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.FF{{/code}}
  desc :Energy per pulse BDA (from e-) - uncorrected values. There are also values saved if there was no beam ... just background noise
  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
--{{/expand}}
  \\
--\\
--
  (% style="color: rgb(0,0,0);" %)**NEW (2021) GMD data recording / evaluation  (MTCA, analog to FLASH2 and XFEL)**
  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
@@ -538,7 +538,7 @@
  //always saved (PBD)//
  DOOCS prop : {{code language="none"}}TTF2.DIAG/PBD.TOROID.ML/12EXP/CHARGE.TD{{/code}}
  DAQ channel: {{code language="none"}}TTF2.DIAG/PBD.TOROID.ML/12EXP{{/code}}
--desc: The bunch pattern as function of time in a burst recorded by toroid diagnostic BEHIND the undulator. (FLASH1)
++desc: The bunch pattern as function of time in a burst recorded by toroide diagnostic BEHIND the undulator. (FLASH1)
  units:
  \\
@@ -611,7 +611,7 @@
  //always saved (PBD)//
  DOOCS prop : {{code language="none"}}TTF2.FEL/TDOLFEL/TDOLFEL/STREAK.CAM.TIME{{/code}}
  DOOCS prop : {{code language="none"}}TTF2.FEL/TDOLFEL/TDOLFEL/STREAK.CAM.TIME{{/code}}
--desc: delay time between the optical laser and the FEL units: ps
++desc: delaytime between the optical laser and the FEL units: ps
  \\
@@ -621,7 +621,7 @@
  ==== User Data (FLASH1) ====
--The data saved specifically for detectors at an experiment will show up in /Experiment/ there is a large number of options for cameras or monitoring of slow properties (motor positions etc) for user experiments. For details please ask your local contact.
++The data saved specifically for detectors at an experiment will show up in /Experiment/ there is a large number of options for cameras or monitoring pslow properties (motor positons etc) for user experiments. For details please ask your local contact.
  NOTE: If parameters for an experiment are included on short notice the correct naming in the HDF5 may not be in time and the data will show up in /uncategorized/ with the DOOCS names
@@ -649,13 +649,13 @@
  {{code language="none"}}/FL1/Experiment/BL3/ADQ412 GHz ADC/CH03/TD{{/code}}
  DOOCS prop : {{code language="none"}}FLASH.FEL/ADC.ADQ.PG/EXP1.CH00/CH00.TD  or CH00.DAQ.TD{{/code}}
--here the {{code language="none"}}CH00.TD{{/code}} is the full ADC trace as it is sampled ( typically several 100.000 samples per pulse train) while the {{code language="none"}}CH00.DAQ.TD{{/code}} trace only has the number of samples which are sent to the DAQ OR if //grouping// is activated the {{code language="none"}}CH00.DAQ.TD{{/code}} contains only the grouped spectra. To read the ADC trace with an online analysis program the {{code language="none"}}CH00.DAQ.TD{{/code}} is used preferably.
++here the {{code language="none"}}CH00.TD{{/code}} is the full ADC trace as it is sampled ( typically several 100.000 samples per pulse train) while the {{code language="none"}}CH00.DAQ.TD{{/code}} trace only has the number of samples which are sent to the DAQ OR if //grouping// is activated the {{code language="none"}}CH00.DAQ.TD{{/code}} conatins only the grouped spectra. To read the ADC trace with an online analysis program the {{code language="none"}}CH00.DAQ.TD{{/code}} is used preferablly.
  DAQ channel: {{code language="none"}}FLASH.FEL/ADC.ADQ.PG/EXP1.CH00{{/code}}
  In addition there are also additional parameters saved like:
  * {{code language="none"}}sample frequency{{/code}}: it shows the sample frequency in MHz (number of samples per µs). NOTE: the clock of the ADC is NOT synchronized to the FLASH timing system. Thus the number of samples between bunches in the bunch train may be not integer numbers which will be show up for long bunch trains.
--* {{code language="none"}}number of samples{{/code}}: total number of samples recorded for each 10 Hz trigger
++* {{code language="none"}}number of samples{{/code}}: total number of samoles recorded for each 10 Hz trigger
  * {{code language="none"}}error (ADC):{{/code}} 0 indicates that there was no error
  ===== MHz ADCs =====
@@ -668,7 +668,7 @@
  In addition there are also additional parameters saved like:
  * {{code language="none"}}sample frequency{{/code}}: it shows the sample frequency in MHz (number of samples per µs). NOTE: the clock of the ADC is NOT synchronized to the FLASH timing system. Thus the number of samples between bunches in the bunch train may be not integer numbers which will be show up for long bunch trains.
--* {{code language="none"}}number of samples{{/code}}: total number of samples recorded for each 10 Hz trigger
++* {{code language="none"}}number of samples{{/code}}: total number of samoles recorded for each 10 Hz trigger
  [[Contents>>doc:||anchor="Contents"]]
@@ -829,7 +829,7 @@
  // saved opon request (PBD2)//
  DOOCS prop : {{code language="none"}}FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
  DAQ channel:{{code language="none"}} FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
--desc : mean wavelength ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
++desc : meanwavelength ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
  units : nm
  \\
@@ -843,11 +843,11 @@
  \\
--If Opis is running typically on the the averaged data is saved. For several experiments it may make sense to save the information for each single bunch. This is up to now done by saving the complete ADC trace of the TOF setup. This is a huge amount of data and needs processing. This has to be performed after the beamtime in close contact to [[Markus Braune>>mailto:markus.braune@desy.de||shape="rect"]] ( responsible for [[OPIS>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/opis_spectrometer/index_eng.html||shape="rect"]])
++If Opis is running typically on the the averaged data is saved. For several experiments it may make sense to save the information for each single bunch. This is up to now done by savng the compleate ADC trace of the TOF setup. This is a huge amount of data and needs processing. This has to be performed after the beamtime in close contact to [[Markus Braune>>mailto:markus.braune@desy.de||shape="rect"]] ( respobsible for [[OPIS>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/opis_spectrometer/index_eng.html||shape="rect"]])
  \\
--In case OPIS was not operating there is still information about the **set wavelength** for the undulators (see below) which may differ by up to 5 % from the actual wavelength due to different settings in the FEL ...
++In case OPIS was not operating there is still informaton about the **set wavelength** for the undulators (see below) which may differ by up to 5 % from the actual wavelength due to different settings in the FEL ...
  \\
@@ -866,21 +866,8 @@
  \\
--===== electron bunch energy =====
++===== undulator settings =====
--{{code language="none"}}/FL2/Electron Diagnostic/Electron energy/energy of first bunch/behind undulators{{/code}}
--//always saved (PBD2)//
--DOOCS prop : {{code language="none"}}FLASH.DIAG/BEAM_ENERGY_MEASUREMENT/FL2XTDS/ENERGY.FLASH2{{/code}}
--DAQ channel: (% style="color: rgb(94,108,132);" %)TTF2.DAQ/PBD2.BEAM.ENERGY.MEAS.ML.COPY/FL2XTDS.ENERGY.FLASH2{{code language="none"}}{{/code}}(%%)
--desc: electron bunch energy measured behind the undulator. Data is saved with 10 Hz - BUT (for computation reasons) only the energy of the FIRST bunch is recorded. The data is also available for (% style="color: rgb(94,108,132);" %)extraction and septum  in the beginning of FLASH2
--
--(% style="letter-spacing: 0.0px;" %)units: (% class="twikiNewLink" %)MeV
--
--\\
--
--(% style="color: rgb(94,108,132);font-weight: 600;letter-spacing: 0.0px;" %)
--undulator settings
--
  {{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/set wavelength{{/code}}
  //always saved (PBD2)//
  DOOCS prop : {{code language="none"}}TTF2.FEEDBACK/FL2.WAVELENGTHCONTROL/FLASH2/WAVELENGTH{{/code}}
@@ -1210,6 +1210,13 @@
  [[Contents>>doc:||anchor="Contents"]]
  \\
--{{/layout-cell}}
--{{/layout-section}}
--{{/layout}}
++
++== Example code showing how to access HDF5 files ==
++
++\\
++
++== HDF5 and DOOCS ==
++
++Here is an outdated [[list with the available properties that are always saved (PBD) for FLASH1 as>>attach:FLASH1__DaqChannel2HdfNamePbd.xlsx]] HDF5 names and the corresponding DOOCS names
++
++[[Contents>>doc:||anchor="Contents"]]

Changes for page The FLASH HDF5 structure

Summary

Details

Applications

Navigation