Changes for page The FLASH HDF5 structure

Last modified by sndueste on 2025/02/06 10:55
From version 58.1
edited by sndueste
on 2022/09/07 17:20
Change comment: There is no comment for this version
To version 48.1
edited by sndueste
on 2021/02/10 10:15
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -19,12 +19,6 @@
19 19  {{code language="none"}}> module load xray{{/code}}
20 20  {{code language="none"}}> hdfview{{/code}}
21 21  
22 -or you can use
23 -
24 -{{code language="none"}}> silx view{{/code}}
25 -
26 -\\
27 -
28 28  [[Contents>>doc:||anchor="Contents"]]
29 29  
30 30  == The FLASH HDF5 format ==
... ... @@ -103,12 +103,12 @@
103 103  
104 104  {{layout-section ac:type="single"}}
105 105  {{layout-cell}}
106 -{{info}}
107 -=== Reference implementation (Python) ===
100 +{{info title="Sample scripts in python"}}
101 +=== Sample scripts / Reference implementation (Python) ===
108 108  
109 -(imperative)  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
103 +[[~[~[image:attach:binder_badge.png~|~|thumbnail="true" width="120"~]~]>>url:https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.desy.de%2Fchristopher.passow%2Fflash-daq-hdf/master||shape="rect"]]
110 110  
111 -(% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
105 +(% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
112 112  {{/info}}
113 113  {{/layout-cell}}
114 114  {{/layout-section}}
... ... @@ -206,7 +206,8 @@
206 206  
207 207  ==== Photon Diagnostics SASE ([[GMD>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/gmd_intensity_and_position/index_eng.html||shape="rect"]]) ====
208 208  
209 -{{expand title="Discontinued GMD format (used until 2021)"}}
203 +\\
204 +
210 210  (% style="color: rgb(0,0,0);" %)**Discontinued GMD data recording / evaluation  (VME + PhotonFlux ML server)**
211 211  
212 212  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
... ... @@ -254,11 +254,10 @@
254 254  DAQ channel: {{code language="none"}}PBD.PHFLUX/BDA.ENERGYPULSE.FF{{/code}}
255 255  desc :Energy per pulse BDA (from e-) - uncorrected values. There are also values saved if there was no beam ... just background noise
256 256  units : a.u. (more or less µJ but need to be calibrated with the "Average energy" for good precision)** [[see here for help>>doc:FLASHUSER.jddd-linked help pages.Calibrating the pulse resolved (electron) data from GMD.WebHome]]**
257 -{{/expand}}
258 258  
259 259  \\
260 260  
261 -(% style="letter-spacing: 0px; color: rgb(0, 0, 0)" %)**NEW (since 2021) GMD data recording / evaluation  (same format as FLASH2 and XFEL)**
255 +(% style="color: rgb(0,0,0);" %)**NEW (2021) GMD data recording / evaluation  (MTCA, analog to FLASH2 and XFEL)**
262 262  
263 263  {{code language="none"}}/FL1/Photon Diagnostic/GMD/Average energy/energy tunnel{{/code}}
264 264  //always saved (PBD)//
... ... @@ -416,27 +416,8 @@
416 416  units: nC
417 417  
418 418  =====
419 -arrival time (BAM) =====
413 +arrival time =====
420 420  
421 -{{info title="BAM information: updates 2022"}}
422 -* The data format of the BAM has been completely altered in the 2022 shutdown
423 -* before 2022 BAMs were always saving the arrival time information for each 1µs bucked regardless if there were electrons in the accelerator or not. IN addition the arrival times for  FL1 and FL2 were saved in the same parameter ...
424 -* THIS is now different. there are new parameters for pulses that go to FL1 and to FL2 (in detail: first time slot of the accelerator and second)
425 -* There has been also a renaming  (and relocation) of the BAMs.
426 -** acc:  4DBC3 → FL0.DBC2
427 -** FL1: 1SFELC →  FL1.SFELC
428 -** FL2: FL2XTDS → (% style="color: rgb(23,43,77);" %)FL2.SEED5
429 -* for more Info: [[LINK to detailed infos from MSK>>url:https://confluence.desy.de/display/SDiagPublic/BAM+Data+Structure||shape="rect"]]
430 -* [[Info collection about the BAMs and how to use the BAM data>>url:https://confluence.desy.de/display/FLASHUSER/Info+collection+for+the+BAM||shape="rect"]]
431 -* [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments>>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
432 -* a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
433 -{{/info}}
434 -
435 -\\
436 -
437 -{{expand title="Discontinued BAM format (used until end 2021)"}}
438 -(% style="color: rgb(0,0,0);" %)**Discontinued BAM data recording **
439 -
440 440  {{code language="none"}}/FL1/Electron Diagnostic/BAM/4DBC3/electron bunch arrival time (low charge){{/code}}
441 441  //always saved (PBD)//
442 442  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/4DBC3/LOW_CHARGE_ARRIVAL_TIME{{/code}}
... ... @@ -444,70 +444,17 @@
444 444  desc: Electron bunch arrival time measured with the BAM inside the accelerator - however shows a very good correlation to the arrivaltime  of the XUV pulses in the experiment (pulse resolved data).
445 445  units: ps (bigger numbers indicate later arrivaltime of the electrons)
446 446  
447 -\\
448 -
449 449  {{code language="none"}}/FL1/Electron Diagnostic/BAM/1SFELC/electron bunch arrival time (low charge){{/code}}
450 450  //always saved (PBD)//
451 451  DOOCS prop : {{code language="none"}}FLASH.SDIAG/BAM/1SFELC/LOW_CHARGE_ARRIVAL_TIME{{/code}}
452 452  DAQ channel: {{code language="none"}}FLASH.SDIAG/BAM.DAQ/1SFELC.LOW_CHARGE_ARRIVAL_TIME{{/code}}
453 453  desc: Electron bunch arrival time measured with the BAM  before the undulator (pulse resolved data). This one was newly installed in 2020.
454 -units: ps (bigger numbers indicate later arrival time of the electrons)
455 -{{/expand}}
427 +units: ps (bigger numbers indicate later arrivaltime of the electrons)
456 456  
457 -(% style="color: rgb(255,102,0);" %)DBC2/electron bunch arrival time{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
458 -//always saved (PBD)//
459 -DOOCS prop : FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIME.absolute.SA1
460 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIME.absolute.SA1 (%%)
461 -desc: Electron bunch arrival time measured with the BAM inside the accelerator (after bunch compressor 2) - It shows a very good correlation to the arrival time  of the XUV pulses in the experiment (see help).  The SA1 indicates the data is recorded for the first time slot at FLASh, typically used for FLASH1
462 -units: fs (bigger numbers (typically) indicate later arrival times of the electrons).
463 -
464 -\\
465 -
466 -(% style="color: rgb(255,102,0);" %)DBC2/error{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
467 -//always saved (PBD)//
468 -DOOCS prop : FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIM(% style="color: rgb(0,0,0);" %)E.bamError(%%)
469 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIME.BAMERROR(%%)
470 -desc: If the value is 0 , the BAM is working well. If it is non-zero there is a problem !!
471 -
472 -\\
473 -
474 -(% style="color: rgb(255,102,0);" %)DBC2/error{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
475 -//always saved (PBD)//
476 -DOOCS prop : FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIM(% style="color: rgb(0,0,0);" %)E..bamStatus.//1//(%%)
477 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/FL0.DBC2/ARRIVAL_TIME.BAMSTATUS(%%)
478 -desc: (% style="letter-spacing: 0.0px;" %) status bit: 0 - data is valid; 1 - beam present; 2 - calibration ongoing; 3 - feedback enabled; 4 - feedback acting;  mostly check for bit 0 == 1 is sufficient
479 -
480 -\\
481 -
482 -(% style="color: rgb(23,43,77);" %)SFELC(% style="color: rgb(255,102,0);" %)/electron bunch arrival time{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
483 -//always saved (PBD)//
484 -DOOCS prop : FLASH.SDIAG/BAM/(% style="color: rgb(23,43,77);" %)FL1.SFELC(%%)/ARRIVAL_TIME.absolute.SA1
485 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/(% style="color: rgb(255, 102, 0); color: rgb(23, 43, 77)" %)FL1.SFELC(% style="color: rgb(255,102,0);" %)/ARRIVAL_TIME.absolute.SA1 (%%)
486 -desc: Electron bunch arrival time measured  with the BAM  before the undulator (pulse resolved data). This one was newly installed in 2020.  The SA1 indicates the data is recorded for the first time slot at FLASh, typically used for FLASH1
487 -units: fs (bigger numbers (typically) indicate later arrival times of the electrons).
488 -
489 -\\
490 -
491 -(% style="color: rgb(23,43,77);" %)SFELC(% style="color: rgb(255,102,0);" %)/error{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
492 -//always saved (PBD)//
493 -DOOCS prop : FLASH.SDIAG/BAM/(% style="color: rgb(23,43,77);" %)FL1.SFELC(%%)/ARRIVAL_TIM(% style="color: rgb(0,0,0);" %)E.bamError(%%)
494 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/(% style="color: rgb(255, 102, 0); color: rgb(23, 43, 77)" %)FL1.SFELC(% style="color: rgb(255,102,0);" %)/ARRIVAL_TIME.BAMERROR(%%)
495 -desc: If the value is 0 , the BAM is working well. If it is non-zero there is a problem !!
496 -
497 -\\
498 -
499 -(% style="color: rgb(23,43,77);" %)SFELC(% style="color: rgb(255,102,0);" %)/error{{code language="none"}}/FL1/Electron Diagnostic/BAM/{{/code}}(%%)
500 -//always saved (PBD)//
501 -DOOCS prop : FLASH.SDIAG/BAM/(% style="color: rgb(23,43,77);" %)FL1.SFELC(%%)/ARRIVAL_TIM(% style="color: rgb(0,0,0);" %)E..bamStatus.//1//(%%)
502 -DAQ channel: (% style="color: rgb(255,102,0);" %)FLASH.SDIAG/BAM/(% style="color: rgb(255, 102, 0); color: rgb(23, 43, 77)" %)FL1.SFELC(% style="color: rgb(255,102,0);" %)/ARRIVAL_TIME.BAMSTATUS(%%)
503 -desc: (% style="" %) status bit: 0 - data is valid; 1 - beam present; 2 - calibration ongoing; 3 - feedback enabled; 4 - feedback acting;  mostly check for bit 0 == 1 is sufficient
504 -
505 -\\
506 -
507 507  {{info title="BAM hints"}}
508 508  * besides the arrivaltime from FLASH1 there is also the FLASH2/3 electron arrival time saved.In case of doubt ask your local contact
509 509  * [[Link a collection of papers related to the BAM and the analysis of pump-probe experiments >>doc:FLASHUSER.Additional helpful things.FLASH beamlines and instruments references.WebHome]]
510 -* [[LINK to detailed infos from MSK>>url:https://confluence.desy.de/display/SDiagPublic/BAM+Data+Structure||shape="rect"]]
432 +* [[LINK to detailed infos from MSK (may only work inside DESY network~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://www.desy.de/~~mbock/pages/BAM_daq_channel_descriptions.html||shape="rect"]]
511 511  * a recent  [[talk about the working principle of the BAM>>attach:BAM-basics and outlook-2018_DESY-template_16-9Format.pdf]]
512 512  {{/info}}
513 513  
... ... @@ -606,7 +606,7 @@
606 606  //always saved (PBD)//
607 607  DOOCS prop : {{code language="none"}}TTF2.DIAG/PBD.TOROID.ML/12EXP/CHARGE.TD{{/code}}
608 608  DAQ channel: {{code language="none"}}TTF2.DIAG/PBD.TOROID.ML/12EXP{{/code}}
609 -desc: The bunch pattern as function of time in a burst recorded by toroid diagnostic BEHIND the undulator. (FLASH1)
531 +desc: The bunch pattern as function of time in a burst recorded by toroide diagnostic BEHIND the undulator. (FLASH1)
610 610  units:
611 611  \\
612 612  
... ... @@ -679,7 +679,7 @@
679 679  //always saved (PBD)//
680 680  DOOCS prop : {{code language="none"}}TTF2.FEL/TDOLFEL/TDOLFEL/STREAK.CAM.TIME{{/code}}
681 681  DOOCS prop : {{code language="none"}}TTF2.FEL/TDOLFEL/TDOLFEL/STREAK.CAM.TIME{{/code}}
682 -desc: delay time between the optical laser and the FEL units: ps
604 +desc: delaytime between the optical laser and the FEL units: ps
683 683  
684 684  \\
685 685  
... ... @@ -689,7 +689,7 @@
689 689  
690 690  ==== User Data (FLASH1) ====
691 691  
692 -The data saved specifically for detectors at an experiment will show up in /Experiment/ there is a large number of options for cameras or monitoring of slow properties (motor positions etc) for user experiments. For details please ask your local contact.
614 +The data saved specifically for detectors at an experiment will show up in /Experiment/ there is a large number of options for cameras or monitoring pslow properties (motor positons etc) for user experiments. For details please ask your local contact.
693 693  
694 694  NOTE: If parameters for an experiment are included on short notice the correct naming in the HDF5 may not be in time and the data will show up in /uncategorized/ with the DOOCS names
695 695  
... ... @@ -717,13 +717,13 @@
717 717  {{code language="none"}}/FL1/Experiment/BL3/ADQ412 GHz ADC/CH03/TD{{/code}}
718 718  
719 719  DOOCS prop : {{code language="none"}}FLASH.FEL/ADC.ADQ.PG/EXP1.CH00/CH00.TD  or CH00.DAQ.TD{{/code}}
720 -here the {{code language="none"}}CH00.TD{{/code}} is the full ADC trace as it is sampled ( typically several 100.000 samples per pulse train) while the {{code language="none"}}CH00.DAQ.TD{{/code}} trace only has the number of samples which are sent to the DAQ OR if //grouping// is activated the {{code language="none"}}CH00.DAQ.TD{{/code}} contains only the grouped spectra. To read the ADC trace with an online analysis program the {{code language="none"}}CH00.DAQ.TD{{/code}} is used preferably.
642 +here the {{code language="none"}}CH00.TD{{/code}} is the full ADC trace as it is sampled ( typically several 100.000 samples per pulse train) while the {{code language="none"}}CH00.DAQ.TD{{/code}} trace only has the number of samples which are sent to the DAQ OR if //grouping// is activated the {{code language="none"}}CH00.DAQ.TD{{/code}} conatins only the grouped spectra. To read the ADC trace with an online analysis program the {{code language="none"}}CH00.DAQ.TD{{/code}} is used preferablly.
721 721  DAQ channel: {{code language="none"}}FLASH.FEL/ADC.ADQ.PG/EXP1.CH00{{/code}}
722 722  
723 723  In addition there are also additional parameters saved like:
724 724  
725 725  * {{code language="none"}}sample frequency{{/code}}: it shows the sample frequency in MHz (number of samples per µs). NOTE: the clock of the ADC is NOT synchronized to the FLASH timing system. Thus the number of samples between bunches in the bunch train may be not integer numbers which will be show up for long bunch trains.
726 -* {{code language="none"}}number of samples{{/code}}: total number of samples recorded for each 10 Hz trigger
648 +* {{code language="none"}}number of samples{{/code}}: total number of samoles recorded for each 10 Hz trigger
727 727  * {{code language="none"}}error (ADC):{{/code}} 0 indicates that there was no error
728 728  
729 729  ===== MHz ADCs =====
... ... @@ -736,7 +736,7 @@
736 736  In addition there are also additional parameters saved like:
737 737  
738 738  * {{code language="none"}}sample frequency{{/code}}: it shows the sample frequency in MHz (number of samples per µs). NOTE: the clock of the ADC is NOT synchronized to the FLASH timing system. Thus the number of samples between bunches in the bunch train may be not integer numbers which will be show up for long bunch trains.
739 -* {{code language="none"}}number of samples{{/code}}: total number of samples recorded for each 10 Hz trigger
661 +* {{code language="none"}}number of samples{{/code}}: total number of samoles recorded for each 10 Hz trigger
740 740  
741 741  [[Contents>>doc:||anchor="Contents"]]
742 742  
... ... @@ -897,7 +897,7 @@
897 897  // saved opon request (PBD2)//
898 898  DOOCS prop : {{code language="none"}}FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
899 899  DAQ channel:{{code language="none"}} FLASH.UTIL/STORE/FL2.TUNNEL.OPIS/VAL040{{/code}}
900 -desc : mean wavelength ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
822 +desc : meanwavelength ( ~~ 1 sec averaging time ) measured in the TUNNEL for a specific bunch out of the bunch train (via photoelectron spectroscopy)
901 901  units : nm
902 902  
903 903  \\
... ... @@ -911,11 +911,11 @@
911 911  
912 912  \\
913 913  
914 -If Opis is running typically on the the averaged data is saved. For several experiments it may make sense to save the information for each single bunch. This is up to now done by saving the complete ADC trace of the TOF setup. This is a huge amount of data and needs processing. This has to be performed after the beamtime in close contact to [[Markus Braune>>mailto:markus.braune@desy.de||shape="rect"]] ( responsible for [[OPIS>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/opis_spectrometer/index_eng.html||shape="rect"]])
836 +If Opis is running typically on the the averaged data is saved. For several experiments it may make sense to save the information for each single bunch. This is up to now done by savng the compleate ADC trace of the TOF setup. This is a huge amount of data and needs processing. This has to be performed after the beamtime in close contact to [[Markus Braune>>mailto:markus.braune@desy.de||shape="rect"]] ( respobsible for [[OPIS>>url:http://photon-science.desy.de/facilities/flash/photon_diagnostics/opis_spectrometer/index_eng.html||shape="rect"]])
915 915  
916 916  \\
917 917  
918 -In case OPIS was not operating there is still information about the **set wavelength** for the undulators (see below) which may differ by up to 5 % from the actual wavelength due to different settings in the FEL ...
840 +In case OPIS was not operating there is still informaton about the **set wavelength** for the undulators (see below) which may differ by up to 5 % from the actual wavelength due to different settings in the FEL ...
919 919  
920 920  \\
921 921  
... ... @@ -934,21 +934,8 @@
934 934  
935 935  \\
936 936  
937 -===== electron bunch energy =====
859 +===== undulator settings =====
938 938  
939 -{{code language="none"}}/FL2/Electron Diagnostic/Electron energy/energy of first bunch/behind undulators{{/code}}
940 -//always saved (PBD2)//
941 -DOOCS prop : {{code language="none"}}FLASH.DIAG/BEAM_ENERGY_MEASUREMENT/FL2XTDS/ENERGY.FLASH2{{/code}}
942 -DAQ channel: (% style="color: rgb(94,108,132);" %)TTF2.DAQ/PBD2.BEAM.ENERGY.MEAS.ML.COPY/FL2XTDS.ENERGY.FLASH2{{code language="none"}}{{/code}}(%%)
943 -desc: electron bunch energy measured behind the undulator. Data is saved with 10 Hz - BUT (for computation reasons) only the energy of the FIRST bunch is recorded. The data is also available for (% style="color: rgb(94,108,132);" %)extraction and septum  in the beginning of FLASH2
944 -
945 -(% style="letter-spacing: 0.0px;" %)units: (% class="twikiNewLink" %)MeV
946 -
947 -\\
948 -
949 -(% style="color: rgb(94,108,132);font-weight: 600;letter-spacing: 0.0px;" %)
950 -undulator settings
951 -
952 952  {{code language="none"}}/FL2/Electron Diagnostic/Undulator setting/set wavelength{{/code}}
953 953  //always saved (PBD2)//
954 954  DOOCS prop : {{code language="none"}}TTF2.FEEDBACK/FL2.WAVELENGTHCONTROL/FLASH2/WAVELENGTH{{/code}}