Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 3.1
edited by sendels
on 2019/05/15 16:07
Change comment: There is no comment for this version
To version 5.1
edited by sendels
on 2019/05/15 16:18
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -63,8 +63,8 @@
63 63  
64 64  [[image:url:http://hasfweb.desy.de/pub/Setup/Index/it-infrastructure2.png||alt="it-infrastructure2.png" width="800" height="393"]]
65 65  
66 -The FLASH control & DAQ system supports [[several devices>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).DOOCS Device Support.WebHome]] and at each beamline are [[doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).MTCA ADCs.WebHome]] available. With the MTCA technology it is possible to [[synchronize own devices in respect to the FEL>>url:http://hasfweb.desy.de/bin/view/Setup/TriggerHowto||shape="rect" class="twikiLink"]].
67 -If the user is using his or her own DAQ system it is further possible to [[receive the trainId via network>>url:http://hasfweb.desy.de/bin/view/Setup/TrainID||shape="rect" class="twikiLink"]] for the purpose of synchronization.
66 +The FLASH control & DAQ system supports [[several devices>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).DOOCS Device Support.WebHome]] and at each beamline are [[doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).MTCA ADCs.WebHome]] available. With the MTCA technology it is possible to [[synchronize own devices in respect to the FEL>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).Trigger Howto.WebHome]].
67 +If the user is using his or her own DAQ system it is further possible to [[receive the trainId via network>>doc:FLASHUSER.TrainID]] for the purpose of synchronization.
68 68  
69 69  \\
70 70  
... ... @@ -98,7 +98,7 @@
98 98  [[image:url:http://hasfweb.desy.de/pub/Setup/Index/timescale_dataAccess.png||alt="timescale_dataAccess.png" width="1000" height="539"]]
99 99  
100 100  At the Free-electron Laser Hamburg ( [[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ( [[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS server and via an API ( [[ONLINE>>url:http://hasfweb.desy.de/bin/view/Setup/OnlineAPIs||shape="rect" class="twikiLink"]]) it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
101 -As correlations of different physical properties are often required all data at FLASH are indexed by the [[trainID>>url:http://hasfweb.desy.de/bin/view/Setup/TrainID||shape="rect" class="twikiLink"]], which identify each of FLASH's pulse trains. The during a beamtime recorded data are stored via a Data Acquisition System ( [[DAQ>>url:http://hasfweb.desy.de/bin/view/Setup/DAQmain||shape="rect" class="twikiLink"]]) which sort all events from the individual DOOCS server by trainID. When requested [[HDF files>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main||shape="rect" class="twikiLink"]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>url:http://hasfweb.desy.de/bin/view/Setup/DAQPbdDescription||shape="rect" class="twikiLink"]] as well as the data created by the [[users>>url:http://hasfweb.desy.de/bin/view/Setup/DAQExp2Description||shape="rect" class="twikiLink"]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create shorter HDF slices via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. For working with this partially incomplete HDF slices we provide an API called [[BeamtimeDaqAccess>>url:http://hasfweb.desy.de/bin/view/Setup/BeamtimeDaqAccess||shape="rect" class="twikiLink"]]. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>url:http://hasfweb.desy.de/bin/view/Setup/ConfigureDAQmonitor||shape="rect" class="twikiLink"]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
101 +As correlations of different physical properties are often required all data at FLASH are indexed by the [[trainID>>doc:FLASHUSER.TrainID]], which identify each of FLASH's pulse trains. The during a beamtime recorded data are stored via a Data Acquisition System ( [[DAQ>>doc:FLASHUSER.Information about the FLASH DAQ system]]) which sort all events from the individual DOOCS server by trainID. When requested [[HDF files>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASHUSER.PBD DAQ]] as well as the data created by the [[users>>doc:FLASHUSER.Description of the Properties in the FLASH1_USER1, 2 and 3 for FLASH1 Users]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create shorter HDF slices via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. For working with this partially incomplete HDF slices we provide an API called [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]]. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:AIDA2020ESST.DAQ Monitor]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
102 102  
103 103  \\
104 104  
... ... @@ -126,17 +126,17 @@
126 126  
127 127  == Online ==
128 128  
129 -To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Matlab>>url:http://hasfweb.desy.de/bin/view/Setup/MatlabDOOCS||shape="rect" class="twikiLink"]], [[Python>>url:http://hasfweb.desy.de/bin/view/Setup/PythonDOOCS||shape="rect" class="twikiLink"]] and [[Labview>>url:http://hasfweb.desy.de/bin/view/Setup/LabviewDOOCS||shape="rect" class="twikiLink"]]. For accessing the control system [[DOOCS addresses>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main||shape="rect" class="twikiLink"]] are required.
129 +To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Matlab>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.MatlabDOOCS.WebHome]], [[Python>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.PythonDOOCS.WebHome]] and [[Labview>>doc:FLASHUSER.How to read and write DOOCS properties with LABVIEW]]. For accessing the control system [[DOOCS addresses>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] are required.
130 130  
131 -A collection (Matlab [ [[scan tool>>url:http://hasfweb.desy.de/bin/view/Setup/MatlabScanTools||shape="rect" class="twikiLink"]] , [[TOF & camera GUIs~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/FSFL/repos/matlab-guis/browse||shape="rect"]], [[others>>url:http://hasfweb.desy.de/bin/view/Setup/UserCodeCollection||shape="rect" class="twikiLink"]]] / [[Juypter notebooks~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/FSFL/repos/jupyter-notebook/browse||shape="rect"]]) for common use cases is available.
131 +A collection (Matlab [ [[scan tool>>doc:FLASHUSER.Matlab Scantools]] , [[TOF & camera GUIs~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/FSFL/repos/matlab-guis/browse||shape="rect"]], [[others>>doc:FLASHUSER.Collection of user codes]]] / [[Juypter notebooks~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/FSFL/repos/jupyter-notebook/browse||shape="rect"]]) for common use cases is available.
132 132  
133 133  == Data acquisition ==
134 134  
135 -Relevant machine data, e.g. pulse energy or arrival time, are are saved in the FLASH Photon diagnostics DAQ (PBD) while experiment related parameter are saved on demand in the FLASH [[User DAQs>>url:http://hasfweb.desy.de/bin/view/Setup/DAQjddd||shape="rect" class="twikiLink"]]. In addition it is disk space availabe for devices and paramter outside the FLASH DAQ, which can be [[synchronized via Ethernet connection>>url:http://hasfweb.desy.de/bin/view/Setup/TrainID||shape="rect" class="twikiLink"]]. The for the beamtime provided storage space, the [[spectrum scale~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]] formerly known as ASAP3/GPFS, is access regulated via [[DOOR~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://door.desy.de/door/index.php||shape="rect"]] and only registered participants have access.
135 +Relevant machine data, e.g. pulse energy or arrival time, are are saved in the FLASH Photon diagnostics DAQ (PBD) while experiment related parameter are saved on demand in the FLASH [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]. In addition it is disk space availabe for devices and paramter outside the FLASH DAQ, which can be [[ synchronized via Ethernet connection>>doc:FLASHUSER.TrainID]]. The for the beamtime provided storage space, the [[spectrum scale~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]] formerly known as ASAP3/GPFS, is access regulated via [[DOOR~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://door.desy.de/door/index.php||shape="rect"]] and only registered participants have access.
136 136  
137 137  The FLASH DAQ system records the data in binary .raw files. On request the data will also be available in the HDF5 format after a conversion during or after the beamtime. Incoming data is collected, sorted and saved into .raw files in chunks of 60 MB to 1 GB which corresponds to tens of seconds up to several minutes. The HDF5 files can be created nearOnline or Offline. In the nearOnline conversion every individual .raw files will be converted to a single HDF5 file to provide the fastest access possible. After the beamtime it is possible to get a HDF5 file per DAQ run which is very convenient as it contains the merged data of the User DAQ and the PBD DAQ.
138 138  
139 -While the DOOCS addresses are rather cryptic the [[HDF5 file>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main||shape="rect" class="twikiLink"]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
139 +While the DOOCS addresses are rather cryptic the [[HDF5 file>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].
140 140  
141 141  \\
142 142  
... ... @@ -144,7 +144,7 @@
144 144  |=(((
145 145  [[DAQ introduction>>url:http://hasfweb.desy.de/pub/Setup/DAQmain/Operator_training_DAQ_May_2016.pdf||shape="rect"]]
146 146  )))|=(((
147 -[[FLASH1 PBD>>url:http://hasfweb.desy.de/bin/view/Setup/DAQPbdDescription||shape="rect" class="twikiLink"]]
147 +[[FLASH1 PBD>>doc:FLASHUSER.PBD DAQ]]
148 148  )))|=(((
149 149  [[FLASH2 PBD>>url:http://hasfweb.desy.de/bin/view/Setup/FL2PhotDiagDataStream||shape="rect" class="twikiLink"]]
150 150  )))|=(((