Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 34.1
edited by sndueste
on 2025/02/05 11:21
Change comment: Renamed back-links.
To version 30.1
edited by sndueste
on 2024/03/13 14:27
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -DAQ and data analysis
1 +DAQ and controls overview
Content
... ... @@ -1,4 +1,4 @@
1 -Before starting a beamtime at FLASH the users have to consider several options regarding the control and data acquisition system where the users are supported by [[the controls group>>url:http://photon-science.desy.de/facilities/flash/beamlines/daq_and_controls/index_eng.html||shape="rect"]].
1 +Before starting a beamtime at FLASH the users have to consider several options regarding the control and data acquisition system where the users are supported by [[the controls group~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|height="12" width="13"~]~]>>url:http://photon-science.desy.de/facilities/flash/beamlines/daq_and_controls/index_eng.html||shape="rect"]].
2 2  
3 3  == Contents ==
4 4  
... ... @@ -84,7 +84,7 @@
84 84  [[image:attach:timescale_dataAccess.png||height="400"]]
85 85  
86 86  At the Free-electron Laser Hamburg ([[FLASH>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS servers and via [[an API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
87 -As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from the accelerator and its diagnostic as well as the data created by the users. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
87 +As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASH.FLASH1 PhotDiag stream]] as well as the data created by the [[users>>doc:FLASH.Configuring the user DAQs]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
88 88  
89 89  
90 90  (% class="wrapped" %)
... ... @@ -113,14 +113,15 @@
113 113  
114 114  == Online ==
115 115  
116 -To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Python>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.PythonDOOCS.WebHome]].  For accessing the control system [[DOOCS addresses>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are required.
116 +To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Matlab>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.MatlabDOOCS.WebHome]], [[Python>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.PythonDOOCS.WebHome]] and [[Labview>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.LabviewDOOCS (no longer supported).WebHome]]. For accessing the control system [[DOOCS addresses>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are required.
117 117  
118 +A collection (Matlab [ [[TOF & camera GUIs>>url:https://stash.desy.de/projects/FSFL/repos/matlab-guis/browse||shape="rect"]]] / [[Juypter notebooks>>url:https://stash.desy.de/projects/FSFL/repos/jupyter-notebook/browse||shape="rect"]]) for common use cases is available.
118 118  
119 119  [[Contents>>doc:||anchor="Contents"]]
120 120  
121 121  == Data acquisition ==
122 122  
123 -Relevant machine data, e.g. pulse energy or arrival time, are are saved in the FLASH Photon diagnostics DAQ (PBD) while experiment related parameter are saved on demand in the FLASH [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]. In addition it is disk space availabe for devices and paramter outside the FLASH DAQ, which can be [[ synchronized via Ethernet connection>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]]. The for the beamtime provided storage space, the [[spectrum scale>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]] formerly known as ASAP3/GPFS, is access regulated via [[DOOR>>url:https://door.desy.de/door/index.php||shape="rect"]] and only registered participants have access.
124 +Relevant machine data, e.g. pulse energy or arrival time, are are saved in the FLASH Photon diagnostics DAQ (PBD) while experiment related parameter are saved on demand in the FLASH [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]. In addition it is disk space availabe for devices and paramter outside the FLASH DAQ, which can be [[ synchronized via Ethernet connection>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]]. The for the beamtime provided storage space, the [[spectrum scale>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]] formerly known as ASAP3/GPFS, is access regulated via [[DOOR>>url:https://door.desy.de/door/index.php||shape="rect"]] and only registered participants have access.
124 124  
125 125  The FLASH DAQ system records the data in binary .raw files. On request the data will also be available in the HDF5 format after a conversion during or after the beamtime. Incoming data is collected, sorted and saved into .raw files in chunks of 60 MB to 1 GB which corresponds to tens of seconds up to several minutes. The HDF5 files can be created nearOnline or Offline. In the nearOnline conversion every individual .raw files will be converted to a single HDF5 file to provide the fastest access possible.
126 126  
... ... @@ -131,13 +131,33 @@
131 131  |=(((
132 132  [[DAQ introductions>>attach:Operator_training_DAQ_May_2016.pdf]]
133 133  )))|=(((
134 -[[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
135 +[[FLASH1 PBD>>doc:FLASH.FLASH1 PhotDiag stream]]
135 135  )))|=(((
137 +[[FLASH2 PBD>>doc:FLASH.FLASH2 PhotDiag stream]]
138 +)))|=(((
139 +[[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
140 +)))|=(((
141 +[[User DAQs subsytems>>doc:FLASH.Configuring the user DAQs]]
142 +)))|=(((
136 136  [[HDF5 files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]
137 137  )))
138 138  
139 139  [[Contents>>doc:||anchor="Contents"]]
140 140  
148 +== nearOnline ==
149 +
150 +==== .raw files ====
151 +
152 +The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible to read with [[Python>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=PythonDAQClientInterface||shape="rect"]], [[C/C++>>url:https://confluence.desy.de/display/MCS/FAQ%3A+How+to+access+DAQ+raw+data+files||shape="rect"]], or[[ Matlab>>url:https://stash.desy.de/projects/FSFL/repos/pydaq-test/raw/src/pydaqh5/dat/channel2HdfName.dat?at=refs%2Fheads%2Fmaster||shape="rect"]] from the files directly.
153 +
154 +FLASH's DAQ .raw files are saved locally and with a time delay of a few minutes backupped via tape ([[dCache>>url:https://confluence.desy.de/display/ASAP3/Managing+data+taken+before+April+2015||shape="rect"]]) by the DESY central IT.
155 +
156 +==== HDF5 files ====
157 +
158 +As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of. For working with these partially incomplete HDF slices we provide [[examples>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]] and an [[API>>url:https://gitlab.desy.de/christopher.passow/fdh-builder||shape="rect"]].
159 +
160 +[[Contents>>doc:||anchor="Contents"]]
161 +
141 141  == Offline ==
142 142  
143 143  The nearOnline HDF slices are available offline too on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants (see (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) or from the outside via [[Gamma portal>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF files can processed with common tools, e.g. with Python, as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.