Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 34.1
edited by sndueste
on 2025/02/05 11:21
Change comment: Renamed back-links.
To version 37.4
edited by sndueste
on 2025/02/05 14:55
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -13,7 +13,7 @@
13 13  
14 14  [[image:attach:overview_flash_control_and_daq.png]]
15 15  
16 -The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ([[DOOCS>>url:https://doocs-web.desy.de/index.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ([[jddd>>url:https://jddd.desy.de/||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a [[DOOCS client API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook>>url:https://ttfinfo.desy.de/elog/showElogs.jsp||shape="rect"]] available in which jddd and other software can print directly.
16 +The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ([[DOOCS>>url:https://doocs-web.desy.de/index.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ([[jddd>>https://doocs.desy.de/||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a DOOCS client API in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook>>url:https://ttfinfo.desy.de/elog/showElogs.jsp||shape="rect"]]  (only inside the DESY network) available in which jddd and other software can print directly.
17 17  
18 18  The data acquisition system (DAQ) collects beamline and experimental data and writes them to hard disk. All data are stored along with a time stamp and, in case of train specific data, along with a (pulse) train ID. There are several ways to access beamline and experimental data stored in the DAQ during or after the beamtime be it programmatically via DAQ access libraries or after data conversion to HDF5 format. While the data transport to the user's home institution is possible, DESY hosts high power computing resources which can be used for analysis.
19 19  
... ... @@ -44,7 +44,7 @@
44 44  
45 45  == IT infrastructure ==
46 46  
47 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster>>url:https://confluence.desy.de/display/MXW/Photon+Science+Resources||shape="rect"]]and [[JupyterHub on Maxwell>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]] (see also: (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]
47 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system>>https://docs.desy.de/asap3/||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster>>https://docs.desy.de/maxwell/||shape="rect"]]and [[JupyterHub on Maxwell>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]] (see also: (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system>>https://docs.desy.de/asap3/||shape="rect"]]
48 48  
49 49  [[image:attach:it-infrastructure2.png||height="400"]]
50 50  
... ... @@ -59,7 +59,7 @@
59 59  )))|=(((
60 60  (% class="content-wrapper" %)
61 61  (((
62 -[[Maxwell cluster>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]]
62 +[[Maxwell cluster>>https://docs.desy.de/maxwell/||shape="rect"]]
63 63  )))
64 64  )))|=(((
65 65  [[doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).MTCA ADCs.WebHome]]
... ... @@ -83,8 +83,8 @@
83 83  
84 84  [[image:attach:timescale_dataAccess.png||height="400"]]
85 85  
86 -At the Free-electron Laser Hamburg ([[FLASH>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS servers and via [[an API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
87 -As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from the accelerator and its diagnostic as well as the data created by the users. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
86 +At the Free-electron Laser Hamburg ([[FLASH>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS>>http://doocs.desy.de/||shape="rect"]]). Devices are implemented via DOOCS servers and via an API it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
87 +As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from the accelerator and its diagnostic as well as the data created by the users. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
88 88  
89 89  
90 90  (% class="wrapped" %)
... ... @@ -113,7 +113,7 @@
113 113  
114 114  == Online ==
115 115  
116 -To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Python>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.PythonDOOCS.WebHome]].  For accessing the control system [[DOOCS addresses>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are required.
116 +To monitor individual parameters online, e.g. ADCs or cameras, the use of jddd is recommended. For more complex tasks the users can use the DOOCS client API for [[Python>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Online data analysis.PythonDOOCS.WebHome]].  For accessing the control system [[DOOCS addresses>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are required.
117 117  
118 118  
119 119  [[Contents>>doc:||anchor="Contents"]]
... ... @@ -124,7 +124,7 @@
124 124  
125 125  The FLASH DAQ system records the data in binary .raw files. On request the data will also be available in the HDF5 format after a conversion during or after the beamtime. Incoming data is collected, sorted and saved into .raw files in chunks of 60 MB to 1 GB which corresponds to tens of seconds up to several minutes. The HDF5 files can be created nearOnline or Offline. In the nearOnline conversion every individual .raw files will be converted to a single HDF5 file to provide the fastest access possible.
126 126  
127 -While the DOOCS addresses are rather cryptic the [[HDF5 file>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's software repository>>url:https://stash.desy.de/projects/FSFL/repos/pydaq-test/raw/src/pydaqh5/dat/channel2HdfName.dat?at=refs%2Fheads%2Fmaster||shape="rect"]].
127 +While the DOOCS addresses are rather cryptic the [[HDF5 file>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's software repository>>url:https://stash.desy.de/projects/FSFL/repos/pydaq-test/raw/src/pydaqh5/dat/channel2HdfName.dat?at=refs%2Fheads%2Fmaster||shape="rect"]].
128 128  
129 129  
130 130  (% class="relative-table wrapped" %)
... ... @@ -133,7 +133,7 @@
133 133  )))|=(((
134 134  [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
135 135  )))|=(((
136 -[[HDF5 files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]
136 +[[HDF5 files>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]
137 137  )))
138 138  
139 139  [[Contents>>doc:||anchor="Contents"]]