Changes for page DAQ and data analysis
Last modified by sndueste on 2025/02/05 14:56
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Author
-
... ... @@ -1,1 +1,1 @@ 1 -XWiki. sndueste1 +XWiki.erlandm - Content
-
... ... @@ -14,7 +14,7 @@ 14 14 15 15 [[image:attach:overview_flash_control_and_daq.png]] 16 16 17 -The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ( 17 +The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ([[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://doocs-web.desy.de/index.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ([[jddd~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://jddd.desy.de/||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a [[DOOCS client API~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/elog/showElogs.jsp||shape="rect"]] available in which jddd and other software can print directly. 18 18 19 19 The data acquisition system (DAQ) collects beamline and experimental data and writes them to hard disk. All data are stored along with a time stamp and, in case of train specific data, along with a (pulse) train ID. There are several ways to access beamline and experimental data stored in the DAQ during or after the beamtime be it programmatically via DAQ access libraries or after data conversion to HDF5 format. While the data transport to the user's home institution is possible, DESY hosts high power computing resources which can be used for analysis. 20 20 ... ... @@ -30,6 +30,8 @@ 30 30 31 31 Use existing jddd panel provided by FLASH or use the jddd editor to create a new panel according to the user's own needs. 32 32 33 +\\ 34 + 33 33 [[ ~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://ttfinfo2.desy.de/common/applications/jddd_xml/run_jnlp.jsp?run=true&file=/svn/FLASH/Hasylab/Photons_beamline_control_main.xml||shape="rect"]][[~[~[image:url:http://hasfweb.desy.de/pub/Setup/Index/jdddEditor.png~|~|alt="jdddEditor.png" width="125"~]~]>>url:https://jddd.desy.de/||shape="rect"]](% class="twikiNewLink" %)[[url:http://hasfweb.desy.de/bin/edit/Ttfinfo2/desy/de/common/applications/jddd/Jnlp?topicparent=Setup.Index;nowysiwyg=0||rel="nofollow" shape="rect"]] 34 34 35 35 \\ ... ... @@ -45,7 +45,7 @@ 45 45 46 46 == IT infrastructure == 47 47 48 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files -system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].50 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]. 49 49 50 50 [[image:attach:it-infrastructure2.png||height="400"]] 51 51 ... ... @@ -87,8 +87,8 @@ 87 87 88 88 [[image:attach:timescale_dataAccess.png||height="400"]] 89 89 90 -At the Free-electron Laser Hamburg ( ( [[ONLINE>>doc:FLASH.Onlineaccess toDOOCSroperties]])it is possible to request data directly from the DOOCS server by knowing the DOOCS address.91 -As correlations of different physical properties are often required all data at FLASH are indexed by the[[trainID>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse trains. The during a beamtime recorded data are stored via a Data Acquisition System (shorterHDF slices via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes.For working with this partially incomplete HDF slices we provide an API called [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Onlinedata analysis]]. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.92 +At the Free-electron Laser Hamburg ([[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS servers and via [[an API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] it is possible to request data directly from the DOOCS server by knowing the DOOCS address. 93 +As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASHUSER.FLASH1 PhotDiag stream]] as well as the data created by the [[users>>doc:FLASHUSER.Configuring the user DAQs]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size. 92 92 93 93 \\ 94 94 ... ... @@ -128,9 +128,9 @@ 128 128 129 129 Relevant machine data, e.g. pulse energy or arrival time, are are saved in the FLASH Photon diagnostics DAQ (PBD) while experiment related parameter are saved on demand in the FLASH [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]. In addition it is disk space availabe for devices and paramter outside the FLASH DAQ, which can be [[ synchronized via Ethernet connection>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]]. The for the beamtime provided storage space, the [[spectrum scale~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]] formerly known as ASAP3/GPFS, is access regulated via [[DOOR~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://door.desy.de/door/index.php||shape="rect"]] and only registered participants have access. 130 130 131 -The FLASH DAQ system records the data in binary .raw files. On request the data will also be available in the HDF5 format after a conversion during or after the beamtime. Incoming data is collected, sorted and saved into .raw files in chunks of 60 MB to 1 GB which corresponds to tens of seconds up to several minutes. The HDF5 files can be created nearOnline or Offline. In the nearOnline conversion every individual .raw files will be converted to a single HDF5 file to provide the fastest access possible. After the beamtime it is possible to get a HDF5 file per DAQ run which is very convenient as it contains the merged data of the User DAQ and the PBD DAQ.133 +The FLASH DAQ system records the data in binary .raw files. On request the data will also be available in the HDF5 format after a conversion during or after the beamtime. Incoming data is collected, sorted and saved into .raw files in chunks of 60 MB to 1 GB which corresponds to tens of seconds up to several minutes. The HDF5 files can be created nearOnline or Offline. In the nearOnline conversion every individual .raw files will be converted to a single HDF5 file to provide the fastest access possible. 132 132 133 -While the DOOCS addresses are rather cryptic the [[HDF5 file>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's Repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/CS/repos/pah/browse/src/camp/data/channel2HdfName.dat||shape="rect"]].135 +While the DOOCS addresses are rather cryptic the [[HDF5 file>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] is structured based on the actual location and function of the devices. A complete list is available in [[DESY's software repository~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://stash.desy.de/projects/FSFL/repos/pydaq-test/raw/src/pydaqh5/dat/channel2HdfName.dat?at=refs%2Fheads%2Fmaster||shape="rect"]]. 134 134 135 135 \\ 136 136 ... ... @@ -155,19 +155,19 @@ 155 155 156 156 ==== .raw files ==== 157 157 158 -The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible [[to read withMatlab from thefilesdirectly>>doc:FLASHUSER.DataAcquisitionndcontrols.DataAccessat FLASH (DAQ, gpfs,\.\.\.).Offlinedataanalysis(DAQ).Forexperts\: manywayso getData from theDAQ.How to Accessthe DAQ Using Matlab.WebHome]].160 +The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible to read with [[Python>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=PythonDAQClientInterface||shape="rect"]], [[C/C++>>url:https://confluence.desy.de/display/MCS/FAQ%3A+How+to+access+DAQ+raw+data+files||shape="rect"]], or[[ Matlab>>url:https://stash.desy.de/projects/FSFL/repos/pydaq-test/raw/src/pydaqh5/dat/channel2HdfName.dat?at=refs%2Fheads%2Fmaster||shape="rect"]] from the files directly. 159 159 160 -FLASH's DAQ .raw files are saved locally and with a time delay of a few minutes backupped via tape ( 162 +FLASH's DAQ .raw files are saved locally and with a time delay of a few minutes backupped via tape ([[dCache~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Managing+data+taken+before+April+2015||shape="rect"]]) by the DESY central IT. 161 161 162 162 ==== HDF5 files ==== 163 163 164 -As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which arealsopresentinthenearOnline HDF5 files(e.g. missing/doubled events).Forthis we providewith[[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Online data analysis]]an API which handlesa lot of theoccurring difficultiesasknowing thefile or DAQ instances. The [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Onlinedata analysis]]library is writtenin Python, buta Matlabwrappervailable.166 +As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of. For working with these partially incomplete HDF slices we provide [[examples>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]] and an [[API>>url:https://gitlab.desy.de/christopher.passow/fdh-builder||shape="rect"]]. 165 165 166 166 [[Contents>>doc:||anchor="Contents"]] 167 167 168 168 == Offline == 169 169 170 -The offline HDF5filescontaining completeDAQ runs willbeput afterthebeamtimeon the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-runfiles can processed with common tools, e.g. with[[Matlab>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] or [[Python>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.172 +The nearOnline HDF slices are available offline too on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF files can processed with common tools, e.g. with Python, as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request. 171 171 172 172 The spectrum scale file system contains the [[substructure~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system. 173 173 ... ... @@ -177,11 +177,4 @@ 177 177 178 178 ---- 179 179 180 -**// 181 -//** 182 - 183 183 \\ 184 - 185 -\\ 186 - 187 -\\