Changes for page Data Access at FLASH

Last modified by sndueste on 2025/02/05 11:22
From version 13.1
edited by sndueste
on 2025/02/05 11:18
Change comment: There is no comment for this version
To version 4.1
edited by sendels
on 2019/06/26 08:39
Change comment: added attachments

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.sendels
Content
... ... @@ -1,9 +1,11 @@
1 -[[image:attach:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.WebHome@timescale_dataAccess.png||height="400"]]
1 +\\
2 2  
3 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/DataAccessGeneralConcept/timescale_dataAccess.png~|~|alt="timescale_dataAccess.png"~]~]>>attach:FLASHUSER.User overview@timescale_dataAccess.png]]
3 3  
4 -At the Free-electron Laser Hamburg ([[FLASH>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS servers and via [[an API >>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]]it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
5 -As correlations of different physical properties are often required all data at FLASH are indexed by the [[train ID>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identifies each pulse train. The data recorded during a beamtime are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sorts all events from the individual DOOCS server by train ID. Requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created to analyze s(% id="cke_bm_2799S" style="display:none" %) (%%)ynchronous data during an experiment via an {{code language="none"}}HDF{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the DAQmonitor, which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
5 +At the Free-electron Laser Hamburg ( [[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ( [[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS server and via an API ( [[ONLINE>>doc:FLASHUSER.Online access to DOOCS properties]]) it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
6 +As correlations of different physical properties are often required all data at FLASH are indexed by the [[trainID>>doc:FLASHUSER.TrainID]], which identify each of FLASH's pulse trains. The during a beamtime recorded data are stored via a Data Acquisition System ( [[DAQ>>doc:FLASHUSER.Information about the FLASH DAQ system]]) which sort all events from the individual DOOCS server by trainID. When requested [[HDF files>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASHUSER.PBD DAQ]] as well as the data created by the [[users>>doc:FLASHUSER.Description of the Properties in the FLASH1_USER1, 2 and 3 for FLASH1 Users]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create shorter HDF slices via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. For working with this partially incomplete HDF slices we provide an API called [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]]. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Online data analysis.DAQmonitor.WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
6 6  
8 +\\
7 7  
8 8  (% class="wrapped" %)
9 9  |(((
... ... @@ -17,14 +17,20 @@
17 17  live up to 3s into the past
18 18  )))
19 19  |(((
22 +nearOnline
23 +)))|(((
24 +a few minutes
25 +)))
26 +|(((
20 20  offline
21 21  )))|(((
22 -a few minutes ... till after the beamtime
29 +after the beamtime
23 23  )))
24 24  
25 -[[image:attach:it-infrastructure.png||height="250"]]
32 +[[~[~[image:url:http://hasfweb.desy.de/pub/Setup/DataAccessGeneralConcept/it-infrastructure.png~|~|alt="it-infrastructure.png"~]~]>>attach:it-infrastructure.png]]
26 26  
27 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have local machines which are used with functional accounts and they have access to the [[beamline files-system>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline  analysis the [[Maxwell cluster>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file-system>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
34 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have local machines which are used with functional accounts and they have access to the [[beamline files-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
28 28  
36 +\\
29 29  
30 30  ----
it-infrastructure.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.XWikiGuest
Size
... ... @@ -1,1 +1,0 @@
1 -88.2 KB
Content