Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 27.1
edited by sndueste
on 2023/02/13 13:07
Change comment: There is no comment for this version
To version 29.1
edited by sndueste
on 2024/03/07 10:25
Change comment: There is no comment for this version

Summary

Details

Page properties
Tags
... ... @@ -1,0 +1,1 @@
1 +favourite
Content
... ... @@ -14,7 +14,7 @@
14 14  
15 15  [[image:attach:overview_flash_control_and_daq.png]]
16 16  
17 -The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ([[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://doocs-web.desy.de/index.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ([[jddd~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://jddd.desy.de/||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a [[DOOCS client API~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/elog/showElogs.jsp||shape="rect"]] available in which jddd and other software can print directly.
17 +The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ([[DOOCS>>url:https://doocs-web.desy.de/index.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ([[jddd>>url:https://jddd.desy.de/||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a [[DOOCS client API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook>>url:https://ttfinfo.desy.de/elog/showElogs.jsp||shape="rect"]] available in which jddd and other software can print directly.
18 18  
19 19  The data acquisition system (DAQ) collects beamline and experimental data and writes them to hard disk. All data are stored along with a time stamp and, in case of train specific data, along with a (pulse) train ID. There are several ways to access beamline and experimental data stored in the DAQ during or after the beamtime be it programmatically via DAQ access libraries or after data conversion to HDF5 format. While the data transport to the user's home institution is possible, DESY hosts high power computing resources which can be used for analysis.
20 20  
... ... @@ -47,7 +47,7 @@
47 47  
48 48  == IT infrastructure ==
49 49  
50 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
50 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/MXW/Photon+Science+Resources||shape="rect"]]and [[JupyterHub on Maxwell>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]] (see also: (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
51 51  
52 52  [[image:attach:it-infrastructure2.png||height="400"]]
53 53  
... ... @@ -90,7 +90,7 @@
90 90  [[image:attach:timescale_dataAccess.png||height="400"]]
91 91  
92 92  At the Free-electron Laser Hamburg ([[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ([[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS servers and via [[an API>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
93 -As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASHUSER.FLASH1 PhotDiag stream]] as well as the data created by the [[users>>doc:FLASHUSER.Configuring the user DAQs]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
93 +As correlations of different physical properties are often required all data at FLASH are indexed by [[train IDs>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).How to read Train IDs at FLASH.WebHome]], which identify each of FLASH's pulse train. The during a beamtime recorded data are stored via a Data Acquisition System ([[DAQ>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]]) which sort all events from the individual DOOCS server by train ID. When requested [[HDF files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASH.FLASH1 PhotDiag stream]] as well as the data created by the [[users>>doc:FLASH.Configuring the user DAQs]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create HDF "slices" via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
94 94  
95 95  \\
96 96  
... ... @@ -140,13 +140,13 @@
140 140  |=(((
141 141  [[DAQ introductions>>attach:Operator_training_DAQ_May_2016.pdf]]
142 142  )))|=(((
143 -[[FLASH1 PBD>>doc:FLASHUSER.FLASH1 PhotDiag stream]]
143 +[[FLASH1 PBD>>doc:FLASH.FLASH1 PhotDiag stream]]
144 144  )))|=(((
145 -[[FLASH2 PBD>>doc:FLASHUSER.FLASH2 PhotDiag stream]]
145 +[[FLASH2 PBD>>doc:FLASH.FLASH2 PhotDiag stream]]
146 146  )))|=(((
147 147  [[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
148 148  )))|=(((
149 -[[User DAQs subsytems>>doc:FLASHUSER.Configuring the user DAQs]]
149 +[[User DAQs subsytems>>doc:FLASH.Configuring the user DAQs]]
150 150  )))|=(((
151 151  [[HDF5 files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]
152 152  )))
... ... @@ -169,7 +169,7 @@
169 169  
170 170  == Offline ==
171 171  
172 -The nearOnline HDF slices are available offline too on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF files can processed with common tools, e.g. with Python, as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
172 +The nearOnline HDF slices are available offline too on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants (see (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF files can processed with common tools, e.g. with Python, as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
173 173  
174 174  The spectrum scale file system contains the [[substructure~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
175 175  
Before beamtime questionnaire.pdf
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +276.0 KB
Content
Operator_training_DAQ_May_2016.pdf
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +3.0 MB
Content
image2023-2-13_13-4-52.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +105.7 KB
Content
image2023-2-13_13-5-51.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +183.5 KB
Content
it-infrastructure2.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +82.9 KB
Content
jddd.jnlp
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +1.9 KB
Content
overview_flash_control_and_daq.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +134.2 KB
Content
run_jnlp.jsp
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +1.7 KB
Content
... ... @@ -1,0 +1,76 @@
1 +
2 +
3 +<?xml version="1.0" encoding="utf-8"?>
4 +<!-- JNLP File for Administration Tool
5 + example call: dyn_jnlp.jsp?file=SINGENERATOR.xml[&run=true] -->
6 +<jnlp spec="1.0+" codebase="http://ttfinfo2.desy.de/common/applications" href="">
7 +
8 + <information>
9 + <title>jddd-run application</title>
10 + <vendor>DESY MCS4</vendor>
11 + <homepage href='http://doocs.desy.de/'/>
12 + <description>jddd Runtime Engine</description>
13 + <description kind="short">jddd Runtime Engine</description>
14 + <icon href='/images/jddd_run.gif'/>
15 + <offline-allowed/>
16 + </information>
17 +
18 + <security>
19 + <all-permissions/>
20 + </security>
21 +
22 + <resources>
23 + <j2se version='1.8+' initial-heap-size='256m' max-heap-size ='1024m' java-vm-args="-XX:+IgnoreUnrecognizedVMOptions --add-modules=java.xml.bind -Dfile.encoding=UTF-8"/>
24 + <jar href="jddd.jar"/>
25 + <jar href="jdoocs.jar"/>
26 + <jar href="jdaq.jar"/>
27 + <jar href="tine.jar"/>
28 + <extension href="http://ttfinfo2.desy.de/common/applications/jdddLibs.jnlp" name="jdddLibs"/>
29 + </resources>
30 +
31 + <application-desc main-class='editor.jddd'>
32 +
33 +
34 +
35 + <argument>-run</argument>
36 +
37 +
38 +
39 +
40 +
41 +
42 +
43 +
44 + <argument>-file</argument>
45 + <argument>/svn/FLASH/Hasylab/Photons_beamline_control_main.xml</argument>
46 + <argument>-repository</argument>
47 + <argument>https://doocssvn/svn/jddd</argument>
48 +
49 +
50 +
51 +
52 +
53 +
54 +
55 +
56 +
57 + <argument>xxx</argument>
58 + <argument>xxx</argument>
59 +
60 +
61 +
62 + <argument>-enshost</argument>
63 + <argument>ldap://ldapenssvr3:ldap://ldapenssvr2:ldap://ldapenssvr1</argument>
64 +
65 +
66 +
67 +
68 +
69 + <argument>xxx</argument>
70 +
71 +
72 + </application-desc>
73 +
74 +
75 +</jnlp>
76 +
timescale_dataAccess.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +108.9 KB
Content
XWiki.XWikiComments[0]
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sendels
Comment
... ... @@ -1,0 +1,1 @@
1 +wrong link, doesn't work
Date
... ... @@ -1,0 +1,1 @@
1 +2019-08-08 08:56:48.431
XWiki.XWikiComments[1]
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sndueste
Comment
... ... @@ -1,0 +1,1 @@
1 +habs schon geändert ...
Date
... ... @@ -1,0 +1,1 @@
1 +2019-08-08 13:01:06.260
Reply To
... ... @@ -1,0 +1,1 @@
1 +0