Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 9.1
edited by sendels
on 2019/06/25 15:06
Change comment: Saved attachments and added confluence links instead of links to hasyweb
To version 5.1
edited by sendels
on 2019/05/15 16:18
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -32,7 +32,7 @@
32 32  
33 33  == Overview ==
34 34  
35 -[[image:attach:overview_flash_control_and_daq.png]]
35 +[[image:url:http://hasfweb.desy.de/pub/Setup/Index/overview_flash_control_and_daq.png||alt="overview_flash_control_and_daq.png" width="748" height="823"]]
36 36  
37 37  The FLASH accelerator, beamlines and experiments are using the Distributed Object Oriented Control System ( [[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Beamline and experiment devices are mainly operated via graphical user interfaces created with the Java DOOCS Data Display ( [[jddd~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://doocs-web.desy.de/access.html||shape="rect"]]) by the FLASH team or the users themself. Further it is provided a [[DOOCS client API~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=DOOCS%20Wiki||shape="rect"]] in Java, Python, Matlab, C or C++ to the users to write own software for data access, online data or device control. User-provided devices can be run in parallel or can be implemented in DOOCS. For each beamline and accelerator is a dedicated [[electronic logbook~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/ttf_apps/showElogs.jsp||shape="rect"]] available in which jddd and other software can print directly.
38 38  
... ... @@ -40,7 +40,7 @@
40 40  
41 41  == Things to consider before the beamtime ==
42 42  
43 -In consultation with the user's local contact the user should check our [[beamtime preparation checklist>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Before beamtime Checklist.WebHome]] and fill in the [[beamtime preparation questionnaire>>attach:Before_beamtime_questionnaire.pdf]].
43 +In consultation with the user's local contact the user should check our [[beamtime preparation checklist>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Before beamtime Checklist.WebHome]] and fill in the [[beamtime preparation questionnaire>>url:http://hasfweb.desy.de/pub/Setup/ITManuals/Before_beamtime_questionnaire.pdf||shape="rect"]].
44 44  
45 45  == Beamline control and photon diagnostics ==
46 46  
... ... @@ -59,20 +59,10 @@
59 59  
60 60  == IT infrastructure ==
61 61  
62 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
62 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>url:http://hasfweb.desy.de/bin/view/Setup/ComputingResourcesUserGuide||shape="rect" class="twikiLink"]] which are used with functional accounts and they have access to the [[beamline files-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file-system~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]].
63 63  
64 -[[image:attach:it-infrastructure2.png||height="400"]]
64 +[[image:url:http://hasfweb.desy.de/pub/Setup/Index/it-infrastructure2.png||alt="it-infrastructure2.png" width="800" height="393"]]
65 65  
66 -\\
67 -
68 -
69 -
70 -{{view-file att--filename="run_jnlp.jsp" height="250"/}}
71 -
72 -\\
73 -
74 -\\
75 -
76 76  The FLASH control & DAQ system supports [[several devices>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).DOOCS Device Support.WebHome]] and at each beamline are [[doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).MTCA ADCs.WebHome]] available. With the MTCA technology it is possible to [[synchronize own devices in respect to the FEL>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).Trigger Howto.WebHome]].
77 77  If the user is using his or her own DAQ system it is further possible to [[receive the trainId via network>>doc:FLASHUSER.TrainID]] for the purpose of synchronization.
78 78  
... ... @@ -80,7 +80,7 @@
80 80  
81 81  (% class="wrapped" %)
82 82  |=(((
83 -[[local>>attach:jddd.jnlp]][[ machines>>attach:jddd.jnlp]]
73 +[[local machines>>url:http://hasfweb.desy.de/bin/view/Setup/ComputingResourcesUserGuide||shape="rect" class="twikiLink"]]
84 84  )))|=(((
85 85  (% class="content-wrapper" %)
86 86  (((
... ... @@ -87,9 +87,9 @@
87 87  [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]]
88 88  )))
89 89  )))|=(((
90 -[[doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).MTCA ADCs.WebHome]]
80 +[[MTCA ADCs>>url:http://hasfweb.desy.de/bin/view/Setup/MtcaAdc||shape="rect" class="twikiLink"]]
91 91  )))|=(((
92 -[[supported devices>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).DOOCS Device Support.WebHome]]
82 +[[supported devices>>url:http://hasfweb.desy.de/bin/view/Setup/DOOCS_supported_devices||shape="rect" class="twikiLink"]]
93 93  )))|=(((
94 94  (% class="content-wrapper" %)
95 95  (((
... ... @@ -96,9 +96,9 @@
96 96  [[supported cameras~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=Camera%20server||shape="rect"]]
97 97  )))
98 98  )))|=(((
99 -[[trigger via MTCA>>doc:FLASHUSER.Data Acquisition and controls.Controls (DOOCS, jDDD,\.\.\.).Trigger Howto.WebHome]]
89 +[[trigger via MTCA>>url:http://hasfweb.desy.de/bin/view/Setup/TriggerHowto||shape="rect" class="twikiLink"]]
100 100  )))|=(((
101 -[[trainId via network>>doc:FLASHUSER.TrainID]]
91 +[[trainId via network>>url:http://hasfweb.desy.de/bin/view/Setup/TrainID||shape="rect" class="twikiLink"]]
102 102  )))
103 103  
104 104  \\
... ... @@ -105,11 +105,9 @@
105 105  
106 106  == Data access ==
107 107  
108 -\\
98 +[[image:url:http://hasfweb.desy.de/pub/Setup/Index/timescale_dataAccess.png||alt="timescale_dataAccess.png" width="1000" height="539"]]
109 109  
110 -[[image:attach:timescale_dataAccess.png||height="400"]]
111 -
112 -At the Free-electron Laser Hamburg ( [[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ( [[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS server and via an API ( [[ONLINE>>doc:FLASHUSER.Online access to DOOCS properties]]) it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
100 +At the Free-electron Laser Hamburg ( [[FLASH~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://flash.desy.de/||shape="rect"]]) we use the Distributed Object Oriented Control System ( [[DOOCS~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://tesla.desy.de/doocs/doocs.html||shape="rect"]]). Devices are implemented via DOOCS server and via an API ( [[ONLINE>>url:http://hasfweb.desy.de/bin/view/Setup/OnlineAPIs||shape="rect" class="twikiLink"]]) it is possible to request data directly from the DOOCS server by knowing the DOOCS address.
113 113  As correlations of different physical properties are often required all data at FLASH are indexed by the [[trainID>>doc:FLASHUSER.TrainID]], which identify each of FLASH's pulse trains. The during a beamtime recorded data are stored via a Data Acquisition System ( [[DAQ>>doc:FLASHUSER.Information about the FLASH DAQ system]]) which sort all events from the individual DOOCS server by trainID. When requested [[HDF files>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] are created after the beamtime which includes the important data from [[the accelerator and its diagnostic>>doc:FLASHUSER.PBD DAQ]] as well as the data created by the [[users>>doc:FLASHUSER.Description of the Properties in the FLASH1_USER1, 2 and 3 for FLASH1 Users]]. This time scale we define as {{code language="none"}}offline{{/code}} as the HDF files are converted after the beamtime is over. For synchronous data during an experiment it is possible to create shorter HDF slices via a {{code language="none"}}nearOnline{{/code}} converter within a few minutes. For working with this partially incomplete HDF slices we provide an API called [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]]. Reading synchronous data via an {{code language="none"}}online{{/code}} API is possible via a configurable DAQ middle layer server, the [[DAQmonitor>>doc:AIDA2020ESST.DAQ Monitor]], which feeds the correlated data back in the control system while it provides a ring buffer with 32 events in size.
114 114  
115 115  \\
... ... @@ -152,19 +152,19 @@
152 152  
153 153  \\
154 154  
155 -(% class="wrapped relative-table" %)
143 +(% class="wrapped" %)
156 156  |=(((
157 -[[DAQ introductions>>attach:Operator_training_DAQ_May_2016.pdf]]
145 +[[DAQ introduction>>url:http://hasfweb.desy.de/pub/Setup/DAQmain/Operator_training_DAQ_May_2016.pdf||shape="rect"]]
158 158  )))|=(((
159 159  [[FLASH1 PBD>>doc:FLASHUSER.PBD DAQ]]
160 160  )))|=(((
161 -[[FLASH2 PBD>>doc:FLASHUSER.FL2PhotDiag stream (status Feb 2017)]]
149 +[[FLASH2 PBD>>url:http://hasfweb.desy.de/bin/view/Setup/FL2PhotDiagDataStream||shape="rect" class="twikiLink"]]
162 162  )))|=(((
163 -[[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
151 +[[User DAQs>>url:http://hasfweb.desy.de/bin/view/Setup/DAQjddd||shape="rect" class="twikiLink"]]
164 164  )))|=(((
165 -[[User DAQs subsytems>>doc:FLASHUSER.Description of the Properties in the FLASH1_USER1, 2 and 3 for FLASH1 Users]]
153 +[[User DAQs subsytems>>url:http://hasfweb.desy.de/bin/view/Setup/DAQExp2Description||shape="rect" class="twikiLink"]]
166 166  )))|=(((
167 -[[HDF5 files>>doc:FLASHUSER.Information about the FLASH HDF5 structure]]
155 +[[HDF5 files>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main||shape="rect" class="twikiLink"]]
168 168  )))
169 169  
170 170  == nearOnline ==
... ... @@ -171,17 +171,17 @@
171 171  
172 172  ==== .raw files ====
173 173  
174 -The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>doc:FLASHUSER.How to get the data out of the DAQ]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible [[to read with Matlab from the files directly>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.How to Access the DAQ Using Matlab.WebHome]].
162 +The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>url:http://hasfweb.desy.de/bin/view/Setup/DAQRootGui#How_to_Use_the_FLASH_DAQ_Data_GU||shape="rect" class="twikiAnchorLink"]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible [[to read with Matlab from the files directly>>url:http://hasfweb.desy.de/bin/view/Setup/DAQMatlabAccessExamples||shape="rect" class="twikiLink"]].
175 175  
176 176  FLASH's DAQ .raw files are saved locally and with a time delay of a few minutes backupped via tape ( [[dCache~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Managing+data+taken+before+April+2015||shape="rect"]]) by the DESY central IT.
177 177  
178 178  ==== HDF5 files ====
179 179  
180 -As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline HDF5 files (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]] library is written in Python, but a Matlab wrapper is available.
168 +As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline HDF5 files (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>url:http://hasfweb.desy.de/bin/view/Setup/BeamtimeDaqAccess||shape="rect" class="twikiLink"]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>url:http://hasfweb.desy.de/bin/view/Setup/BeamtimeDaqAccess||shape="rect" class="twikiLink"]] library is written in Python, but a Matlab wrapper is available.
181 181  
182 182  == Offline ==
183 183  
184 -The offline HDF5 files containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] or [[Python>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
172 +The offline HDF5 files containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main#Samples_how_to_read_HDF5_with_Ma||shape="rect" class="twikiAnchorLink"]] or [[Python>>url:http://hasfweb.desy.de/bin/view/Setup/FlashH5||shape="rect" class="twikiLink"]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
185 185  
186 186  The spectrum scale file system contains the [[substructure~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
187 187  
... ... @@ -190,9 +190,3 @@
190 190  ----
191 191  
192 192  **//NOTE~://** Content below the STOPPUBLISH variable is not published. Use this space as a scrap area.
193 -
194 -\\
195 -
196 -\\
197 -
198 -\\