Skip to Content
Last modified by sndueste on 2025-09-08 15:36
From version 27.1
edited by erlandm
on 2021-06-17 15:46
Change comment: There is no comment for this version
To version 25.1
edited by sndueste
on 2021-02-10 15:24
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.erlandm
1 +XWiki.sndueste
Content
... ... @@ -67,7 +67,7 @@
67 67  
68 68  ==== Offline DAQ monitor ====
69 69  
70 -This tool looks if a predefined selection of parameters is saved in the raw file. When a raw.-file reached its maximum size ( ~~ 1 GB) it is closed and a new file is opened. Once closed, the Offline DAQ monitor reads the file and checks if there are entries for the selected parameters. It shows the fraction of events containing data. For “fast” data (data written with 10 Hz and bunch ID synchronous like images, ADC traces) this should be close to 100%. For “slow” data (saved about every second e.g. pressures , temperatures...) this is about 10-20%. **However it is to note** that this program just check if there is **anything** saved. It does not check if it is the proper data.
70 +This tool looks if a predefined selection of parameters is saved in the raw file. When a raw.-file reached its maximum size ( ~~ 1 GB) it is closed and a new file is opened. Once closed, the Offline DAQ monitor reads the file and checks if there are entris for the selected parameters. It shows the fraction of events containing data. For “fast” data (data written with 10 Hz and bunch ID synchronous like images, ADC traces) this should be close to 100%. For “slow” data (saved about every second e.g. pressures , temperatures...) this is about 10-20%. **However it is to note** that this program just check if there is **anything** saved. It does not check if it is the proper data.
71 71  
72 72  \\
73 73  
... ... @@ -79,9 +79,9 @@
79 79  
80 80  \\
81 81  
82 -The DAQ Data GUI is a general tool to select and visualize data that was saved in the DAQ . It has in addition simple analysis options like histogram, mean, min or max values as function of time etc. Also correlations between different parameters can be analyzed in the tool. More details can be found [[How to Use the FLASH DAQ Data GUI?~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]] and [[ here (DAQdataGUI link collection) >>doc:FLASH.How to get the data out of the DAQ]]
82 +The DAQ Data GUI is a general tool to select and visualise data that was saved in the DAQ . It has in addition simple analysis options like histogram, mean, min or max values as function of time etc. Also correlations between different parameters can be analyzed in the tool. More details can be found [[How to Use the FLASH DAQ Data GUI?~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]] and [[ here (DAQdataGUI link collection) >>doc:FLASH.How to get the data out of the DAQ]]
83 83  
84 -The tool (separate Java program) can be started [[DAQdataGUI~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI-experimental.jnlp||shape="rect"]] or in the DAQ control panel ( lower right) on your local (Desy) PC. In some cases the access via Windows does not work. An option which should always work is to log onto {{code language="none"}}flashlxuser1,flashlxuser2{{/code}} or {{code language="none"}}flashlxuser3{{/code}} and start {{code language="none"}}DAQdataGUI{{/code}} from the command line.
84 +The tool (separate Java program) can be started [[DAQdataGUI~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI-experimental.jnlp||shape="rect"]] or in the DAQ control panel ( lower right) on your local (Desy) PC. In some cases the access via Windows does not work. An option which should always work is to log onto {{code language="none"}}flashlxuser1{{/code}} or {{code language="none"}}flashlxuser2{{/code}} and start {{code language="none"}}DAQdataGUI{{/code}} from the command line.
85 85  
86 86  
87 87  \\To access the desired data one has to:
... ... @@ -99,15 +99,16 @@
99 99  The (% class="twikiNewLink" %)DAQdataGui(%%) is a powerful tool to visualize the saved data and to do very preliminary analysis. however for more detailed analysis the data has to be read into analysis programs ( matlab, python, Origin ...) here are different options one can use (what option is the best and how to set it up has to be discussed before hand with the FLASH DAQ experts ...)
100 100  
101 101  * The FLASH DAQ system records the data in binary " {{code language="none"}}.raw{{/code}} " files.
102 -** (% class="confluence-link" %)reading the .raw files [[directly via Python>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=PythonDAQClientInterface||shape="rect"]], or [[via Matlab >>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.How to Access the DAQ Using Matlab.WebHome]](%%)(no longer recommended) works, yet not for all data types equally well, and needs precise knowledge of the desired Parameters (useful to have a quick look / preliminary analysis on few parameters).
102 +** [[reading the data via Matlab >>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.How to Access the DAQ Using Matlab.WebHome]] works not for all data types equally well and needs precise knowledge of the desired Parameters (useful to have a quick look / preliminary analysis on few parameters).
103 103  ** Note that the raw data is only accessible from certain computers within the DESY network and thus not very handy for offline analysis later on ...
104 104  
105 105  \\
106 106  
107 107  * To ease the data analysis we IN ADDITION provide //ON REQUEST// the data from the FLASH DAQ as HDF5 format
108 -** ONLINE HDF5s: Intro to the [[HDF5 structure used at FLASH>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]. This contains also a list of the most used Parameters saved at FLASH for analysis - including the HDF5 names, DOOCS and DAQ names ...
109 -** [[Examples>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]] and an [[API>>url:https://gitlab.desy.de/christopher.passow/fdh-builder||shape="rect"]] to make use of the online HDF5s
110 -** [[Short examples>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] and [[user provided examples>>doc:FLASH.OUTDATED - Collection of user codes]] on how to read HDF5 in Matlab (no longer recommended).
108 +** ONLINE HDF5s: To make efficient use of the HDF5 files for analysis during the beamtime and afterwards we provide a [[very useful tool (Beamtime Daq Access)>>doc:FLASHUSER.Near-Online data analysis]]. Using this tool (working for Python and Matlab) one can get the data of parameters per run or for specific bunch ID intervals etc. without having to care from which subset of HDF5 files the data is read ...
109 +** Summary HDF5 files include complete runs. They will be put after the beamtime on the gpfs storage and are available from computers within DESY (for registered Beamtime participants) and also access
110 +** Intro to the [[HDF5 structure used at FLASH>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]. This contains also a list of the most used Parameters saved at FLASH for analysis - including the HDF5 names, DOOCS and DAQ names ...
111 +** [[Short examples>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] and [[user provided examples>>doc:FLASH.Collection of user codes]] on how to read HDF5 in Matlab
111 111  
112 112  The HDF5 files (online and summary) are saved in the [["gpfs" system ~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Architecture||shape="rect"]]. it can be accessed from [[Windows and Linux computers at DESY~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Mounting+the+storage+system+at+the+beamline||shape="rect"]] by the persons ( logins) which are registered for the beamtime in DOOR ( functional accounts
113 113  
... ... @@ -135,9 +135,9 @@
135 135  [[image:attach:RCGUI_2.jpg||height="400"]]
136 136  \\
137 137  
138 -Using the jddd DAQ control panel one can start and stop runs but in order to configure the DAQ which data to save one has to use a a separate *DAQ run control * java application. Since it needs write access to DOOCS internal file systems, which can not be made available elsewhere one has to start the system on our DOOCS control computers {{code language="none"}}flashlxuser1.desy.de, flashlxuser2.desy.de {{/code}} or {{code language="none"}}flashlxuser3.desy.de{{/code}}.
139 +Using the jddd DAQ control panel one can start and stop runs but in order to configure the DAQ which data to save one has to use a a separate *DAQ run control * java application. Since it needs write access to DOOCS internal file systems, which can not be made available elsewhere one has to start the system on our DOOCS control computers [[flashlxuser1.desy.de>>url:http://flashlxuser1.desy.de||shape="rect"]]{{code language="none"}}{{/code}} or [[flashlxuser2.desy.de>>url:http://flashlxuser2.desy.de||shape="rect"]]{{code language="none"}}{{/code}}.
139 139  Log in with the beamline account (bl1user, bl2user, ..., fl24user...) either via X-Win32 from windows or via ssh from Linux machines.
140 -Once logged in to {{code language="none"}}flashlxuser1,2,3{{/code}} the DAQ run control can be started. Depending on the DAQ you want to use the commands are:
141 +Once logged in to {{code language="none"}}flashlxuser1{{/code}} or {{code language="none"}}flashlxuser2{{/code}} the DAQ run control can be started. Depending on the DAQ you want to use the commands are:
141 141  
142 142  \\
143 143  
... ... @@ -182,7 +182,7 @@
182 182  
183 183  \\
184 184  
185 -Using the {{code language="none"}}Add{{/code}} and {{code language="none"}}Rem(ove){{/code}} buttons one can move the desired "subsystems" in the included side - meaning that they are saved. The "subsystems" contain typically several DOOCS parameters. E.g. {{code language="none"}}EXPERIMENT_MHZ_ADC_BL1{{/code}} contains all relevant DOOCS properties of the 4 MHZ ADC channels available at BL1. To get a detailed information about the saved parameters one can have a look to the "Show Properties in Subsystems" button . In addition, a list of all possible subsystems and their description can be found [[here>>doc:FLASHUSER.Configuring the user DAQs]].
186 +Using the {{code language="none"}}Add{{/code}} and {{code language="none"}}Rem(ove){{/code}} buttons one can move the desired "subsystems" in the included side - meaning that they are saved. The "subsystems" contain typically several DOOCS parameters. E.g. {{code language="none"}}EXPERIMENT_MHZ_ADC_BL1{{/code}} contains all relevant DOOCS properties of the 4 MHZ ADC channels available at BL1. To get a detailled information about the saved parameters one can have a look to the "Show Properties in Subsystems" button . In addition, a list of all possible subsystems and their description can be found [[here>>doc:FLASHUSER.Configuring the user DAQs]].
186 186  
187 187  \\
188 188  
... ... @@ -234,7 +234,7 @@
234 234  * 350 MB: Often a good compromise ...
235 235  * 100 MB: It is not preferred to have many "small" files for storing the data. For very small data rates this is an option to not wait too long till the next file is created.
236 236  
237 -To choose the desired file size one has to choose a run mode configuration in the RunControlGUI
238 +To choose the desired filesize one has to choose a run mode configuration in the RunControlGUI
238 238  
239 239  [[~[~[image:attach:image2020-1-17_13-39-38.png~|~|height="400"~]~]>>attach:image2020-1-17_13-39-38.png]]
240 240  
... ... @@ -272,7 +272,7 @@
272 272  
273 273  === No //START// or //STOP// buttons are visible in the DAQ control panel ===
274 274  
275 -~* The jddd GUI only shows the Start and stop button if the DAQ is in the "Idle" state. sometimes the DAQ "just" gets stuck in the stopping phase. Then stopping again or starting form the not "Idle" state can help. For this goto the "Advanced" tab of the jddd DAQ control . there the START and STOP buttons are always available ... independent of the state of the DAQ. If this does not help you may have to restart the DAQ ...
276 +~* The jddd GUI only shows the Start and stop bunnon if the DAQ is in the "Idle" state. sometimes the DAQ "just" gets stuck in the stopping phase. Then stopping again or starting form the not "Idle" state can help. For this goto the "Advanced" tab of the jddd DAQ control . there the START and STOP buttons are always available ... independent of the state of the DAQ. If this does not help you may have to restart the DAQ ...
276 276  
277 277  \\
278 278  
... ... @@ -314,7 +314,7 @@
314 314  \\
315 315  
316 316  * Generally there 3 ways to "restart" the DAQ
317 -** 1. The least interfering way is the "all to INITIALIZE" that tries to reset the DAQ processes ... helps for simple "hickups"
318 +** 1. The least interferring way is the "all to INITIALIZE" that tries to reset the DAQ processes ... helps for simple "hickups"
318 318  ** 2. The "re-start DAQ" really reboots the whole DAQ process. The settings still should be as before ... this is the "Usual" restart
319 319  ** 3. In very hard and severe cases there is the ultimate restart button: "re-start DAQ (no RCSHM)"
320 320  
... ... @@ -324,7 +324,7 @@
324 324  \\
325 325  
326 326  * if the DAQ gets stuck in the starting or stopping procedure one can try to initialize the DAQ or in the worst case restart it.
327 -** **to be able to restart the DAQ one has to run jddd on the {{code language="none"}}flashlxuser1,flashlxuser2{{/code}} or {{code language="none"}}flashlxuser3{{/code}} (up to now ( 2021) the restat on our consoles (cons0...12) is NOT reliably possible ) : log in at one of these machines with your beamline account (e.g. bl1user or fl24user). Start jddd with the {{code language="none"}}flash{{/code}} command. Click on "Photons" -> Experimental Hall "Photons main panel"**
328 +** **to be able to restart the DAQ one has to run jddd on the {{code language="none"}}flashlxuser1{{/code}} or {{code language="none"}}flashlxuser2{{/code}} (up to now ( 2020) the restat on our consoles (cons0...12) is NOT possible yet [[image:url:http://hasfweb.desy.de/pub/TWiki/SmiliesPlugin/frown.gif||alt="frown"]]) : log in at one of these machines with your beamline account (e.g. bl1user or fl24user). Start jddd with the {{code language="none"}}flash{{/code}} command. Click on "Photons" -> Experimental Hall "Photons main panel"**
328 328  ** The DAQ "expert" panel can be reached from the User DAQ control panel.
329 329  
330 330   [[image:attach:DAQ_expert.jpg||height="400"]]
... ... @@ -387,7 +387,7 @@
387 387  
388 388  OR one can do the hard way:
389 389  
390 -On flashlxuser1,2 or 3 start RCGUIFL1USER1 or appropriate to the DAQ you want to reconfigure. Select the tab "Run Modes". To then reload or re-init the run control database you need the expert menu entry:
391 +On flashlxuser1 or 2 start RCGUIFL1USER1 or appropriate to the DAQ you want to reconfigure. Select the tab "Run Modes". To then reload or re-init the run control database you need the expert menu entry:
391 391  
392 392  {{code language="none"}}Options -> "Show Expert Menu"{{/code}}
393 393