Version 18.2 by sndueste on 2025/02/05 11:21
Show last authors
1
2
3 {{info title="Typical way to get to DAQ data: HDF5 files"}}
4 **Generally we provide the DAQ data as easy to access HDF5 files. For  more info goto  [[HDF5 file docu>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]. and [[doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).DEMO - Working with FLASH data.WebHome]]**
5
6 To get direct access to the saved .raw files there are several (sometimes not anymore supported) ways for experts:
7 {{/info}}
8
9
10 == Manuals Overview for experts ==
11
12
13
14 {{toc/}}
15
16 === DAQ Data Access Related Graphical User Interfaces ===
17
18 ==== How to Use the FLASH DAQ Data GUI? ====
19
20 [[How to Use the FLASH DAQ Data GUI?>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=FLASHDAQdataGUI||shape="rect"]] (DOOCS Wiki, Rybnikov, 2013)
21
22 The DAQ Data GUI is a general tool to select and visualise DAQ data. It also contains a DAQ File Manager. The File Manager is used to explore files stored on disk or tape and to retrieve data files from the tape archive too. The DAQ Data GUI can display data retrieved remotely from DOOCS servers. It also can access data stored locally in ROOT formatted .raw files, or files stored locally in HDF5 format.
23
24 Start the [[DAQdataGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI-experimental.jnlp||shape="rect"]] via Java Webstart.
25
26 **Use the DAQ data GUI locally for staged files:** login at flashlxuser1 (others may not work). One should just put in the run numbrt and define the directory ( with the field in the upper right) and then press get files. (No hooks for local mode or dates or something else ...!!)
27
28
29 ==== How to Use the DAQ Converter GUI? ====
30
31 The DAQ Converter GUI is used to convert DAQ data stored in the ROOT format, a self describing hierarchical binary format optimised for fast access, into ASCII formatted data for offline analysis. The GUI allows to select the files and channels from files to be converted. The GUI also has an interface to plug-in converters to write other data formats (Rybnikov et al., 2010).
32
33 Start the [[DAQexperiemntGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQexperimentGUI.jnlp||shape="rect"]] via Java Webstart.
34
35
36 ==== How to Stage Data From DAQ? ====
37
38 [[How to Stage Data From DAQ?>>url:https://ttfinfo.desy.de/FLASHWiki/attach/FLASHDAQfileMANAGER/How%20to%20stage%20data%20from%20DAQ.pdf||shape="rect"]] (Brenner, 2013)
39
40 This manual explains step by step how to use the DAQ File Manager GUI started via the [[DAQdataGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI-experimental.jnlp||shape="rect"]] to copy DAQ data files from tape to disk for offline analysis, if one does not want to use one of the APIs listed below.
41
42
43 ==== How to Discover Channels Recorded by the DAQ? ====
44
45 You can discover the channels recorded by the DAQ using the [[DAQdataGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI-experimental.jnlp||shape="rect"]] or the [[DAQexperiemntGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQexperimentGUI.jnlp||shape="rect"]], if you have access to the DESY network.
46
47 If you also have access to the .raw files directly, you can use the library {{code language="none"}}jdaq-use{{/code}} to create a searcheable text document. The library includes a command line application producing a list of channels discovered in a given .raw file or at a given point in time. Download this [[jdaq-use jar>>url:http://hasfweb.desy.de/pub/Setup/DAQRootGui/jdaq-use-0.0.4-j1.7.jar||shape="rect"]]. On a computer with access to the MCS-4 NFS filesystem the following command creates the list of channels as follows:
48
49
50 (% class="code" %)
51 (((
52 java -ea -jar jdaq-use-0.0.4-j1.7.jar FLASH1_EXP /daq_data/fel/EXP2 FLASH1_EXP_stream_2_run17666_file1_20170610T065636.1.raw
53 )))
54
55 In case you get no access, you could also use the [[DAQ Channel Info Database>>url:http://aps.anl.gov/~~nda/ilc/daq_startPage.php||shape="rect"]] as explained on the [[help pages>>url:http://aps.anl.gov/~~nda/ilc/daq_help.htm||shape="rect"]] by Ned Arnold of the [[Argonne Advanced Photon Source>>url:http://www.aps.anl.gov/||shape="rect"]].
56
57 TODO: Explain how to use "rpc_test" to discover DAQ data and channels.
58
59
60 === Application Program Interfaces to Access DAQ Data ===
61
62
63 * C++: [[Instructions how to use the TTF2 data processing framework>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=CDAQInterface||shape="rect"]] .
64 The C++ library sources are available from the DOOCS [[CVS repository>>url:http://doocs.desy.de/cgi-bin/viewvc.cgi/source/daq/DAQUser/||shape="rect"]] .
65
66 * Java
67
68 * Matlab:
69 ** [[How to Access the DAQ Using Matlab >>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.How to Access the DAQ Using Matlab.WebHome]] (no longer recommended)
70 ** [[MATLAB Interface to DAQ Data>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp#section-MATLABClientInterface-MATLABInterfaceToDAQData?page=MATLABClientInterface||shape="rect"]], the MCS-4 Wiki page
71
72 * Octave: [[DAQ to Octave Library for FLASH data analysis>>attach:Martins2009.pdf]]
73
74 * Python: [[PythonDAQClientInterface>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=PythonDAQClientInterface||shape="rect"]]
75
76 * [[DAQ Access From Jddd>>url:https://ttfinfo.desy.de/FLASHWiki/attach/How%20to%20use%20JDDD%20for%20accessing%20DAQ%20data/DaqInJddd.doc.pdf||shape="rect"]] (Rehlich, 2010)
77
78 === DAQ Run Control Graphical User Interfaces ===
79
80
81 ==== How to Start the DAQ Status GUI? ====
82
83 The DAQ Status GUI displays the current DAQ data flow from frontend through collectors, event builder to the dCache. The GUI can be started via Java Webstart for the [[Photon Beam DAQ>>url:http://ttfinfo2.desy.de/common/applications/jddd_xml/svn_jnlp.jsp?run=true&file=/svn/FLASH/DAQ/daq_status_pbd.xml||shape="rect"]] and the [[secondary experiment DAQ>>url:http://ttfinfo2.desy.de/common/applications/jddd_xml/svn_jnlp.jsp?run=true&file=/svn/FLASH/DAQ/daq_status_exp2.xml||shape="rect"]], respectively.
84
85
86 ==== How to Start the Run Control GUI? ====
87
88 The Run Control GUI is used to configure the DAQ run controller reflected in the run control data base. The Run Control GUI can be run on the central DOOCS user machines only. To start the GUI login with X-forwarding enabled to e.g. "flashlxuser1" and execute the command "RCGUIPBD" for the photon beam DAQ run control or the command "RCGUIEXP2" for the secondary experiment DAQ.
89
90
91 ==== How to Monitor the Experiment DAQ? ====
92
93 [[How to Monitor the Experiment DAQ?>>url:https://ttfinfo.desy.de/FLASHWiki/Wiki.jsp?page=How%20to%20monitor%20the%20Experiment%20DAQ||shape="rect"]] (DOOCS Wiki, NN, 2011)
94
95 This page explains, how to check, if the DAQ is running, i.e. if the DAQ is collecting and writing data. Possibly the page is not quite up to date ~-~- the uninitiated user is hardly able to follow the instructions given.
96
97
98 === General Introductions to the FLASH DAQ ===
99
100
101 * [[Access to FLASH DAQ Data>>url:http://fla.desy.de/localfsExplorer_read?currentPath=/afs/desy.de/group/fla/www/publications/2011/Rybnikov_FELseminar2011.pdf||shape="rect"]] (Rybnikov 2011)
102
103 * Data Acquisition System (Agababyan et al. in [[ICFA Beam Dynamics Newsletter, 47>>url:http://www-bd.fnal.gov/icfabd/Newsletter47.pdf||shape="rect"]] , pages 149-154, 2008)
104
105 * [[FLASH DAQ Data Management and Access Tools>>url:http://tesla.desy.de/doocs/doocs_papers/PCaPAC2010/frcoma02.pdf||shape="rect"]] (Rybnikov et al., 2013)
106
107 * More general literature on DOOCS and the DAQ is listed on the [[DOOCS Publications>>url:http://tesla.desy.de/doocs/doocs_papers/papers.html||shape="rect"]] page.
108
109 === Other Resources for DAQ and dCache ===
110
111 N.B. The general DESY info on its dCache do not directly apply to the FLASH DAQ dCache.
112
113
114 * [[The dCache Book>>url:http://www.dcache.org/manuals/||shape="rect"]]
115
116 * [[The Mass Storage Service (dCache/OSM)>>url:https://it.desy.de/e5/e47139/e193654/index_ger.html?preview=preview||shape="rect"]]
117
118 == Contact Info ==
119
120 for Questions / Problems:
121
122 * DAQ: [[Vladimir_Rybnikov>>url:mailto://vladimir.rybnikov@desy.de||shape="rect"]] FLA Tel : 4846
123
124