How to Access the DAQ Using Matlab

This page collects and comments pragmatic example Matlab scripts to access data stored on the FLASH DAQ. The [[main documentation of how to access the DAQ>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=MATLABClientInterface#section-MATLABClientInterface-MATLABInterfaceToDAQData||shape="rect"]] is maintained by MCS 4. Alternatives to access DAQ data are summarised in [[our how-to>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.WebHome]].

2

3

== Contents ==

* [[Requirements>>doc:||anchor="Requirements"]]

8

* [[User Guide>>doc:||anchor="User Guide"]]

9

* [[Single Channel Access>>doc:||anchor="Single Channel Access"]]

10

** [[Preliminaries>>doc:||anchor="Preliminaries"]]

11

** [[Examples>>doc:||anchor="Examples"]]

12

** [[Notes>>doc:||anchor="Notes"]]

13

** [[Resources>>doc:||anchor="Resources"]]

== Requirements ==

The scripts use the Mex functions {{code language="none"}}daq_read{{/code}} and {{code language="none"}}daq_read_svr{{/code}} provided by DESY's group MCS 4 to access the DAQ. These functions are provided for a specific version of MAC OS and GNU/Linux, here Ubuntu 16.04 64 bit, only. Also, support is limited to Matlab release R2016b.

20

21

For full functionality these Mex functions also need access to the MCS 4 file system exported via NFS. Access to that file system is limited to nodes run by MCS 4. Therefore, to use the full functionality you are limited to run your code on {{code language="none"}}flashlxuser1{{/code}} and {{code language="none"}}2{{/code}} or the workstations {{code language="none"}}hasfcons0{{/code}} and {{code language="none"}}10{{/code}}.

22

23

Limited functionality of the Mex functions is offered on other GNU/Linux, Ubuntu 16.04, machines on the DESY network, provided the package {{code language="none"}}doocs-matlab-extern-func{{/code}} is installed. In the Flash hall these are {{code language="none"}}hasfcons1{{/code}}, {{code language="none"}}2{{/code}}, and {{code language="none"}}4{{/code}} to {{code language="none"}}9{{/code}}, {{code language="none"}}hasfpg0{{/code}}, and {{code language="none"}}2{{/code}}, and {{code language="none"}}hasfumidaq{{/code}}.

24

25

In any case, make sure, {{code language="none"}}/local/lib{{/code}} is on your Matlab path.

== User Guide ==

Both Mex functions, {{code language="none"}}daq_read{{/code}} and {{code language="none"}}daq_read_svr{{/code}}, take as a parameter either a filename or a cell array of strings. The choice of parameter type determines, which range of functionality is provided. Neither is a subset of the other. Even though the cell array of strings seems a very natural start, be prepared to have to switch to XML-ish file definitions to get at the desired DAQ data (the XML-ish definitions of the requested data also can be created, or used by the [[DAQdataGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI.jnlp||shape="rect"]]).

32

33

The function **daq_read**{{code language="none"}}{{/code}} requires the DAQ data files in {{code language="none"}}.raw{{/code}} format to be available on the file system of the computer, where you run the script.

34

35

The function **daq_read_svr**{{code language="none"}}{{/code}} uses the DAQ data servers running on {{code language="none"}}flashlxuser1{{/code}}, {{code language="none"}}flashlxuser2{{/code}}, and {{code language="none"}}flashdaqsvr[2-6]{{/code}}. You preferably choose a server running locally, by setting the environment variable {{code language="none"}}TTF2_DAQ_SERVERS{{/code}} accordingly. So, on {{code language="none"}}flashlxuser1{{/code}} you would set for example in your Matlab code:

(% class="code" %)

(((

setenv('TTF2_DAQ_SERVERS', ['TTF2.DAQ/DAQ.SERVER4/DAQ.DATA.SVR/:' ...

40

'TTF2.DAQ/DAQ.SERVER3/DAQ.DATA.SVR/:TTF2.DAQ/DAQ.SERVER2/DAQ.DATA.SVR/:' ...

41

'TTF2.DAQ/DAQ.SERVER1/DAQ.DATA.SVR/]');

42

)))

43

44

Both Mex functions suffer from a memory leak. Therefore try and limit the number of calls to these functions in your code. Also, for efficiency it is advised to request a few longer intervals of data than many shorter chunks.

45

46

Often the specification of the run number is mandatory. To figure out what run number your data belong to, use the [[DAQdataGUI>>url:http://ttfinfo2.desy.de/common/applications/DAQdataGUI.jnlp||shape="rect"]]. Sometimes however, the specification of the run number causes the Mex function call to fail. You have to find out what works by trial and error.

47

48

Less frequently you need to specify the standard data location, ie. the path to the {{code language="none"}}.raw{{/code}} files on the DAQ server, called the //data directory//. The files' default location depends on the DAQ stream, alias experiment name, your data have been recorded with (see the table below). Sometimes you also need to specify the location of the run catalogue, which depends on the DAQ used to record the data, called the //run catalogue directory// (see table).

49

50

51

(% class="wrapped" %)

|=(((

Experiment Name

)))|=(((

Default Data Directory

)))|=(((

Catalogue Directory

)))

|(((

GMD_DATA

)))|(((

/daq_data/hasylab/GMD

)))|(((

/daq/ttf2/adm

)))

|(((

GMD_DATA_BACKUP

)))|(((

-

)))|(((

/daq/ttf2/admtemp

)))

|(((

PPLAS

)))|(((

/daq_data/hasylab/GMD

)))|(((

/daq/ttf2/adm

)))

|(((

FLASH1_EXP

)))|(((

/daq_data/fel/EXP2

)))|(((

/daq/ttf2/adm

)))

|(((

FLASH1_USER1

)))|(((

/daq_data/flash/FL1USER1

)))|(((

/daq/ttf2/adm

)))

|(((

FLASH1_USER2

)))|(((

/daq_data/fel/EXP2

)))|(((

/daq/ttf2/adm

)))

|(((

FL2PhotDiag

)))|(((

/daq_data/flash2/LINAC

)))|(((

/daq/ttf2/adm

)))

|(((

FLASH2_USER1

)))|(((

/daq_data/flash2/FL2USER1

)))|(((

/daq/ttf2/adm

)))

|(((

linac

)))|(((

/daq_data/flash/LINAC

)))|(((

/daq/ttf2/admtemp

)))

== Single Channel Access ==

126

127

Rational: During a beamtime you might want to confirm that data are recorded by the FLASH DAQ. In a situation like this, simple access to just one DAQ channel helps. This paragraph shows, how to retrieve a single channel of DAQ data for a given time period using the Matlab {{code language="none"}}DaqRead{{/code}} class.

128

129

The term //channel// is DAQ lingo and corresponds (almost) to what is called a //property// elsewhere in DOOCS. A mapping of channels to DOOCS properties and an explanation of their meaning is given [[here>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]].

130

131

132

=== Preliminaries ===

133

134

To use {{code language="none"}}DaqRead{{/code}} the requirements listed above must be met and the M files [[DaqRead.m>>url:https://stash.desy.de/projects/FSFL/repos/daqread/browse/src/DaqRead.m||shape="rect"]]{{code language="none"}}{{/code}} //and// [[DaqServerReader.m>>url:https://stash.desy.de/projects/FSFL/repos/daqread/browse/src/DaqServerReader.m||shape="rect"]]{{code language="none"}}{{/code}} must be on the Matlab search path. We suggest, you download the files from [[their repository>>url:https://stash.desy.de/projects/FSFL/repos/daqread||shape="rect"]] to the place where you keep your Matlab libraries, ie. the Matlab userpath (type {{code language="none"}}userpath{{/code}} into the Matlab command window to find out, where that is).

135

136

To check your set-up enter at the Matlab command window:

(% class="code" %)

(((

>> which DaqRead

>> which daq_read

)))

To get help with the use of {{code language="none"}}DaqRead{{/code}} type:

(% class="code" %)

(((

>> help DaqRead

)))

or more specifically for help on a method, {{code language="none"}}data{{/code}} in this example, enter:

(% class="code" %)

(((

>> help DaqRead.data

)))

=== Examples ===

To discover if the DAQ has recorded ADC data you could plot a single sampling point versus time or more precisely versus the pulse ID:

(% class="code" %)

(((

>> [trace, ids]= DaqRead().fl1user2().data('2017-07-25 17:40', 1, 'FLASH.FEL/ADC.SIS.BL2/EXP2.CH00');

168

>> plot(ids - ids(1), trace(2000,:), 'b.');

169

)))

170

171

Note, the {{code language="none"}}DaqRead{{/code}} object is configured to use the FLASH PBD DAQ by default. Therefore you have to specify the DAQ, the FLASH1 USER2 DAQ in this case. The returned matrix {{code language="none"}}trace{{/code}} contains the recorded ADC data, the sampling points in the columns and a row for each pulse.

172

173

The next example shows how to use the default of {{code language="none"}}DaqRead{{/code}} for the PBD DAQ. To see the SASE electron energy over 10 minutes recorded with the PBD DAQ you would enter for example:

(% class="code" %)

(((

>> sase= DaqRead().data('2016-03-28 10:59', 10, 'PBD.PHFLUX/BDA.ENERGYPULSE.FF');

>> imagesc(sase);

)))

Here, the energy pulse {{code language="none"}}sase{{/code}} is sliced to account for a repetition rate of 200 kHz; only at every 5th column a bunch is expected.

183

184

To investigate which data are recorded with a specific DAQ channel at a certain point in time try (intentionally without a semicolon at the end):

(% class="code" %)

(((

>> structs= DaqRead().structs('2016-03-28 10:59', 0.1, 'PBD.PHOTONWL.ML/WAVE_LENGTH')

190

)))

191

192

This request gives you just six seconds (0.1 minutes) of data as an array of structures. To then display just the spectrum use:

(% class="code" %)

(((

>> imagesc([structs.data]);

198

)))

199

200

The examples given above will cease to function when the {{code language="none"}}.raw{{/code}} files are deleted from disk.

=== Notes ===

The {{code language="none"}}DaqRead.data{{/code}} and {{code language="none"}}DaqRead.structs{{/code}} methods as a **side effect** write the XML-ish data definition to the {{code language="none"}}/tmp{{/code}} directory with the filename concatenated from user ID and process ID. The file is not deleted automatically, to allow for the analysis of failing calls to the Mex functions {{code language="none"}}daq_read{{/code}} or {{code language="none"}}daq_read_svr{{/code}}.

206

207

If you want to **reuse** a {{code language="none"}}DaqRead{{/code}} object or you don't like the cascading of messages, the previous examples could be rewritten using **single method invocations** per statement as:

(% class="code" %)

(((

>> reader= DaqRead; % create a reader object

213

>> reader.local; % configure it to use local access, ie. daq_read

214

>> reader.pbd; % configure it explicitly to use the default PBD DAQ

215

>> wl= reader.structs('2016-03-28 10:59', 1, 'PBD.PHOTONWL.ML/WAVE_LENGTH');

216

>> sase= reader.data('2016-03-28 10:59', 1, 'PBD.PHFLUX/BDA.ENERGYPULSE.FF');

>> disp(wl);

>> % ...

)))

The calls of {{code language="none"}}DaqRead.data{{/code}} or {{code language="none"}}DaqRead.structs{{/code}} fail fairly frequently due to failures of the calls to the underlying Mex functions {{code language="none"}}daq_read{{/code}} and {{code language="none"}}daq_read_svr{{/code}} respectively. These functions return very limited error messages. Most of the error output goes to the **operating system console**, which can not be captured by Matlab. Therefore, if your calls error, consult the output on the operating system console on which you have started Matlab. If you start Matlab using a GUI, you don't get to see the error output at all.

=== Resources ===

* [[Source code repository>>url:https://stash.desy.de/projects/FSFL/repos/daqread||shape="rect"]] of the Matlab class {{code language="none"}}DaqRead{{/code}} to read single channels of the Flash DAQ

229

* [[DaqServerReader.m>>url:https://ttfinfo.desy.de/DOOCSWiki/Wiki.jsp?page=MATLABClientInterface#section-MATLABClientInterface-MATLABInterfaceToDAQData||shape="rect"]] : Wiki detailing the use of Mex functions {{code language="none"}}daq_read{{/code}} and {{code language="none"}}daq_read_svr{{/code}}.

230

231

232

and here the list with server names:

(% class="wrapped" %)

237

|=(((

238