Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 5.1
edited by sendels
on 2019/05/15 16:18
Change comment: There is no comment for this version
To version 7.1
edited by sendels
on 2019/05/16 08:43
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -146,13 +146,13 @@
146 146  )))|=(((
147 147  [[FLASH1 PBD>>doc:FLASHUSER.PBD DAQ]]
148 148  )))|=(((
149 -[[FLASH2 PBD>>url:http://hasfweb.desy.de/bin/view/Setup/FL2PhotDiagDataStream||shape="rect" class="twikiLink"]]
149 +[[FLASH2 PBD>>doc:FLASHUSER.FL2PhotDiag stream (status Feb 2017)]]
150 150  )))|=(((
151 -[[User DAQs>>url:http://hasfweb.desy.de/bin/view/Setup/DAQjddd||shape="rect" class="twikiLink"]]
151 +[[User DAQs>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).How to save data in the User DAQ.WebHome]]
152 152  )))|=(((
153 -[[User DAQs subsytems>>url:http://hasfweb.desy.de/bin/view/Setup/DAQExp2Description||shape="rect" class="twikiLink"]]
153 +[[User DAQs subsytems>>doc:FLASHUSER.Description of the Properties in the FLASH1_USER1, 2 and 3 for FLASH1 Users]]
154 154  )))|=(((
155 -[[HDF5 files>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main||shape="rect" class="twikiLink"]]
155 +[[HDF5 files>>doc:FLASHUSER.Information about the FLASH HDF5 structure]]
156 156  )))
157 157  
158 158  == nearOnline ==
... ... @@ -159,17 +159,17 @@
159 159  
160 160  ==== .raw files ====
161 161  
162 -The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>url:http://hasfweb.desy.de/bin/view/Setup/DAQRootGui#How_to_Use_the_FLASH_DAQ_Data_GU||shape="rect" class="twikiAnchorLink"]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible [[to read with Matlab from the files directly>>url:http://hasfweb.desy.de/bin/view/Setup/DAQMatlabAccessExamples||shape="rect" class="twikiLink"]].
162 +The .raw files are only accessible from certain computers within the DESY network and in generell it is not recommended to use them directly. For checking the file content and doing simple analysis, e.g. histogram, line outs, we provide the [[DAQdataGUI>>doc:FLASHUSER.How to get the data out of the DAQ]]. If the user already have very precise knowledge of the desired parameters and it's types it is possible [[to read with Matlab from the files directly>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).For experts\: many ways to get Data from the DAQ.How to Access the DAQ Using Matlab.WebHome]].
163 163  
164 164  FLASH's DAQ .raw files are saved locally and with a time delay of a few minutes backupped via tape ( [[dCache~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Managing+data+taken+before+April+2015||shape="rect"]]) by the DESY central IT.
165 165  
166 166  ==== HDF5 files ====
167 167  
168 -As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline HDF5 files (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>url:http://hasfweb.desy.de/bin/view/Setup/BeamtimeDaqAccess||shape="rect" class="twikiLink"]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>url:http://hasfweb.desy.de/bin/view/Setup/BeamtimeDaqAccess||shape="rect" class="twikiLink"]] library is written in Python, but a Matlab wrapper is available.
168 +As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline HDF5 files (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>doc:FLASHUSER.Beamtime DAQ Access]] library is written in Python, but a Matlab wrapper is available.
169 169  
170 170  == Offline ==
171 171  
172 -The offline HDF5 files containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>url:http://hasfweb.desy.de/bin/view/Setup/HDF5main#Samples_how_to_read_HDF5_with_Ma||shape="rect" class="twikiAnchorLink"]] or [[Python>>url:http://hasfweb.desy.de/bin/view/Setup/FlashH5||shape="rect" class="twikiLink"]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
172 +The offline HDF5 files containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>doc:FLASHUSER.Information about the FLASH HDF5 structure]] or [[Python>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
173 173  
174 174  The spectrum scale file system contains the [[substructure~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
175 175