Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 23.1
edited by sndueste
on 2020/03/26 15:51
Change comment: There is no comment for this version
To version 22.1
edited by sndueste
on 2020/01/07 13:37
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -161,13 +161,13 @@
161 161  
162 162  ==== HDF5 files ====
163 163  
164 -As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline [[HDF5 files>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Online data analysis]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Online data analysis]] library is written in Python, but a Matlab wrapper is available.
164 +As the .raw files are highly optimized for writing speed there are some issues which have to be taken care of which are also present in the nearOnline HDF5 files (e.g. missing/doubled events). For this we provide with [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Online data analysis]] an API which handles a lot of the occurring difficulties as knowing the file or DAQ instances. The [[BeamtimeDaqAccess>>doc:FLASHUSER.Near-Online data analysis]] library is written in Python, but a Matlab wrapper is available.
165 165  
166 166  [[Contents>>doc:||anchor="Contents"]]
167 167  
168 168  == Offline ==
169 169  
170 -The offline [[HDF5 files >>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] or [[Python>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
170 +The offline HDF5 files containing complete DAQ runs will be put after the beamtime on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants or from the outside via [[Gamma portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF-by-run files can processed with common tools, e.g. with [[Matlab>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] or [[Python>>doc:FLASHUSER.FLASHH5 for easier access of FLASH's HDF5 files]], as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
171 171  
172 172  The spectrum scale file system contains the [[substructure~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
173 173