Last modified by sndueste on 2025/02/05 15:00
From version 14.1
edited by erlandm
on 2021/06/22 17:32
Change comment: There is no comment for this version
To version 12.1
edited by erlandm
on 2020/02/27 10:53
Change comment: Fixed link to attachment

Summary

Details

Page properties
Content
... ... @@ -4,7 +4,7 @@
4 4  
5 5  == Introduction ==
6 6  
7 -As explained in the [[doc:FLASH.User overview]] page the data recorded at FLASH is stored in an internal DAQ format (.raw) which is converted with a few minutes delay to easily accessible HDF5 ([[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]) files which are the basis for data analysis. These  files are stored on the [[ASAP3 File System>>url:https://confluence.desy.de/display/ASAP3/ASAP3++Data+Storage+for+PETRA+III||rel="nofollow" shape="rect"]] which is accessible from DESYs  [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Maxwell+for+Photon+Science||rel="nofollow" shape="rect"]] which is recommended to be used for the analysis by users. H5py and Pandas allow efficient access to the recorded data in the HDF5 files. In addition the HDF5 files are accessible from the beamline computers and via a web interface:  [[Gamma Portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://gamma-portal.desy.de||shape="rect"]]
7 +As explained in the [[doc:FLASH.User overview]] page the data recorded at FLASH is stored in an internal DAQ format (.raw) which is converted with a few minutes delay to easily accessible HDF5 ([[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]) files which are the basis for data analysis. These  files are stored on the [[ASAP3 File System>>url:https://confluence.desy.de/display/ASAP3/ASAP3++Data+Storage+for+PETRA+III||rel="nofollow" shape="rect"]] which is accessible from DESYs  [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Maxwell+for+Photon+Science||rel="nofollow" shape="rect"]] which is recommended to be used for the analysis by users. A library provided by FLASH ( [[BeamtimeDaqAccess>>url:https://stash.desy.de/projects/CS/repos/pah||rel="nofollow" shape="rect" class="external-link"]] ) allows efficient access to the recorded data in the HDF5 files. In addition the HDF5 files are accessible from the beamline computers and via a web interface:  [[Gamma Portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://gamma-portal.desy.de||shape="rect"]]
8 8  
9 9  \\
10 10  
... ... @@ -31,6 +31,10 @@
31 31  
32 32  The extensive documentation can be found at [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Photon+Science+on+Maxwell||rel="nofollow" shape="rect"]]
33 33  
34 -In particular we want to mention the option [[doc:MXW.JupyterHub on Maxwell]] , available [[doc:IS.Software]] , for more advanced computing demands [[Running Jobs on Maxwell.>>doc:IS.Running Jobs on Maxwell]]
34 +In particular we want to mention the option [[doc:IS.JupyterHub on Maxwell]] , available [[doc:IS.Software]] , for more advanced computing demands [[Running Jobs on Maxwell.>>doc:IS.Running Jobs on Maxwell]]
35 35  
36 +== Where to store data ==
37 +
38 +There are a number of data stores not linked to the ASAP3 file systems available to cover for storage needs unrelated to experiments or beamtimes. Our page "[[doc:FLASH.Where to store data]]" presents an overview of the storage resources and the [[unified access to disks on DESY's GNU/Linux and MS Windows>>url:https://confluence.desy.de/display/FLASH/Where+to+store+data?preview=%2F131370944%2F143223821%2FFLASH-Disk-Storage-Update.pdf||shape="rect"]] PCs.
39 +
36 36  \\