Changes for page DAQ and data analysis

Last modified by sndueste on 2025/02/05 14:56
From version 37.3
edited by sndueste
on 2025/02/05 14:53
Change comment: There is no comment for this version
To version 38.1
edited by sndueste
on 2025/02/05 14:56
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -44,7 +44,7 @@
44 44  
45 45  == IT infrastructure ==
46 46  
47 -During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system>>https://docs.desy.de/asap3/||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster>>url:https://confluence.desy.de/display/MXW/Photon+Science+Resources||shape="rect"]]and [[JupyterHub on Maxwell>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]] (see also: (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system>>https://docs.desy.de/asap3/||shape="rect"]]
47 +During a beamtime at FLASH we have two IT infrastructures each with different purpose. In the FLASH halls you have [[local machines>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.ComputingResourcesUserGuide.WebHome]] which are used with functional accounts and they have access to the [[beamline files system>>https://docs.desy.de/asap3/||shape="rect"]] for your current experiment. For more demanding task we could also provide workstations which can be dedicated to a single user experiment. For offline and nearOnline analysis the [[Maxwell cluster>>https://docs.desy.de/maxwell/||shape="rect"]]and [[JupyterHub on Maxwell>>https://docs.desy.de/maxwell/services/jupyterhub/||shape="rect"]] (see also: (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) for high performance computing is available. On the Maxwell cluster you have to work with personal accounts as this regulates data access to the [[core file system>>https://docs.desy.de/asap3/||shape="rect"]]
48 48  
49 49  [[image:attach:it-infrastructure2.png||height="400"]]
50 50  
... ... @@ -142,7 +142,7 @@
142 142  
143 143  The nearOnline HDF slices are available offline too on the spectrum scale file system. Access is granted from the within the DESY network for beamtime participants (see (% class="plugin_pagetree_children_span" %)[[ComputingResourcesUserGuide>>url:https://confluence.desy.de/display/FLASHUSER/ComputingResourcesUserGuide?src=contextnavpagetreemode||shape="rect"]](%%)) or from the outside via [[Gamma portal>>url:https://confluence.desy.de/display/ASAP3/The+Gamma+Portal||shape="rect"]]. HDF files can processed with common tools, e.g. with Python, as during the conversion a lot of discrepancies have been resolved. As .raw files are rarely used for analysis they are saved on the FLASH DAQ servers and will only be put on the spectrum scale file system on special request.
144 144  
145 -The spectrum scale file system contains the [[substructure>>url:https://confluence.desy.de/display/ASAP3/Directory+Structure||shape="rect"]]: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
145 +The spectrum scale file system contains the substructure: {{code language="none"}}raw{{/code}}, {{code language="none"}}processed{{/code}}, {{code language="none"}}shared{{/code}} and {{code language="none"}}scratch{{/code}}. The (% class="WYSIWYG_TT" %)raw (%%)folder contains the data recorded by the experiment (user data and HDF5) and will be set read only shortly after the beamtime is over. {{code language="none"}}Processed{{/code}} is typically used to store processed data and analysis software. Both folder are backupped by the spectrum scale file system.
146 146  
147 147  [[Contents>>doc:||anchor="Contents"]]
148 148