Last modified by sndueste on 2025/02/05 15:00
From version 12.1
edited by erlandm
on 2020/02/27 10:53
Change comment: Fixed link to attachment
To version 18.1
edited by sndueste
on 2025/02/05 11:21
Change comment: Renamed back-links.

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.erlandm
1 +XWiki.sndueste
Content
... ... @@ -1,12 +1,10 @@
1 1  = Brief Guide to Computing Resources at FLASH =
2 2  
3 -\\
4 4  
5 5  == Introduction ==
6 6  
7 -As explained in the [[doc:FLASH.User overview]] page the data recorded at FLASH is stored in an internal DAQ format (.raw) which is converted with a few minutes delay to easily accessible HDF5 ([[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]) files which are the basis for data analysis. These  files are stored on the [[ASAP3 File System>>url:https://confluence.desy.de/display/ASAP3/ASAP3++Data+Storage+for+PETRA+III||rel="nofollow" shape="rect"]] which is accessible from DESYs  [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Maxwell+for+Photon+Science||rel="nofollow" shape="rect"]] which is recommended to be used for the analysis by users. A library provided by FLASH ( [[BeamtimeDaqAccess>>url:https://stash.desy.de/projects/CS/repos/pah||rel="nofollow" shape="rect" class="external-link"]] ) allows efficient access to the recorded data in the HDF5 files. In addition the HDF5 files are accessible from the beamline computers and via a web interface:  [[Gamma Portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://gamma-portal.desy.de||shape="rect"]]
6 +As explained in the [[User overview>>doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.WebHome]] page the data recorded at FLASH is stored in an internal DAQ format (.raw) which is converted with a few minutes delay to easily accessible HDF5 ([[doc:FLASHUSER.Data Acquisition and controls.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]) files which are the basis for data analysis. These  files are stored on the [[ASAP3 File System>>url:https://confluence.desy.de/display/ASAP3/ASAP3++Data+Storage+for+PETRA+III||rel="nofollow" shape="rect"]] which is accessible from DESYs  [[Maxwell Cluster>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] which is recommended to be used for the analysis by users. H5py and Pandas allow efficient access to the recorded data in the HDF5 files. In addition the HDF5 files are accessible from the beamline computers and via a web interface:  [[Gamma Portal>>url:https://gamma-portal.desy.de||shape="rect"]]
8 8  
9 -\\
10 10  
11 11  == Access to the Maxwell cluster ==
12 12  
... ... @@ -15,26 +15,21 @@
15 15  {{info title="PSX account"}}
16 16  The scientific (internally called: PSX)  account is a "real" account that is needed to be able to log into DESY computers ( mainly used for Maxwell). This is different from the[[ DOOR>>url:https://door.desy.de/door/index.php||shape="rect"]] account that you need for administrative purposes to be registered for the experiment, do safety training etc.
17 17  
18 -To get a PSX account please talk to your local contact well before the beamtime. The local contact will take care of the paper work and send you more instructions.
16 +To get a PSX account please talk to your local contact well before the beamtime.
19 19  {{/info}}
20 20  
21 -* Scientific account (PSX) users instead go to **[[desy-ps-cpu.desy.de>>url:http://desy-ps-cpu.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}. To connect to the Maxwell cluster from outside the DESY network ~-~- eduroam or the DESY guest network are considered to be outside ~-~- then you can SSH tunnel your connection through **[[desy-ps-ext.desy.de>>url:http://desy-ps-ext.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}} or more conveniently directly use the [[remote login(% class="confluence-embedded-file-wrapper confluence-embedded-manual-size" %)~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Remote+Login||rel="nofollow" shape="rect"]](%%) to **[[max-display.desy.de>>url:https://max-display.desy.de:3443/||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}.
22 -* If you have a default DESY AFS (DESY employees) account login to **[[max-fsc.desy.de>>url:http://max-fsc.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}} using your personal account. (From outside of DESY network tunnel through **[[bastion.desy.de>>url:http://bastion.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}} or use the [[remote login(% class="confluence-embedded-file-wrapper confluence-embedded-manual-size" %)~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Remote+Login||rel="nofollow" shape="rect"]](%%) to **[[max-display.desy.de>>url:http://max-display.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}.
19 +* To connect to the Maxwell cluster as PhotonScience user (or staff) please have a look to the [[Photon Science Resources - Maxwell >>url:https://confluence.desy.de/display/MXW/Photon+Science+Resources||shape="rect"]] pages.
20 +* From outside the DESY network ~-~- eduroam or the DESY guest network are considered to be outside ~-~- then you can SSH tunnel your connection through **[[max-display.desy.de>>url:https://confluence.desy.de/display/MXW/Interactive+Login||shape="rect"]]**{{code language="none"}}{{/code}}.
23 23  
24 -\\
25 -
26 -{{info title="login with functional accounts"}}
27 -**Note**: Please, always do use your personal account to work on the Maxwell cluster or any central DESY computing resources. Don't use the functional accounts like vuvfuser or bl1user. Using functional accounts on shared resources causes many difficult to predict problems, even if it might appear to work for you in the beginning. Also, **never** try to login to **bastion**{{code language="none"}}{{/code}}, **desy-ps-ext**{{code language="none"}}{{/code}}, or **max-display**{{code language="none"}}{{/code}} using one of the functional accounts.
22 +{{info title="
23 +login with functional accounts"}}
24 +**Note**: Please, always do use your personal account to work on the Maxwell cluster or any central DESY computing resources. Don't use the functional accounts like vuvfuser or bl1user. Using functional accounts on shared resources causes many difficult to predict problems, even if it might appear to work for you in the beginning. Also, **never** try to login to **bastion **{{code language="none"}}{{/code}}or **max-display**{{code language="none"}}{{/code}} using one of the functional accounts.
28 28  {{/info}}
29 29  
30 30  == Maxwell cluster ==
31 31  
32 -The extensive documentation can be found at [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Photon+Science+on+Maxwell||rel="nofollow" shape="rect"]]
29 +The extensive documentation can be found at [[Maxwell Cluster>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]]
33 33  
34 -In particular we want to mention the option [[doc:IS.JupyterHub on Maxwell]] , available [[doc:IS.Software]] , for more advanced computing demands [[Running Jobs on Maxwell.>>doc:IS.Running Jobs on Maxwell]]
31 +In particular we want to mention the option [[JupyterHub2 on Maxwell>>doc:MXW.JupyterHub2 on Maxwell]] , available [[Software>>doc:IS.Software]] , for more advanced computing demands [[Running Jobs on Maxwell.>>doc:MXW.Introduction to SLURM jobs]]
35 35  
36 -== Where to store data ==
37 -
38 -There are a number of data stores not linked to the ASAP3 file systems available to cover for storage needs unrelated to experiments or beamtimes. Our page "[[doc:FLASH.Where to store data]]" presents an overview of the storage resources and the [[unified access to disks on DESY's GNU/Linux and MS Windows>>url:https://confluence.desy.de/display/FLASH/Where+to+store+data?preview=%2F131370944%2F143223821%2FFLASH-Disk-Storage-Update.pdf||shape="rect"]] PCs.
39 -
40 -\\
33 +