Last modified by sndueste on 2025/02/05 15:00
From version 15.1
edited by sendels
on 2021/07/08 11:54
Change comment: There is no comment for this version
To version 5.1
edited by sendels
on 2019/08/09 10:27
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,36 +1,126 @@
1 -= Brief Guide to Computing Resources at FLASH =
1 += Brief User Guide to Computing Resources at FLASH =
2 2  
3 3  \\
4 4  
5 -== Introduction ==
5 +* Brief User Guide to Computing Resources at FLASH
6 +** CAMP at BL1: hasfumidaq
7 +** Maxwell HPC Cluster
8 +** Gamma Portal
9 +** Flash Control System
6 6  
7 -As explained in the [[doc:FLASH.User overview]] page the data recorded at FLASH is stored in an internal DAQ format (.raw) which is converted with a few minutes delay to easily accessible HDF5 ([[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]]) files which are the basis for data analysis. These  files are stored on the [[ASAP3 File System>>url:https://confluence.desy.de/display/ASAP3/ASAP3++Data+Storage+for+PETRA+III||rel="nofollow" shape="rect"]] which is accessible from DESYs  [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Maxwell+for+Photon+Science||rel="nofollow" shape="rect"]] which is recommended to be used for the analysis by users. H5py and Pandas allow efficient access to the recorded data in the HDF5 files. In addition the HDF5 files are accessible from the beamline computers and via a web interface:  [[Gamma Portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://gamma-portal.desy.de||shape="rect"]]
11 +\\
8 8  
9 9  \\
10 10  
11 -== Access to the Maxwell cluster ==
15 +== CAMP at BL1: hasfumidaq ==
12 12  
13 -The [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Photon+Science+on+Maxwell||rel="nofollow" shape="rect"]] is a powerful farm of inter-connected PC nodes running Scientific GNU/Linux. **To access the cluster you need a DESY AFS computing account or a so called scientific account (PSX).**
17 +**TL;DR:** Use the [[Maxwell cluster~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]] instead.
14 14  
15 -{{info title="PSX account"}}
16 -The scientific (internally called: PSX)  account is a "real" account that is needed to be able to log into DESY computers ( mainly used for Maxwell). This is different from the[[ DOOR>>url:https://door.desy.de/door/index.php||shape="rect"]] account that you need for administrative purposes to be registered for the experiment, do safety training etc.
19 +The [[CAMP~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:http://photon-science.desy.de/facilities/flash/beamlines/bl_beamlines_flash1/camp/index_eng.html||shape="rect"]] cooperation used to operate a server for data acquisition and online analysis named **hasfumidaq**{{code language="none"}}{{/code}} (HASylab Flash Ultra fast Molecular Imaging DAQ). Currently the server runs as a backup only. It is going to be decommissioned, most probably early 2020.
17 17  
18 -To get a PSX account please talk to your local contact well before the beamtime. The local contact will take care of the paper work and send you more instructions.
19 -{{/info}}
21 +\\
20 20  
21 -* Scientific account (PSX) users instead go to **[[desy-ps-cpu.desy.de>>url:http://desy-ps-cpu.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}. To connect to the Maxwell cluster from outside the DESY network ~-~- eduroam or the DESY guest network are considered to be outside ~-~- then you can SSH tunnel your connection through **[[max-display.desy.de>>url:https://max-display.desy.de:3443/||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}.
22 -* If you have a default DESY AFS (DESY employees) account login to **[[max-fsc.desy.de>>url:http://max-fsc.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}} using your personal account. (From outside of DESY network tunnel through **[[bastion.desy.de>>url:http://bastion.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}} or use the [[remote login(% class="confluence-embedded-file-wrapper confluence-embedded-manual-size" %)~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Remote+Login||rel="nofollow" shape="rect"]](%%) to **[[max-display.desy.de>>url:http://max-display.desy.de||rel="nofollow" shape="rect" class="external-link"]]**{{code language="none"}}{{/code}}.
23 +=== Hardware and Operating System ===
23 23  
25 +The machine {{code language="none"}}hasfumidaq{{/code}} is a 12 core Dell Precision T5500 with 12 GByte RAM and more than 80 TByte disk space running a GNU/Linux, Ubuntu 16.04, operating system.
26 +
27 +The local disk space of {{code language="none"}}hasfumidaq{{/code}} is **not** under any backup regime, ie. what is deleted or overwritten here is gone for good. The storage layout is as follows:
28 +
24 24  \\
25 25  
26 -{{info title="login with functional accounts"}}
27 -**Note**: Please, always do use your personal account to work on the Maxwell cluster or any central DESY computing resources. Don't use the functional accounts like vuvfuser or bl1user. Using functional accounts on shared resources causes many difficult to predict problems, even if it might appear to work for you in the beginning. Also, **never** try to login to **bastion **{{code language="none"}}{{/code}}or **max-display**{{code language="none"}}{{/code}} using one of the functional accounts.
28 -{{/info}}
31 +(% class="wrapped" %)
32 +|=(((
33 +mount point
34 +)))|=(((
35 +type
36 +)))|=(((
37 +purpose
38 +)))
39 +|(((
40 +{{code language="none"}}
41 +~/
42 +{{/code}}
43 +)))|(((
44 +AFS network
45 +)))|(((
46 +programs and precious data needing backup
47 +)))
48 +|(((
49 +{{code language="none"}}
50 +/var/acqu/
51 +{{/code}}
52 +)))|(((
53 +local disk
54 +)))|(((
55 +short term data acquisition files
56 +)))
57 +|(((
58 +{{code language="none"}}
59 +/var/data/
60 +{{/code}}
61 +)))|(((
62 +local disk
63 +)))|(((
64 +midterm data storage and reproducible analysis
65 +)))
66 +|(((
67 +{{code language="none"}}
68 +/pnfs/desy.de/flash1/disk/
69 +{{/code}}
70 +)))|(((
71 +NFS network
72 +)))|(((
73 +longer term data storage for off line analysis
74 +)))
29 29  
30 -== Maxwell cluster ==
76 +For data acquisition {{code language="none"}}/var/acqu/{{/code}} is exported via NFS to {{code language="none"}}hasfcamppnccd1{{/code}} and {{code language="none"}}hasfcamppnccd2{{/code}}. There also is a CIFS (SAMBA) export of {{code language="none"}}/var/acqu/{{/code}} for user bl1acqus. Yet the latter is supposed to be used for MS Windows data acquisition nodes only.
31 31  
32 -The extensive documentation can be found at [[Maxwell Cluster>>url:https://confluence.desy.de/display/IS/Photon+Science+on+Maxwell||rel="nofollow" shape="rect"]]
78 +Note, there is a caveat with the PNFS (perfectly normal file system) disk: Files are not modifiable on this file system, once they are created.
33 33  
34 -In particular we want to mention the option [[doc:MXW.JupyterHub on Maxwell]] , available [[doc:IS.Software]] , for more advanced computing demands [[Running Jobs on Maxwell.>>doc:MXW.Introduction to SLURM jobs]]
80 +\\
35 35  
82 +=== User Login ===
83 +
84 +{{code language="none"}}Hasfumidaq{{/code}} provides logins to AFS account owners, who are members of the groups {{code language="none"}}hasylab{{/code}} or {{code language="none"}}cfel{{/code}}. Accordingly, you have access to your DESY AFS home directory on {{code language="none"}}hasfumidaq{{/code}}.
85 +
86 +If you use {{code language="none"}}hasfumidaq{{/code}} make sure you do not interfere with data acquisition at BL1, i.e. do not occupy many processor cores at full load and do not incur heavy network traffic. Mind, a single Matlab GUI creates a lot of network traffic already. For offline data analysis use the Petra III workgroup cluster instead. Please, do use your **personal account** working on {{code language="none"}}hasfumidaq{{/code}}. The functional account "bl1user" here should be restricted to data acquisition and the account "vuvfuser" to online data analysis for guests, who do not have a DESY AFS account.
87 +
36 36  \\
89 +
90 +=== Software and Administration ===
91 +
92 +For online or quasi online data analysis the usual set of scientific **software** is installed on hasfumidaq: Python with all the number crunching libraries, Java, HDFView in a current version, h5tools, Matlab R2013b and the DOOCS tools and libraries. If you need more, talk to the admins.
93 +
94 +**Administrators** of {{code language="none"}}hasfumidaq{{/code}} are the staff at FS-EC (Blume, Fleck, Rothkirch, Sendel), the staff at IT (Flemming et al.), the CAMP staff (Passow, Erk) and the IT people at FS-FL (Düsterer, Grunewald).
95 +
96 +\\
97 +
98 +== Maxwell HPC Cluster ==
99 +
100 +The Maxwell HPC cluster consists of a set of 64 core servers with 512 GByte of RAM each. They run a GNU/Linux of (% class="twikiNewLink" %)CentOS(%%) Linux 7 (Core) flavour. You login to **max-fsc**{{code language="none"}}{{/code}} and are connected to the server with lowest load. Please see [[the docs on confluence for details~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/IS/Maxwell||shape="rect"]].
101 +
102 +Your AFS home directory should be available as a link at ~~/afs/. You have an ASAP3 home directory common to all Maxwell nodes.
103 +
104 +On all {{code language="none"}}max-fsc{{/code}} nodes you have access to [[desy.de/flash1/disk/>>url:http://desy.de/flash1/disk/||shape="rect"]]{{code language="none"}}/pnfs/{{/code}} just as on {{code language="none"}}hasfumidaq{{/code}}. For intermediate storage of large files use the {{code language="none"}}/scratch{{/code}} space local to each node.
105 +
106 +Also the ASAP3 Core File System is mounted on all Maxwell nodes. It is accessible at /asap3/flash/gpfs~/~//data~/~/. Please refer to the [[documentation on confluence~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://confluence.desy.de/display/ASAP3||shape="rect"]].
107 +
108 +The cluster also is the location your files will be staged to, if you access them through the Gamma Portal.
109 +
110 +\\
111 +
112 +== Gamma Portal ==
113 +
114 +The [[Gamma Portal~[~[image:url:http://hasfweb.desy.de/pub/TWiki/TWikiDocGraphics/external-link.gif~|~|width="13" height="12"~]~]>>url:https://gamma-portal.desy.de||shape="rect"]] is a web front end to DESY's dCache tape archive.
115 +
116 +Your data are migrated upon request to the responsible beam line scientist to the tape archive. The archive is organised along beamtime application numbers and provides elaborate access control.
117 +
118 +You access your data using your DOOR account. The web front end allows you to download your data. Also, if you have a DESY AFS account, it permits you to stage the data. That means a copy of your data is created on the PNFS disk for analysis on the Petra III workgroup cluster.
119 +
120 +\\
121 +
122 +== Flash Control System ==
123 +
124 +In addition to the services outlined above, DESY's group MCS-4 provides data acquisition services related to accelerator operations and beam diagnostics. For details you are referred to Stefan's [[doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).WebHome]] pages.
125 +
126 +For online DAQ access and analysis you login at {{code language="none"}}flashlxuser1{{/code}} or {{code language="none"}}flashlxuser2{{/code}}, using one of the functional accounts.