Skip to Content

Changes for page 05. Data processing

Last modified by tabermah on 2024-12-11 11:17
From version 25.1
edited by tabermah
on 2022-09-08 13:22
Change comment: There is no comment for this version
To version 28.1
edited by tabermah
on 2024-03-21 15:14
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,57 +1,134 @@
1 -\\
1 += {{id name="11.Dataprocessing-11.1.Automaticdataprocessing"/}}**11.1. Automatic data processing** =
2 2  
3 -**11.1. Automatic data processing**
3 +Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP ([[https:~~/~~/www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html>>url:https://www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html||shape="rect"]], current version 3.1.9) (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color: rgb(0,0,0);text-decoration: none;" %)Zhang et al. (2006) (% style="text-decoration: none;text-align: left;" %)//J. Appl. Cryst.//(% style="color: rgb(0,0,0);text-decoration: none;" %) (% style="text-decoration: none;text-align: left;" %)**39**(% style="color: rgb(0,0,0);text-decoration: none;" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
4 4  
5 -Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color: rgb(0,0,0);text-decoration: none;" %)Zhang et al. (2006). (% style="text-decoration: none;text-align: left;" %)//J. Appl. Cryst.//(% style="color: rgb(0,0,0);text-decoration: none;" %) (% style="text-decoration: none;text-align: left;" %)**39**(% style="color: rgb(0,0,0);text-decoration: none;" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
5 += {{id name="11.Dataprocessing-11.2.Manualdataprocessing"/}}**11.2. Manual data processing** =
6 6  
7 -\\
8 -
9 -**11.2. Manual data processing onsite**
10 -
11 11  Please do not use the control computers of the beamline (haspp11user01-04) nor the haspp11eval01 for data processing (latter is now fully dedicated to autoprocessing).
12 12  
13 -Use Maxwell-cluster for data processing as described in the next chapter; on-site, you can directly ssh to username@desy-ps-cpu. Access to Maxwell requires a Scientific Account.
9 +Use Maxwell-cluster for data processing as described in the next chapter. Access to Maxwell requires a Scientific Account.
14 14  
15 15  Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Scientific Accounts>>doc:FSEC.Science Accounts.WebHome||shape="rect"]].
16 16  
17 -\\
13 +(% style="text-decoration: none;" %)
14 +== {{id name="11.Dataprocessing-Prerequisites"/}}Prerequisites ==
18 18  
19 -//Accessing data from outside the DESY network~://
16 +(% style="text-align: justify;" %)
17 +Local users can perform manual data processing as described at the beamline. Users having remote access can also process their data only if they are through their [[DESY Science Accounts>>doc:FSEC.Science Accounts.WebHome||style="text-decoration: none;" shape="rect"]]. Users may apply for science accounts to analyze data measured at Petra III or Flash on resources located at the DESY computer center. Science accounts have a standard lifetime of 3 years (renewable).
20 20  
21 -Accessing the Maxwell cluster is easiest with Max-display:  connect to [[https:~~/~~/max-display.desy.de:3443>>url:https://max-display.desy.de:3443||shape="rect"]] via web browser with the credentials of your scientific account.
19 +(% style="text-align: justify;" %)
20 +**Applying for a Science Account:**
22 22  
23 -Accesss via terminal window:
24 -ssh (% class="Object" %)[[username@desy-ps-ext.desy.de>>mailto:username@desy-ps-ext.desy.de||shape="rect"]](%%)
25 -ssh username@desy-ps-cpu
22 +1. the user submits a request to the beamline scientist/manager
23 +1. the user receives a pre-filled form by Email within a few business days from the DESY-FS Administrator
24 +1. the form has to be signed and sent back to the Administrator (hard-copy as scan/photo by email/fax in pdf format)
25 +1. the user receives the initial credentials from the beamline scientist/manager who forwarded the account request. If on site at DESY, the user can authenticate him-/herself by valid ID card or passport at FS-EC's Administrators to retrieve the initial password for the account.
26 26  
27 -Give your password when prompted
28 -Navigate to data by:
29 -cd /asap3/petra3/gpfs/p11/2020/data/beamtimeID
27 +(% style="" %)
28 +**Multi Factor Authentication (MFA) at DESY:**
30 30  
30 +(% style="" %)
31 +For accessing the DESY computers a two-factor authentication is needed. After you have received your Science Account, you will need to setup an Authenticator App to your smartphone. You can find the instructions [[here>>url:https://it.desy.de/services/mfa/external_people/index_eng.html||style="text-decoration: none;" shape="rect"]].
32 +
33 +(% style="text-decoration: none;" %)
34 +== {{id name="11.Dataprocessing-AccessingData"/}}Accessing Data ==
35 +
36 +(% style="text-align: justify;" %)
37 +There are three methods to access DESY computers from outside using a scientific account:
38 +
39 +1. **Command line:**
40 +in a Linux terminal or Cygwin (windows) type:
41 +1*. ssh [[username@max-fs-display.desy.de>>mailto:username@max-fs-display.desy.de||shape="rect"]]
42 +1*. enter your Science Account password
43 +1*. enter your 2FA token
44 +1*. cd /asap3/petra3/gpfs/p11/YEAR/data/**YourBeamTimeID**
45 +1. **​Graphical User Interface:**
46 +in a web browser type:
47 +1*. [[https:~~/~~/max-fs-display.desy.de:3389/>>url:https://max-fs-display.desy.de:3389/||shape="rect"]]
48 +1*. enter your Science Account credentials
49 +1*. enter your 2FA token
50 +1*. launch e.g. an XFCE session
51 +1*. open a terminal
52 +1*. cd /asap3/petra3/gpfs/p11/YEAR/data/**YourBeamTimeID**
53 +1. **Installing and using the FastX3 client:​**
54 +1*. visit [[Downloads DESYcloud>>url:https://confluence.desy.de/pages/viewpage.action?spaceKey=IS&title=Downloads+DESYcloud||style="text-decoration: none;" shape="rect"]]. Login with your DESY account.
55 +1*. click on the desycloud link next to the starnet entry. If you get to an empty page, just reload the page.
56 +1*. go to StarNet_FastX3/client/3.3 (or the highest version listed) and download the package matching your OS
57 +1*1. FastX-*-setup.exe and FastX3.msi are regular windows installer
58 +1*1. FastX-*-setup_nonroot.exe is a "portable" windows installer, e.g. can be placed on a USB stick
59 +1*1. FastX3-*.dmg is the MacOSX installer
60 +1*1. FastX3-*.rhel7.x86_64.tar.gz is a tarball for Linux, works also for Ubuntu or Debian (possibly requiring some additional packages)
61 +1*. install and launch FastX3
62 +1*. configure a new https connection (host: [[max-fs-display.desy.de>>url:http://max-fs-display.desy.de||shape="rect"]], port:3389, auth: SSH)
63 +1*. double click on the new connection
64 +1*. enter your 2FA token
65 +1*. launch e.g. an XFCE session
66 +1*. open a terminal
67 +1*. cd /asap3/petra3/gpfs/p11/YEAR/data/**YourBeamTimeID**
68 +
69 +(% style="text-align: justify;" %)
70 +Pros & cons:​
71 +
72 +* ssh: simple, convenient, but no GPU acceleration and X11 over ssh is slow. Sessions are not persistent.
73 +* browser: simple, fast,works everywhere, GPU hardware acceleration, but has some limitations (copy&paste, browser capturing key shortcuts). Supports session sharing
74 +* client: fast, works best, but requires installation of a client
75 +* browser and client offer persistent sessions: when disconnected your session continues to run without any client connected. Reconnecting to the display server gives access to the running session.
76 +
31 31  Data is accessible to users who were registered participants to the experiment in Door prior to the beamtime ID being opened at the beamline. If you cannot access the data, please contact your LC or the beamline manager.
32 32  
33 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Our processing folders now contain two folders, 'full' and 'manual'. 'Manual' contains a template that has all the correct parameters and processing will run just by typing xds_par there. However, to shorten the processing time and make use of the computational structure, you should use a script to queue the jobs in slurm. A template script (xds.sh) can be found at /asap3/petra3/gpfs/common/p11/xds.sh
79 +(% style="text-decoration: none;" %)
80 +== {{id name="11.Dataprocessing-Dataprocessing"/}}Data processing ==
34 34  
35 -The partition in the script is defined psx for external users (ps should be used for internal users of Photon Science).
36 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Copy the script to the folder where you run the processing and launch by (%%)
37 -(% style="color: rgb(51,51,51);text-decoration: none;" %)sbatch xds.sh(%%)
38 -(% style="color: rgb(51,51,51);text-decoration: none;" %)It will find a free node for your job and run faster than just typing xds_par(%%)
39 -(% style="color: rgb(51,51,51);text-decoration: none;" %)The xds-log you get in a file xds-job-~#~##.out
82 +(% style="text-align: justify;" %)
83 +The processing folders contain two folders, 'xdsapp' and 'manual'. 'Manual' contains a template for manual processing having all the correct parameters. However, to shorten the processing time and make use of the computational structure, reprocessing your data on the maxwell nodes should be done by using a script to queue the jobs in slurm:
40 40  
41 -(% style="color: rgb(51,51,51);text-decoration: none;" %)nxds can be used for processing Serial Crystallography data in a similar way than xds. As the computing time is long, nxds should should be used via a script to queue the jobs in slurm. The template script (nxds.sh) as well as an example of input file (nXDS.INP) can be found in /asap3/gpfs/common/p11/.
85 +* a template script (xds.sh) can be found at /asap3/petra3/gpfs/common/p11/
86 +* (% style="color: rgb(0,0,0);" %)the partition in the script is defined psx for external users (ps should be used for internal users of Photon Science)
87 +* (% style="color: rgb(0,0,0);" %)copy the script to the folder where you run the processing
88 +* (% style="color: rgb(0,0,0);" %)launch the script by typing sbatch xds.sh
89 +* (% style="color: rgb(0,0,0);" %)the script will find a free node for your job to run fast
90 +* (% style="color: rgb(0,0,0);" %)the xds-log you get in a file xds-job-~#~##.out.
42 42  
43 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Copy the script and the nXDS.INP to the folder where you run the processing. The nXDS.INP file needs some editing to fit your data path and data collection parameters (wavelength, detector distance, oscillation...). The processing is launch by
44 -sbatch nxds.sh
92 +(% style="text-align: justify;" %)
93 +**Processing Serial Crystallography data** by nxds should be done via a script to queue the jobs in slurm:
45 45  
46 -//List of software you can use is at~://
47 -(% class="Object" %)[[https:~~/~~/confluence.desy.de/display/MXW/Photon+Science>>doc:MXW.Photon Science.WebHome||shape="rect"]](%%)
48 -\\//You can store analysis results in scratch_cc (temporary/testing results) or processed (final results).//
49 -\\//To use crystallographic programs~://
50 -module avail
51 -module load xray     # sets the environment for xds and helper GUIs such as XDSAPP
52 -module load ccp4/6.4
53 -module load phenix 
95 +* a template script (nxds.sh) as well as an example of input file (nXDS.INP) can be found at /asap3/gpfs/common/p11/
96 +* the partition in the script is defined psx for external users (ps should be used for internal users of Photon Science)
97 +* (% style="color: rgb(0,0,0);" %)copy the script and the nXDS.INP to the folder where you run the processing
98 +* (% style="color: rgb(0,0,0);" %)the nXDS.INP file needs some editing to fit your data path and data collection parameters (wavelength, detector distance, oscillation...)
99 +* (% style="color: rgb(0,0,0);" %)the processing is launch by typing sbatch nxds.sh
54 54  
101 +(% style="" %)
102 +**If manual processing through a terminal or GUI is absolutely necessary:**
103 +
104 +* allocate a node to yourself by typing in terminal: salloc ~-~-node=1 ~-~-partition=psx ~-~-time=xx:xx:xx
105 +* ssh to the node allocated to you
106 +* load the modules of the needed crystallographic softwares on the node (instructions below)
107 +* HOX! terminate the allocation after usage by typing exit twice on your terminal window
108 +
109 +(% style="text-decoration: none;" %)
110 +== {{id name="11.Dataprocessing-CrystallographicandScientificSoftwarePackages"/}}Crystallographic and Scientific Software Packages ==
111 +
112 +(% style="text-align: justify;" %)
113 +Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on [[confluence>>doc:IS.Software.WebHome||style="text-decoration: none;" shape="rect"]].
114 +
115 +(% style="text-align: justify;" %)
116 +Crystallographic software packages are in the [[Photon Science section>>doc:MXW.Photon Science.WebHome||shape="rect"]].
117 +
118 +(% style="text-align: justify;" %)
119 +You can store analysis results in scratch_cc (temporary/testing results) or processed (final results).
120 +
121 +(% style="text-align: justify;" %)
122 +To use crystallographic programs type in terminal (for example):
123 +
124 +* **module avail** #to check what programs and versions are available
125 +* **module load xray** # sets the environment for xds and helper GUIs such as (% style="color: rgb(0,0,0);" %)XDSAPP
126 +* **module load ccp4/7** # sets the environment for ccp4 version 7
127 +* **module load phenix** # sets the environment for phenix
128 +
129 +(% style="text-align: justify;" %)
130 +Once the correct module is loaded, the program can be called via the command line by typing its shortcut or alias (xdsapp, phenix, albula....etc).
131 +
55 55  \\
56 56  
57 57  \\
Confluence.Code.ConfluencePageClass[0]
Id
... ... @@ -1,1 +1,1 @@
1 -280856310
1 +379127221
URL
... ... @@ -1,1 +1,1 @@
1 -https://confluence.desy.de/spaces/11/pages/280856310/11. Data processing
1 +https://confluence.desy.de/spaces/11/pages/379127221/11. Data processing