Skip to Content

Changes for page 05. Data processing

Last modified by tabermah on 2024-12-11 11:17
From version 27.1
edited by tabermah
on 2023-03-01 14:59
Change comment: There is no comment for this version
To version 13.1
edited by hakanj
on 2020-12-18 14:57
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -11. Data processing
1 +10. Data processing
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.tabermah
1 +XWiki.hakanj
Content
... ... @@ -1,50 +1,39 @@
1 1  \\
2 2  
3 -**11.1. Automatic data processing**
3 +**10.1. Automatic data processing**
4 4  
5 -Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP ([[https:~~/~~/www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html>>url:https://www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html||shape="rect"]], current version 3.1.9) (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color: rgb(0,0,0);text-decoration: none;" %)Zhang et al. (2006) (% style="text-decoration: none;text-align: left;" %)//J. Appl. Cryst.//(% style="color: rgb(0,0,0);text-decoration: none;" %) (% style="text-decoration: none;text-align: left;" %)**39**(% style="color: rgb(0,0,0);text-decoration: none;" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
5 +Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color: rgb(0,0,0);text-decoration: none;" %)Zhang et al. (2006). (% style="text-decoration: none;text-align: left;" %)//J. Appl. Cryst.//(% style="color: rgb(0,0,0);text-decoration: none;" %) (% style="text-decoration: none;text-align: left;" %)**39**(% style="color: rgb(0,0,0);text-decoration: none;" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
6 6  
7 7  \\
8 8  
9 -**11.2. Manual data processing onsite**
9 +**10.3. Manual data processing onsite**
10 10  
11 -Please do not use the control computers of the beamline (haspp11user01-04) nor the haspp11eval01 for data processing (latter is now fully dedicated to autoprocessing).
11 +Open a terminal (see session 3.2) and type ssh haspp11eval01. Please do not use the control computers of the beamline (haspp11user01-04) for data processing.
12 12  
13 -Use Maxwell-cluster for data processing as described in the next chapter; on-site, you can directly ssh to username@desy-ps-cpu. Access to Maxwell requires a Scientific Account.
13 +Here you can launch crystallographic programs directly //e.g.//:
14 14  
15 -Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Scientific Accounts>>doc:FSEC.Science Accounts.WebHome||shape="rect"]].
15 +* for XDSAPP type in the terminal (% style="color: rgb(0,128,0);" %)xdsapp
16 +* for imosflm type in the terminal (% style="color: rgb(0,128,0);" %)imosflm
17 +* for CCP4 type in the terminal (% style="color: rgb(0,128,0);" %)ccp4i
18 +* for phenix type in the terminal (% style="color: rgb(0,128,0);" %)phenix
16 16  
17 17  \\
18 18  
19 -//Accessing data from outside the DESY network~://
22 +**10.4. Manual data processing after your beamtime or during remote session**
20 20  
21 -Accessing the Maxwell cluster is easiest with Max-display:  connect to [[https:~~/~~/max-fs-display.desy.de:3389/auth/ssh/>>url:https://max-fs-display.desy.de:3389/auth/ssh/||shape="rect"]] via web browser with the credentials of your scientific account.
24 +Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Science Accounts>>doc:FSEC.Science Accounts.WebHome||shape="rect"]].
22 22  
23 -Accesss via terminal window:
26 +//Accessing data from outside the DESY network~://
27 +In a terminal window:
24 24  ssh (% class="Object" %)[[username@desy-ps-ext.desy.de>>mailto:username@desy-ps-ext.desy.de||shape="rect"]](%%)
25 25  ssh username@desy-ps-cpu
26 -
27 27  Give your password when prompted
28 28  Navigate to data by:
29 29  cd /asap3/petra3/gpfs/p11/2020/data/beamtimeID
30 30  
31 31  Data is accessible to users who were registered participants to the experiment in Door prior to the beamtime ID being opened at the beamline. If you cannot access the data, please contact your LC or the beamline manager.
32 -
33 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Our processing folders now contain two folders, 'full' and 'manual'. 'Manual' contains a template that has all the correct parameters and processing will run just by typing xds_par there. However, to shorten the processing time and make use of the computational structure, you should use a script to queue the jobs in slurm. A template script (xds.sh) can be found at /asap3/petra3/gpfs/common/p11/xds.sh
34 -
35 -The partition in the script is defined psx for external users (ps should be used for internal users of Photon Science).
36 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Copy the script to the folder where you run the processing and launch by (%%)
37 -(% style="color: rgb(51,51,51);text-decoration: none;" %)sbatch xds.sh(%%)
38 -(% style="color: rgb(51,51,51);text-decoration: none;" %)It will find a free node for your job and run faster than just typing xds_par(%%)
39 -(% style="color: rgb(51,51,51);text-decoration: none;" %)The xds-log you get in a file xds-job-~#~##.out
40 -
41 -(% style="color: rgb(51,51,51);text-decoration: none;" %)nxds can be used for processing Serial Crystallography data in a similar way than xds. As the computing time is long, nxds should should be used via a script to queue the jobs in slurm. The template script (nxds.sh) as well as an example of input file (nXDS.INP) can be found in /asap3/gpfs/common/p11/.
42 -
43 -(% style="color: rgb(51,51,51);text-decoration: none;" %)Copy the script and the nXDS.INP to the folder where you run the processing. The nXDS.INP file needs some editing to fit your data path and data collection parameters (wavelength, detector distance, oscillation...). The processing is launch by
44 -sbatch nxds.sh
45 -
46 -//List of software you can use is at~://
47 -(% class="Object" %)[[https:~~/~~/confluence.desy.de/display/MXW/Photon+Science>>doc:MXW.Photon Science.WebHome||shape="rect"]](%%)
35 +\\//List of software you can use is at~://
36 +(% class="Object" %)[[https:~~/~~/confluence.desy.de/display/IS/Photon+Science>>doc:IS.Photon Science.WebHome||shape="rect"]](%%)
48 48  \\//You can store analysis results in scratch_cc (temporary/testing results) or processed (final results).//
49 49  \\//To use crystallographic programs~://
50 50  module avail
... ... @@ -51,11 +51,3 @@
51 51  module load xray     # sets the environment for xds and helper GUIs such as XDSAPP
52 52  module load ccp4/6.4
53 53  module load phenix 
54 -
55 -\\
56 -
57 -\\
58 -
59 -(((
60 -\\
61 -)))
Confluence.Code.ConfluencePageClass[0]
Id
... ... @@ -1,1 +1,1 @@
1 -353491615
1 +188798286
Title
... ... @@ -1,1 +1,1 @@
1 -11. Data processing
1 +10. Data processing
URL
... ... @@ -1,1 +1,1 @@
1 -https://confluence.desy.de/spaces/11/pages/353491615/11. Data processing
1 +https://confluence.desy.de/spaces/11/pages/188798286/10. Data processing