Skip to Content

Changes for page 05. Data processing

Last modified by tabermah on 2024-12-11 11:17
From version 37.1
edited by tabermah
on 2024-09-24 11:14
Change comment: There is no comment for this version
To version 38.1
edited by tabermah
on 2024-12-03 10:59
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,6 +1,6 @@
1 1  = {{id name="11.Dataprocessing-11.1.Automaticdataprocessing"/}}**11.1. Automatic data processing** =
2 2  
3 -Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP ([[https:~~/~~/www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html>>url:https://www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html||shape="rect"]], current version 3.1.9) (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color: rgb(0,0,0);text-decoration: none;" %)Zhang et al. (2006) (% style="text-decoration: none;text-align: left;" %)//J. Appl. Cryst.//(% style="color: rgb(0,0,0);text-decoration: none;" %) (% style="text-decoration: none;text-align: left;" %)**39**(% style="color: rgb(0,0,0);text-decoration: none;" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
3 +Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP ([[https:~~/~~/www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html>>url:https://www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html||shape="rect"]], current version 3.1.9) (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color:#000000; text-decoration:none" %)Zhang et al. (2006) (% style="text-align:left; text-decoration:none" %)//J. Appl. Cryst.//(% style="color:#000000; text-decoration:none" %) (% style="text-align:left; text-decoration:none" %)**39**(% style="color:#000000; text-decoration:none" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
4 4  
5 5  = {{id name="11.Dataprocessing-11.2.Manualdataprocessing"/}}**11.2. Manual data processing** =
6 6  
... ... @@ -8,13 +8,13 @@
8 8  
9 9  Use Maxwell-cluster for data processing as described in the next chapter. Access to Maxwell requires a Scientific Account.
10 10  
11 -Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Scientific Accounts>>doc:FSEC.Science Accounts.WebHome||shape="rect"]].
11 +Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Scientific Accounts>>https://xwiki.desy.de/xwiki/bin/view/FSEC/FS-IT%20Topics/Science%20Accounts/||shape="rect"]].
12 12  
13 13  (% style="text-decoration: none;" %)
14 14  == {{id name="11.Dataprocessing-Prerequisites"/}}Prerequisites ==
15 15  
16 16  (% style="text-align: justify;" %)
17 -Local users can perform manual data processing as described at the beamline. Users having remote access can also process their data only if they are through their [[DESY Science Accounts>>doc:FSEC.Science Accounts.WebHome||style="text-decoration: none;" shape="rect"]]. Users may apply for science accounts to analyze data measured at Petra III or Flash on resources located at the DESY computer center. Science accounts have a standard lifetime of 3 years (renewable).
17 +Local users can perform manual data processing as described at the beamline. Users having remote access can also process their data only if they are through their [[DESY Science Accounts>>https://xwiki.desy.de/xwiki/bin/view/FSEC/FS-IT%20Topics/Science%20Accounts/||shape="rect" style="text-decoration: none;"]]. Users may apply for science accounts to analyze data measured at Petra III or Flash on resources located at the DESY computer center. Science accounts have a standard lifetime of 3 years (renewable).
18 18  
19 19  (% style="text-align: justify;" %)
20 20  **Applying for a Science Account:**
... ... @@ -26,7 +26,7 @@
26 26  
27 27  **Multi Factor Authentication (MFA) at DESY:**
28 28  
29 -For accessing the DESY computers a two-factor authentication is needed. After you have received your Science Account, you will need to setup an Authenticator App to your smartphone. You can find the instructions [[here>>url:https://it.desy.de/services/mfa/external_people/index_eng.html||style="text-decoration: none;" shape="rect"]].
29 +For accessing the DESY computers a two-factor authentication is needed. After you have received your Science Account, you will need to setup an Authenticator App to your smartphone. You can find the instructions [[here>>url:https://it.desy.de/services/mfa/external_people/index_eng.html||shape="rect" style="text-decoration: none;"]].
30 30  
31 31  (% style="text-decoration: none;" %)
32 32  == {{id name="11.Dataprocessing-AccessingData"/}}Accessing Data ==
... ... @@ -49,7 +49,7 @@
49 49  1*. open a terminal
50 50  1*. cd /asap3/petra3/gpfs/p11/YEAR/data/**YourBeamTimeID**
51 51  1. **Installing and using the FastX3 client:​**
52 -1*. visit [[Downloads DESYcloud>>url:https://confluence.desy.de/pages/viewpage.action?spaceKey=IS&title=Downloads+DESYcloud||style="text-decoration: none;" shape="rect"]]. Login with your DESY account.
52 +1*. visit [[Downloads DESYcloud>>url:https://confluence.desy.de/pages/viewpage.action?spaceKey=IS&title=Downloads+DESYcloud||shape="rect" style="text-decoration: none;"]]. Login with your DESY account.
53 53  1*. click on the desycloud link next to the starnet entry. If you get to an empty page, just reload the page.
54 54  1*. go to StarNet_FastX3/client/3.3 (or the highest version listed) and download the package matching your OS
55 55  1*1. FastX-*-setup.exe and FastX3.msi are regular windows installer
... ... @@ -81,13 +81,13 @@
81 81  The processing folders contain two folders, 'xdsapp' and 'manual'. 'Manual' contains a template for manual processing having all the correct parameters. However, to shorten the processing time and make use of the computational structure, reprocessing your data on the maxwell nodes should be done by using a script to queue the jobs in slurm:
82 82  
83 83  * template scripts can be found at /asap3/petra3/gpfs/common/p11/processing_scripts/
84 -** (% style="color: rgb(0,0,0);text-decoration: none;" %)xds_users.sh For (% style="color: rgb(0, 0, 0); text-decoration: none; color: rgb(255, 0, 0)" %)external users
85 -** xds_inhouse.sh For (% style="color: rgb(255,0,0);" %)in-house users
86 -* (% style="color: rgb(0,0,0);" %)differences on the scripts are on the partition (i.e. //pscpu// should be used for internal users of Photon Science and //psxcpu// for external users) and the xds version used for processing
87 -* (% style="color: rgb(0,0,0);" %)copy the script to the folder where you run the processing
88 -* (% style="color: rgb(0,0,0);" %)launch the script by typing //sbatch xds//(% style="color: rgb(0, 0, 0); color: rgb(255, 0, 0)" %)//_//(% style="color: rgb(0, 0, 0); color: rgb(0, 0, 0); text-decoration: none; color: rgb(255, 0, 0)" %)//name_of_the_script//(% style="color: rgb(0, 0, 0); color: rgb(0, 0, 0); text-decoration: none" %)//.sh// (e.g., //sbatch xds_users.sh //for external users )
89 -* (% style="color: rgb(0,0,0);" %)the script will find a free node for your job to run fast
90 -* (% style="color: rgb(0,0,0);" %)the xds-log you get in a file xds-job-~#~##.out.
84 +** (% style="color:#000000; text-decoration:none" %)xds_users.sh For (% style="color:#ff0000; text-decoration:none" %)external users
85 +** xds_inhouse.sh For (% style="color:#ff0000" %)in-house users
86 +* (% style="color:#000000" %)differences on the scripts are on the partition (i.e. //pscpu// should be used for internal users of Photon Science and //psxcpu// for external users) and the xds version used for processing
87 +* (% style="color:#000000" %)copy the script to the folder where you run the processing
88 +* (% style="color:#000000" %)launch the script by typing //sbatch xds//(% style="color:#ff0000" %)//_//(% style="color:#ff0000; text-decoration:none" %)//name_of_the_script//(% style="color:#000000; text-decoration:none" %)//.sh// (e.g., //sbatch xds_users.sh //for external users )
89 +* (% style="color:#000000" %)the script will find a free node for your job to run fast
90 +* (% style="color:#000000" %)the xds-log you get in a file xds-job-~#~##.out.
91 91  
92 92  (% style="text-align: justify;" %)
93 93  **Processing Serial Crystallography data** by nxds should be done via a script to queue the jobs in slurm:
... ... @@ -94,9 +94,9 @@
94 94  
95 95  * a template script (nxds.sh) as well as an example of input file (nXDS.INP) can be found at /asap3/gpfs/common/p11/processing_scripts/
96 96  * the partition in the script is defined psxpcu for external academic users (pscpu should be used for internal users of Photon Science)
97 -* (% style="color: rgb(0,0,0);" %)copy the script and the nXDS.INP to the folder where you run the processing
98 -* (% style="color: rgb(0,0,0);" %)the nXDS.INP file needs some editing to fit your data path and data collection parameters (wavelength, detector distance, oscillation...)
99 -* (% style="color: rgb(0,0,0);" %)the processing is launch by typing sbatch nxds.sh
97 +* (% style="color:#000000" %)copy the script and the nXDS.INP to the folder where you run the processing
98 +* (% style="color:#000000" %)the nXDS.INP file needs some editing to fit your data path and data collection parameters (wavelength, detector distance, oscillation...)
99 +* (% style="color:#000000" %)the processing is launch by typing sbatch nxds.sh
100 100  
101 101  **If manual processing through a terminal or GUI is absolutely necessary:**
102 102  
... ... @@ -109,7 +109,7 @@
109 109  == {{id name="11.Dataprocessing-CrystallographicandScientificSoftwarePackages"/}}Crystallographic and Scientific Software Packages ==
110 110  
111 111  (% style="text-align: justify;" %)
112 -Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on [[confluence>>doc:IS.Software.WebHome||style="text-decoration: none;" shape="rect"]].
112 +Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on [[confluence>>doc:IS.Software.WebHome||shape="rect" style="text-decoration: none;"]].
113 113  
114 114  (% style="text-align: justify;" %)
115 115  Crystallographic software packages are in the [[Photon Science section>>doc:MXW.Photon Science.WebHome||shape="rect"]].
... ... @@ -121,8 +121,8 @@
121 121  To use crystallographic programs type in terminal (for example):
122 122  
123 123  * **module avail** #to check what programs and versions are available
124 -* **module load xray** # sets the environment for old xds and helper GUIs such as (% style="color: rgb(0,0,0);" %)XDSAPP
125 -* (% style="color: rgb(0,0,0);" %)**module load maxwell xds** # sets the environment for new xds
124 +* **module load xray** # sets the environment for old xds and helper GUIs such as (% style="color:#000000" %)XDSAPP
125 +* (% style="color:#000000" %)**module load maxwell xds** # sets the environment for new xds
126 126  * **module load ccp4/7** # sets the environment for ccp4 version 7
127 127  * **module load phenix** # sets the environment for phenix
128 128  
... ... @@ -129,10 +129,8 @@
129 129  (% style="text-align: justify;" %)
130 130  Once the correct module is loaded, the program can be called via the command line by typing its shortcut or alias (xdsapp, phenix, albula....etc).
131 131  
132 -\\
133 133  
134 -\\
135 135  
136 136  (((
137 -\\
135 +
138 138  )))