Skip to Content

Changes for page 05. Data processing

Last modified by tabermah on 2024-12-11 11:17
From version 38.1
edited by tabermah
on 2024-12-03 10:59
Change comment: There is no comment for this version
To version 43.1
edited by tabermah
on 2024-12-03 11:36
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -11. Data processing
1 +05. Data processing
Content
... ... @@ -1,8 +1,8 @@
1 -= {{id name="11.Dataprocessing-11.1.Automaticdataprocessing"/}}**11.1. Automatic data processing** =
1 += 5.1. Automatic data processing =
2 2  
3 3  Unless you un-check the option for the specific acquisition run, autoprocessing will be executed. For standard rotational data autoprocessing is via XDSAPP ([[https:~~/~~/www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html>>url:https://www.helmholtz-berlin.de/forschung/oe/ps/macromolecular-crystallography/hzb-mx-software/xdsapp3/index_en.html||shape="rect"]], current version 3.1.9) (Sparta et al. (2016) //J. Apple. Cryst.// **49**, 1085-1092) for strategy calculation we use Mosflm (Leslie //Acta D//**62** (2006) 48-57) and Spotfinder ((% style="color:#000000; text-decoration:none" %)Zhang et al. (2006) (% style="text-align:left; text-decoration:none" %)//J. Appl. Cryst.//(% style="color:#000000; text-decoration:none" %) (% style="text-align:left; text-decoration:none" %)**39**(% style="color:#000000; text-decoration:none" %), 112–119)(%%) for producing the heatmap for the grid scans. The results are compiled into HTML-files to give you a convenient way of checking them. The output files are in the processed folders of the corresponding datasets.  A self refreshing index of all results is created in the beamtimes processed folder. During the beamtime you can open it from file::~/~//gpfs/current/processed/index.html in a browser. The HTML-file is also contained in your data backup.
4 4  
5 -= {{id name="11.Dataprocessing-11.2.Manualdataprocessing"/}}**11.2. Manual data processing** =
5 += 5.2. Manual data processing =
6 6  
7 7  Please do not use the control computers of the beamline (haspp11user01-04) nor the haspp11eval01 for data processing (latter is now fully dedicated to autoprocessing).
8 8  
... ... @@ -11,7 +11,7 @@
11 11  Manual data processing after the beamtime or during a remote session is only available for registered [[DESY Scientific Accounts>>https://xwiki.desy.de/xwiki/bin/view/FSEC/FS-IT%20Topics/Science%20Accounts/||shape="rect"]].
12 12  
13 13  (% style="text-decoration: none;" %)
14 -== {{id name="11.Dataprocessing-Prerequisites"/}}Prerequisites ==
14 +== Prerequisites ==
15 15  
16 16  (% style="text-align: justify;" %)
17 17  Local users can perform manual data processing as described at the beamline. Users having remote access can also process their data only if they are through their [[DESY Science Accounts>>https://xwiki.desy.de/xwiki/bin/view/FSEC/FS-IT%20Topics/Science%20Accounts/||shape="rect" style="text-decoration: none;"]]. Users may apply for science accounts to analyze data measured at Petra III or Flash on resources located at the DESY computer center. Science accounts have a standard lifetime of 3 years (renewable).
... ... @@ -29,7 +29,7 @@
29 29  For accessing the DESY computers a two-factor authentication is needed. After you have received your Science Account, you will need to setup an Authenticator App to your smartphone. You can find the instructions [[here>>url:https://it.desy.de/services/mfa/external_people/index_eng.html||shape="rect" style="text-decoration: none;"]].
30 30  
31 31  (% style="text-decoration: none;" %)
32 -== {{id name="11.Dataprocessing-AccessingData"/}}Accessing Data ==
32 +== Accessing Data ==
33 33  
34 34  (% style="text-align: justify;" %)
35 35  There are three methods to access DESY computers from outside using a scientific account:
... ... @@ -75,7 +75,7 @@
75 75  Data is accessible to users who were registered participants to the experiment in Door prior to the beamtime ID being opened at the beamline. If you cannot access the data, please contact your LC or the beamline manager.
76 76  
77 77  (% style="text-decoration: none;" %)
78 -== {{id name="11.Dataprocessing-Dataprocessing"/}}Data processing ==
78 +== Data processing ==
79 79  
80 80  (% style="text-align: justify;" %)
81 81  The processing folders contain two folders, 'xdsapp' and 'manual'. 'Manual' contains a template for manual processing having all the correct parameters. However, to shorten the processing time and make use of the computational structure, reprocessing your data on the maxwell nodes should be done by using a script to queue the jobs in slurm:
... ... @@ -106,13 +106,13 @@
106 106  * HOX! terminate the allocation after usage by typing exit twice on your terminal window
107 107  
108 108  (% style="text-decoration: none;" %)
109 -== {{id name="11.Dataprocessing-CrystallographicandScientificSoftwarePackages"/}}Crystallographic and Scientific Software Packages ==
109 +== Crystallographic and Scientific Software Packages ==
110 110  
111 111  (% style="text-align: justify;" %)
112 -Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on [[confluence>>doc:IS.Software.WebHome||shape="rect" style="text-decoration: none;"]].
112 +Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on [[confluence>>https://confluence.desy.de/display/IS/Software||shape="rect" style="text-decoration: none;"]].
113 113  
114 114  (% style="text-align: justify;" %)
115 -Crystallographic software packages are in the [[Photon Science section>>doc:MXW.Photon Science.WebHome||shape="rect"]].
115 +Crystallographic software packages are in the [[Photon Science section>>https://confluence.desy.de/display/MXW/Photon+Science||shape="rect"]].
116 116  
117 117  (% style="text-align: justify;" %)
118 118  You can store analysis results in scratch_cc (temporary/testing results) or processed (final results).