Skip to Content

Changes for page 06. Data Backup

Last modified by tabermah on 2024-12-10 14:54
From version 18.2
edited by tabermah
on 2024-12-02 15:49
Change comment: There is no comment for this version
To version 14.1
edited by hakanj
on 2022-08-23 15:49
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.tabermah
1 +XWiki.hakanj
Tags
... ... @@ -1,1 +1,0 @@
1 -favourite
Content
... ... @@ -1,25 +1,32 @@
1 -= 12.1. Backup to USB-drive during beamtime =
1 +**12.1. Backup to USB-drive during beamtime**
2 2  
3 3  If you bring a USB-drive, you may sync the data onto it during the beamtime. We provide a dedicated PC for this purpose, which is running debian. It is advisable to bring a drive which is known to work fast on Linux systems, as some chipsets might be too slow to keep up with the speed at which you take data. Please also note, that ExFAT is currently not supported. FAT32, NTFS, ext4 should all work fine.
4 4  
5 -There is a script prepared, which will continuously sync the data onto your hard-drive.  Source and destination path have to be adapted beforehand. Your local contact will help you with this. I(% style="color:#172b4d" %)nstructions(%%) can also be found (% style="color:#172b4d" %)in the hutch.
5 +There is a script prepared, which will continuously sync the data onto your hard-drive.  Source and destination path have to be adapted beforehand. Your local contact will help you with this. I(% style="color: rgb(23,43,77);" %)nstructions(%%) can also be found (% style="color: rgb(23,43,77);" %)in the hutch.
6 6  
7 7  There are two possibilities as where to copy your data from:
8 8  
9 -* The core file system /asap3/petra3/gpfs/p11/<YEAR>/data/<BEAMTIME>. This is the way we recommend as it has several advantages. You will need to log in with a (% style="color:#172b4d" %) (%%)[[DESY Science Account>>https://xwiki.desy.de/xwiki/bin/view/FSEC/FS-IT%20Topics/Science%20Accounts/||rel="nofollow" shape="rect" style="text-decoration: none;"]] which was added to the beamtimes authorized list by the PI(% style="color:#172b4d" %).
10 -* (% style="color:#172b4d" %)The beamline file system /gpfs/current/. You may log in as p11user to do this. Any data resulting from manual processing done on the Maxwell resource will not be visible here and thus not be contained in your backup. As soon as the next user group starts and your beamtime is closed, you will have to stop copying.
9 +* The core file system /asap3/petra3/gpfs/p11/<YEAR>/data/<BEAMTIME>. This is the way we recommend as it has several advantages. You will need to log in with a (% style="color: rgb(23,43,77);" %) (%%)[[DESY Science Account>>doc:FSEC.Science Accounts.WebHome||style="text-decoration: none;" rel="nofollow" shape="rect"]] which was added to the beamtimes authorized list by the PI(% style="color: rgb(23,43,77);" %).
10 +* (% style="color: rgb(23,43,77);" %)The beamline file system /gpfs/current/. You may log in as p11user to do this. Any data resulting from manual processing done on the Maxwell resource will not be visible here and thus not be contained in your backup. As soon as the next user group starts and your beamtime is closed, you will have to stop copying.
11 11  
12 -= 12.2. Download your data during or after the beamtime via FTP =
12 +\\
13 13  
14 -Data are available for download via FTP protocol through the [[Gamma Portal>>https://docs.desy.de/asap3/gamma-portal/||shape="rect"]] as it is described in [[here>>https://docs.desy.de/asap3/gamma-portal/accessing-beamtime-data-using-sftp/||shape="rect"]].
14 +**12.2. Download your data during or after the beamtime via FTP**
15 15  
16 -= 12.3. Download your data during or after the beamtime using Globus =
16 +Data are available for download via FTP protocol through the [[Gamma Portal>>url:https://gamma-portal.desy.de/||shape="rect"]] as it is described in [[here>>doc:ASAP3.The Gamma Portal.WebHome||shape="rect"]].
17 17  
18 -1. A third option is to use [[Globus Online>>url:https://www.globus.org/data-transfer]]: this is available exclusively for Users with Science Accounts and an Institute Subscription to the Globus Group:
19 -1*. check with your local administrator if your institute is a subscriber of the Globus Group. If so, log in into [[Globus Online>>url:https://app.globus.org/]] by choosing your Organization and Identity Provider
20 -1*. create an Endpoint on your machine by downloading the suitable "Globus Connect" for you operating system
21 -1*. choose DESY in the "Collection Search" and browse to your Beamtime folder
22 -1*. start transferring data in the file manager section - further instructions can be found on [[FS-EC's confluence page>>url:https://confluence.desy.de/display/FSEC/Globus+Online]]
18 +Alternative URL:
19 +(% class="Object" %)[[https:~~/~~/ias.desy.de/pls/apex/f?p=260:1>>url:https://ias.desy.de/pls/apex/f?p=260:1||shape="rect"]](%%)
20 +\\
23 23  
22 +\\
24 24  
25 -
24 +**12.3. Download your data during or after the beamtime using Globus**
25 +
26 +[[https:~~/~~/www.globus.org/data-transfer>>url:https://www.globus.org/data-transfer||shape="rect"]]
27 +
28 +todo: Infos about our endpoint and how to use it.
29 +
30 +\\
31 +
32 +\\
Confluence.Code.ConfluencePageClass[0]
Id
... ... @@ -1,1 +1,1 @@
1 -175930706
1 +272734567
URL
... ... @@ -1,1 +1,1 @@
1 -https://confluence.desy.de/spaces/11/pages/175930706/12. Data Backup
1 +https://confluence.desy.de/spaces/11/pages/272734567/12. Data Backup