Skip to Content
Last modified by makuadm on 2026-01-07 06:21
From version 36.1
edited by sndueste
on 2023-09-29 11:07
Change comment: There is no comment for this version
To version 45.1
edited by makuadm
on 2026-01-06 21:19
Change comment: Task [xwiki:FLASHUSER.Offline data analysis (DAQ).DEMO - Working with FLASH data.Tasks.Task_34.WebHome] has been updated!

Summary

Details

Page properties
Parent
... ... @@ -1,0 +1,1 @@
1 +FLASHUSER.Offline data analysis (DAQ).WebHome
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sndueste
1 +XWiki.makuadm
Content
... ... @@ -1,33 +1,23 @@
1 -Experimental data is recorded as HDF files[link] on the GPFS file system[link]. The access rights[link to ACLs] are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[link]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh[link], Maxwell-Display Server[link] or JuyterHub[link]. We recommmend using the JupyterHub for data exploration and the SLURM resources[link] for high performances computing.
1 +Experimental data is recorded as HDF files[[~[link~]>>doc:FS-FLASH USER tmp.Data Acquisition and controls 1.DAQ and controls overview.Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] on the GPFS file system[[~[link~]>>https://docs.desy.de/asap3/]]. The access rights are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[[~[link~]>>url:https://gamma-portal.desy.de/||shape="rect"]]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh, Maxwell-Display Server[[~[link~]>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] or JuyterHub[[~[link~]>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]]. We recommend using the JupyterHub for data exploration and the SLURM resources for high performances computing - see FAB for easy usage of the infrastructure.
2 2  
3 -For simplified acccess we provide a conda module flashh5[link] which can be installed in a personal conda environment[link] on the Maxwell Cluster. Example on the usage can be found here [link - repo + binder]
3 +{{info title="How to login JupyterHub"}}
4 +=== ===
4 4  
5 -\\
6 +{{view-file att--filename="tmp.mp4" height="150"/}}
7 +{{/info}}
6 6  
7 -{{expand title="How to login JupyterHub"}}
8 -
9 -
10 -{{view-file att--filename="tmp.mp4" height="250"/}}
11 -{{/expand}}
12 -
13 -\\
14 -
15 -\\
16 -
17 17  {{info}}
18 -=== There are different options that help you to work with the FLASH HDF5 data in Python ===
10 +=== There are different options that help you to work with the FLASH HDF5 data in Python ===
19 19  
20 -The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
12 +* The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
13 +* and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
21 21  
22 -and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
15 +(% class="Object" %)See also the collection of Demo data and sample scripts : [[doc:FS-FLASH USER tmp.Data Acquisition and controls 1.DAQ and controls overview.Offline data analysis (DAQ).Collection of HDF5 sample data from different beamlines.WebHome]]
23 23  {{/info}}
24 24  
25 -\\
26 26  
27 27  [[~[~[image:attach:image2023-9-29_11-1-37.png~]~]>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]]
28 28  
29 -\\
30 -
31 31  {{expand title="older ideas ..."}}
32 32  (% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
33 33  
... ... @@ -37,45 +37,12 @@
37 37  
38 38  (% class="task-list" %)
39 39  (((
40 -{{task reference="/Tasks/Task_18" status="InProgress"}}
41 -Short descriptions including Links:   → as Text\\
42 -
43 -(% class="task-list" %)
44 -(((
45 -
46 -
47 -{{task reference="/Tasks/Task_19" status="InProgress"}}
48 -GPFS
30 +{{task reference="/Tasks/Task_18" status="InProgress" completeDate="" createDate="2026-01-06" reporter=""}}
31 +Short descriptions including Links:   → as Text
32 +\\
49 49  {{/task}}
50 -
51 -{{task reference="/Tasks/Task_40" status="InProgress"}}
52 -Access rights
53 -{{/task}}
54 -
55 -{{task reference="/Tasks/Task_41" status="InProgress"}}
56 -Gamma Portal
57 -{{/task}}
58 -
59 -{{task reference="/Tasks/Task_42" status="InProgress"}}
60 -Maxwell
61 -{{/task}}
62 -
63 -{{task reference="/Tasks/Task_20" status="InProgress"}}
64 -JupyterHub
65 -{{/task}}
66 -
67 -{{task reference="/Tasks/Task_21" status="InProgress"}}
68 -conda ?
69 -{{/task}}
70 -
71 -{{task reference="/Tasks/Task_36" status="InProgress"}}
72 -explain install from channel instead of fixed environment, but can use environment file from example repository
73 -{{/task}}
74 74  )))
75 -{{/task}}
76 -)))
77 77  
78 -\\
79 79  
80 80  (% class="task-list" %)
81 81  (((
... ... @@ -97,7 +97,6 @@
97 97  {{/task}}
98 98  )))
99 99  
100 -\\
101 101  
102 102  (% class="task-list" %)
103 103  (((
... ... @@ -115,7 +115,6 @@
115 115  {{/task}}
116 116  )))
117 117  
118 -\\
119 119  
120 120  (% class="task-list" %)
121 121  (((
... ... @@ -133,7 +133,6 @@
133 133  {{/task}}
134 134  )))
135 135  
136 -\\
137 137  
138 138  (% class="task-list" %)
139 139  (((
... ... @@ -156,11 +156,11 @@
156 156  use slix
157 157  {{/task}}
158 158  
159 -{{task reference="/Tasks/Task_33" status="InProgress"}}
113 +{{task reference="/Tasks/Task_33" status="InProgress" completeDate="" createDate="2026-01-06" reporter=""}}
160 160  use hdfview plugin in jupterLab
161 161  {{/task}}
162 162  
163 -{{task reference="/Tasks/Task_34" status="InProgress"}}
117 +{{task reference="/Tasks/Task_34" status="InProgress" completeDate="" createDate="2026-01-06" reporter=""}}
164 164  create conda env with flashh5
165 165  {{/task}}
166 166  )))
... ... @@ -167,15 +167,12 @@
167 167  {{/task}}
168 168  )))
169 169  
170 -\\
171 171  
172 -\\
173 173  
174 174  ----
175 175  
176 176  ==== under review ====
177 177  
178 -\\
179 179  
180 180  {{code language="bash"}}
181 181  conda create -n flashh5 python=3.10 # 3.10 not necessary, but would prefer 3.8 or higher
... ... @@ -192,9 +192,8 @@
192 192  ## delete from: /home/$USER/.local/share/jupyter/kernels/
193 193  {{/code}}
194 194  
195 -\\
196 -
197 -{{code language="py" title="moved to repository?"}}
146 +{{code language="py" title="
147 +moved to repository?"}}
198 198  class RunDirectory:
199 199  
200 200   def get_run_table(): # more or less information? RunComment | Number of Files | start & stop time ?
... ... @@ -239,7 +239,6 @@
239 239  # run.to_xarray(daq_map)
240 240  {{/code}}
241 241  
242 -\\
243 243  
244 -\\
193 +
245 245  {{/expand}}
image2023-9-29_11-0-13.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.rangeadm
Size
... ... @@ -1,0 +1,1 @@
1 +123.1 KB
Content
image2023-9-29_11-1-37.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.rangeadm
Size
... ... @@ -1,0 +1,1 @@
1 +90.4 KB
Content
tmp.mp4
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.rangeadm
Size
... ... @@ -1,0 +1,1 @@
1 +1.1 MB
Content