Skip to Content

Wiki source code of DEMO - Working with FLASH data

Version 38.1 by sndueste on 2023-09-29 11:15
Hide last authors
sndueste 37.1 1 Experimental data is recorded as HDF files[[~[link~]>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] on the GPFS file system[[~[link~]>>doc:ASAP3.ASAP3 Data Storage for PETRA III]]. The access rights are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[[~[link~]>>url:https://gamma-portal.desy.de/||shape="rect"]]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh, Maxwell-Display Server[[~[link~]>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] or JuyterHub[[~[link~]>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]]. We recommend using the JupyterHub for data exploration and the SLURM resources for high performances computing - see FAB for easy usage of the infrastructure.
cpassow 21.1 2
sndueste 38.1 3 {{info title="How to login JupyterHub"}}
4 === ===
cpassow 26.1 5
sndueste 38.1 6 {{view-file att--filename="tmp.mp4" height="150"/}}
7 {{/info}}
cpassow 26.1 8
9 \\
10
sndueste 35.1 11 {{info}}
sndueste 36.1 12 === There are different options that help you to work with the FLASH HDF5 data in Python ===
sndueste 35.1 13
sndueste 36.1 14 The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
sndueste 35.1 15
sndueste 36.1 16 and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
sndueste 35.1 17 {{/info}}
18
cpassow 26.1 19 \\
20
sndueste 36.1 21 [[~[~[image:attach:image2023-9-29_11-1-37.png~]~]>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]]
sndueste 33.1 22
23 \\
24
sndueste 34.1 25 {{expand title="older ideas ..."}}
sndueste 36.1 26 (% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
sndueste 33.1 27
cpassow 21.1 28 ----
29
30 === TODO ===
31
cpassow 22.1 32 (% class="task-list" %)
33 (((
34 {{task reference="/Tasks/Task_18" status="InProgress"}}
35 Short descriptions including Links:   → as Text\\
cpassow 2.1 36
cpassow 22.1 37 (% class="task-list" %)
38 (((
39
40
41 {{task reference="/Tasks/Task_19" status="InProgress"}}
42 GPFS
43 {{/task}}
44
cpassow 27.1 45 {{task reference="/Tasks/Task_40" status="InProgress"}}
46 Access rights
47 {{/task}}
48
49 {{task reference="/Tasks/Task_41" status="InProgress"}}
50 Gamma Portal
51 {{/task}}
52
53 {{task reference="/Tasks/Task_42" status="InProgress"}}
54 Maxwell
55 {{/task}}
56
sndueste 32.1 57 {{task reference="/Tasks/Task_20" status="InProgress"}}
cpassow 22.1 58 JupyterHub
59 {{/task}}
60
61 {{task reference="/Tasks/Task_21" status="InProgress"}}
62 conda ?
63 {{/task}}
cpassow 23.1 64
65 {{task reference="/Tasks/Task_36" status="InProgress"}}
66 explain install from channel instead of fixed environment, but can use environment file from example repository
67 {{/task}}
cpassow 22.1 68 )))
69 {{/task}}
70 )))
71
cpassow 8.1 72 \\
73
cpassow 22.1 74 (% class="task-list" %)
75 (((
76 {{task reference="/Tasks/Task_22" status="InProgress"}}
77 distribution
cpassow 21.1 78
cpassow 22.1 79 (% class="task-list" %)
80 (((
81
82
83 {{task reference="/Tasks/Task_23" status="InProgress"}}
84 channel  (where to host?)
85 {{/task}}
86
87 {{task reference="/Tasks/Task_24" status="InProgress"}}
88 environment file (repository with examples)
89 {{/task}}
90 )))
91 {{/task}}
92 )))
93
cpassow 21.1 94 \\
95
cpassow 22.1 96 (% class="task-list" %)
97 (((
98 {{task reference="/Tasks/Task_25" status="InProgress"}}
99 Documentation
cpassow 2.1 100
cpassow 22.1 101 (% class="task-list" %)
102 (((
103
104
105 {{task reference="/Tasks/Task_26" status="InProgress"}}
106 here VS repository VS sphinx
107 {{/task}}
108 )))
109 {{/task}}
110 )))
111
cpassow 2.1 112 \\
113
cpassow 22.1 114 (% class="task-list" %)
115 (((
116 {{task reference="/Tasks/Task_27" status="InProgress"}}
117 Binder
cpassow 2.1 118
cpassow 22.1 119 (% class="task-list" %)
120 (((
121
122
123 {{task reference="/Tasks/Task_28" status="InProgress"}}
124 examples with Stefan
125 {{/task}}
126 )))
127 {{/task}}
128 )))
129
cpassow 2.1 130 \\
131
cpassow 22.1 132 (% class="task-list" %)
133 (((
134 {{task reference="/Tasks/Task_29" status="InProgress"}}
135 Screencast
cpassow 15.1 136
cpassow 22.1 137 (% class="task-list" %)
138 (((
139
140
141 {{task reference="/Tasks/Task_30" status="InProgress"}}
142 login in jupyter hub
143 {{/task}}
144
145 {{task reference="/Tasks/Task_31" status="InProgress"}}
146 login maxwell display
147 {{/task}}
148
149 {{task reference="/Tasks/Task_32" status="InProgress"}}
150 use slix
151 {{/task}}
152
153 {{task reference="/Tasks/Task_33" status="InProgress"}}
154 use hdfview plugin in jupterLab
155 {{/task}}
156
157 {{task reference="/Tasks/Task_34" status="InProgress"}}
158 create conda env with flashh5
159 {{/task}}
160 )))
161 {{/task}}
162 )))
163
cpassow 2.1 164 \\
165
cpassow 15.1 166 \\
167
cpassow 21.1 168 ----
cpassow 2.1 169
cpassow 21.1 170 ==== under review ====
cpassow 4.1 171
172 \\
173
cpassow 8.1 174 {{code language="bash"}}
cpassow 9.1 175 conda create -n flashh5 python=3.10 # 3.10 not necessary, but would prefer 3.8 or higher
cpassow 8.1 176 source activate flashh5
cpassow 10.1 177 conda install ipython numpy pandas #TODO: fix dependcies
cpassow 8.1 178 conda install -c https://www.desy.de/~cpassow/condarepo/ flashh5
179
180 ## on jhub
cpassow 11.1 181 conda install ipykernel
cpassow 8.1 182 python -m ipykernel install --user --name flashh5 --display-name "flashh5"
183
184
cpassow 11.1 185 ## to remove on jhub
186 ## delete from: /home/$USER/.local/share/jupyter/kernels/
cpassow 8.1 187 {{/code}}
188
189 \\
190
cpassow 6.1 191 {{code language="py" title="moved to repository?"}}
cpassow 4.1 192 class RunDirectory:
193
cpassow 21.1 194 def get_run_table(): # more or less information? RunComment | Number of Files | start & stop time ?
cpassow 4.1 195 ...
196
cpassow 21.1 197 def get_run(daq, run_number): # daq is not needed!
198
cpassow 4.1 199 ...
200
201
cpassow 25.1 202 class Run: # constructor optional without RunDirectory or use there self.path
cpassow 4.1 203
cpassow 7.1 204 def get_files():
cpassow 4.1 205 ...
206
cpassow 18.1 207 def get_channels(): # of file #1
cpassow 4.1 208 ...
209
cpassow 8.1 210 def get_start_time(): # better as attribute?
cpassow 4.1 211 ...
212
cpassow 8.1 213 def get_stop_time(): # which? | better as attribute?
cpassow 4.1 214 ...
cpassow 12.1 215
cpassow 19.1 216 def to_df(daq_map): # to_df(daq_map, slice) slice=[0:4] -> throw Exception
cpassow 4.1 217 ...
218
cpassow 7.1 219 def to_series(channel):
cpassow 4.1 220 ...
221
cpassow 7.1 222 def to_array(channel):
cpassow 4.1 223 ...
224 {{/code}}
225
cpassow 6.1 226 {{code language="py" title="ideas"}}
cpassow 10.1 227 run.to_df(daq_map)
228 run.to_series(daq_adr or daq_map) # on channel only?
229 run.to_array(daq_adr) # on channel only?
cpassow 5.1 230
cpassow 21.1 231 ## interesting?
cpassow 10.1 232 # run.to_dask(daq_map)
233 # run.to_xarray(daq_map)
cpassow 5.1 234 {{/code}}
235
cpassow 4.1 236 \\
cpassow 30.1 237
238 \\
sndueste 33.1 239 {{/expand}}