Skip to Content

Wiki source code of DEMO - Working with FLASH data

Version 37.1 by sndueste on 2023-09-29 11:13
Hide last authors
sndueste 37.1 1 Experimental data is recorded as HDF files[[~[link~]>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] on the GPFS file system[[~[link~]>>doc:ASAP3.ASAP3 Data Storage for PETRA III]]. The access rights are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[[~[link~]>>url:https://gamma-portal.desy.de/||shape="rect"]]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh, Maxwell-Display Server[[~[link~]>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] or JuyterHub[[~[link~]>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]]. We recommend using the JupyterHub for data exploration and the SLURM resources for high performances computing - see FAB for easy usage of the infrastructure.
cpassow 21.1 2
cpassow 26.1 3 \\
4
sndueste 33.1 5 {{expand title="How to login JupyterHub"}}
cpassow 26.1 6
7
8 {{view-file att--filename="tmp.mp4" height="250"/}}
9 {{/expand}}
10
11 \\
12
sndueste 36.1 13 \\
14
sndueste 35.1 15 {{info}}
sndueste 36.1 16 === There are different options that help you to work with the FLASH HDF5 data in Python ===
sndueste 35.1 17
sndueste 36.1 18 The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
sndueste 35.1 19
sndueste 36.1 20 and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
sndueste 35.1 21 {{/info}}
22
cpassow 26.1 23 \\
24
sndueste 36.1 25 [[~[~[image:attach:image2023-9-29_11-1-37.png~]~]>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]]
sndueste 33.1 26
27 \\
28
sndueste 34.1 29 {{expand title="older ideas ..."}}
sndueste 36.1 30 (% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
sndueste 33.1 31
cpassow 21.1 32 ----
33
34 === TODO ===
35
cpassow 22.1 36 (% class="task-list" %)
37 (((
38 {{task reference="/Tasks/Task_18" status="InProgress"}}
39 Short descriptions including Links:   → as Text\\
cpassow 2.1 40
cpassow 22.1 41 (% class="task-list" %)
42 (((
43
44
45 {{task reference="/Tasks/Task_19" status="InProgress"}}
46 GPFS
47 {{/task}}
48
cpassow 27.1 49 {{task reference="/Tasks/Task_40" status="InProgress"}}
50 Access rights
51 {{/task}}
52
53 {{task reference="/Tasks/Task_41" status="InProgress"}}
54 Gamma Portal
55 {{/task}}
56
57 {{task reference="/Tasks/Task_42" status="InProgress"}}
58 Maxwell
59 {{/task}}
60
sndueste 32.1 61 {{task reference="/Tasks/Task_20" status="InProgress"}}
cpassow 22.1 62 JupyterHub
63 {{/task}}
64
65 {{task reference="/Tasks/Task_21" status="InProgress"}}
66 conda ?
67 {{/task}}
cpassow 23.1 68
69 {{task reference="/Tasks/Task_36" status="InProgress"}}
70 explain install from channel instead of fixed environment, but can use environment file from example repository
71 {{/task}}
cpassow 22.1 72 )))
73 {{/task}}
74 )))
75
cpassow 8.1 76 \\
77
cpassow 22.1 78 (% class="task-list" %)
79 (((
80 {{task reference="/Tasks/Task_22" status="InProgress"}}
81 distribution
cpassow 21.1 82
cpassow 22.1 83 (% class="task-list" %)
84 (((
85
86
87 {{task reference="/Tasks/Task_23" status="InProgress"}}
88 channel  (where to host?)
89 {{/task}}
90
91 {{task reference="/Tasks/Task_24" status="InProgress"}}
92 environment file (repository with examples)
93 {{/task}}
94 )))
95 {{/task}}
96 )))
97
cpassow 21.1 98 \\
99
cpassow 22.1 100 (% class="task-list" %)
101 (((
102 {{task reference="/Tasks/Task_25" status="InProgress"}}
103 Documentation
cpassow 2.1 104
cpassow 22.1 105 (% class="task-list" %)
106 (((
107
108
109 {{task reference="/Tasks/Task_26" status="InProgress"}}
110 here VS repository VS sphinx
111 {{/task}}
112 )))
113 {{/task}}
114 )))
115
cpassow 2.1 116 \\
117
cpassow 22.1 118 (% class="task-list" %)
119 (((
120 {{task reference="/Tasks/Task_27" status="InProgress"}}
121 Binder
cpassow 2.1 122
cpassow 22.1 123 (% class="task-list" %)
124 (((
125
126
127 {{task reference="/Tasks/Task_28" status="InProgress"}}
128 examples with Stefan
129 {{/task}}
130 )))
131 {{/task}}
132 )))
133
cpassow 2.1 134 \\
135
cpassow 22.1 136 (% class="task-list" %)
137 (((
138 {{task reference="/Tasks/Task_29" status="InProgress"}}
139 Screencast
cpassow 15.1 140
cpassow 22.1 141 (% class="task-list" %)
142 (((
143
144
145 {{task reference="/Tasks/Task_30" status="InProgress"}}
146 login in jupyter hub
147 {{/task}}
148
149 {{task reference="/Tasks/Task_31" status="InProgress"}}
150 login maxwell display
151 {{/task}}
152
153 {{task reference="/Tasks/Task_32" status="InProgress"}}
154 use slix
155 {{/task}}
156
157 {{task reference="/Tasks/Task_33" status="InProgress"}}
158 use hdfview plugin in jupterLab
159 {{/task}}
160
161 {{task reference="/Tasks/Task_34" status="InProgress"}}
162 create conda env with flashh5
163 {{/task}}
164 )))
165 {{/task}}
166 )))
167
cpassow 2.1 168 \\
169
cpassow 15.1 170 \\
171
cpassow 21.1 172 ----
cpassow 2.1 173
cpassow 21.1 174 ==== under review ====
cpassow 4.1 175
176 \\
177
cpassow 8.1 178 {{code language="bash"}}
cpassow 9.1 179 conda create -n flashh5 python=3.10 # 3.10 not necessary, but would prefer 3.8 or higher
cpassow 8.1 180 source activate flashh5
cpassow 10.1 181 conda install ipython numpy pandas #TODO: fix dependcies
cpassow 8.1 182 conda install -c https://www.desy.de/~cpassow/condarepo/ flashh5
183
184 ## on jhub
cpassow 11.1 185 conda install ipykernel
cpassow 8.1 186 python -m ipykernel install --user --name flashh5 --display-name "flashh5"
187
188
cpassow 11.1 189 ## to remove on jhub
190 ## delete from: /home/$USER/.local/share/jupyter/kernels/
cpassow 8.1 191 {{/code}}
192
193 \\
194
cpassow 6.1 195 {{code language="py" title="moved to repository?"}}
cpassow 4.1 196 class RunDirectory:
197
cpassow 21.1 198 def get_run_table(): # more or less information? RunComment | Number of Files | start & stop time ?
cpassow 4.1 199 ...
200
cpassow 21.1 201 def get_run(daq, run_number): # daq is not needed!
202
cpassow 4.1 203 ...
204
205
cpassow 25.1 206 class Run: # constructor optional without RunDirectory or use there self.path
cpassow 4.1 207
cpassow 7.1 208 def get_files():
cpassow 4.1 209 ...
210
cpassow 18.1 211 def get_channels(): # of file #1
cpassow 4.1 212 ...
213
cpassow 8.1 214 def get_start_time(): # better as attribute?
cpassow 4.1 215 ...
216
cpassow 8.1 217 def get_stop_time(): # which? | better as attribute?
cpassow 4.1 218 ...
cpassow 12.1 219
cpassow 19.1 220 def to_df(daq_map): # to_df(daq_map, slice) slice=[0:4] -> throw Exception
cpassow 4.1 221 ...
222
cpassow 7.1 223 def to_series(channel):
cpassow 4.1 224 ...
225
cpassow 7.1 226 def to_array(channel):
cpassow 4.1 227 ...
228 {{/code}}
229
cpassow 6.1 230 {{code language="py" title="ideas"}}
cpassow 10.1 231 run.to_df(daq_map)
232 run.to_series(daq_adr or daq_map) # on channel only?
233 run.to_array(daq_adr) # on channel only?
cpassow 5.1 234
cpassow 21.1 235 ## interesting?
cpassow 10.1 236 # run.to_dask(daq_map)
237 # run.to_xarray(daq_map)
cpassow 5.1 238 {{/code}}
239
cpassow 4.1 240 \\
cpassow 30.1 241
242 \\
sndueste 33.1 243 {{/expand}}