Skip to Content

Wiki source code of DEMO - Working with FLASH data

Version 37.1 by sndueste on 2023-09-29 11:13
Show last authors
1 Experimental data is recorded as HDF files[[~[link~]>>doc:FLASHUSER.Data Acquisition and controls.Data Access at FLASH (DAQ, gpfs,\.\.\.).Offline data analysis (DAQ).The FLASH HDF5 structure.WebHome]] on the GPFS file system[[~[link~]>>doc:ASAP3.ASAP3 Data Storage for PETRA III]]. The access rights are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[[~[link~]>>url:https://gamma-portal.desy.de/||shape="rect"]]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh, Maxwell-Display Server[[~[link~]>>url:https://confluence.desy.de/display/MXW/Maxwell+Cluster||shape="rect"]] or JuyterHub[[~[link~]>>url:https://confluence.desy.de/display/MXW/JupyterHub+on+Maxwell||shape="rect"]]. We recommend using the JupyterHub for data exploration and the SLURM resources for high performances computing - see FAB for easy usage of the infrastructure.
2
3 \\
4
5 {{expand title="How to login JupyterHub"}}
6
7
8 {{view-file att--filename="tmp.mp4" height="250"/}}
9 {{/expand}}
10
11 \\
12
13 \\
14
15 {{info}}
16 === There are different options that help you to work with the FLASH HDF5 data in Python ===
17
18 The currently developed option for large data sets: [[the FAB package>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]] ... see below
19
20 and for smaller projects:  (% class="Object" %)[[https:~~/~~/gitlab.desy.de/christopher.passow/flash-daq-hdf>>url:https://gitlab.desy.de/christopher.passow/flash-daq-hdf||shape="rect"]]
21 {{/info}}
22
23 \\
24
25 [[~[~[image:attach:image2023-9-29_11-1-37.png~]~]>>url:https://hasfcpkg.desy.de/fab/fab.html||shape="rect"]]
26
27 \\
28
29 {{expand title="older ideas ..."}}
30 (% class="Object" %)(object oriented) [[https:~~/~~/gitlab.desy.de/christopher.passow/fdh-builder>>url:https://gitlab.desy.de/christopher.passow/fdh-builder.git||shape="rect"]]
31
32 ----
33
34 === TODO ===
35
36 (% class="task-list" %)
37 (((
38 {{task reference="/Tasks/Task_18" status="InProgress"}}
39 Short descriptions including Links:   → as Text\\
40
41 (% class="task-list" %)
42 (((
43
44
45 {{task reference="/Tasks/Task_19" status="InProgress"}}
46 GPFS
47 {{/task}}
48
49 {{task reference="/Tasks/Task_40" status="InProgress"}}
50 Access rights
51 {{/task}}
52
53 {{task reference="/Tasks/Task_41" status="InProgress"}}
54 Gamma Portal
55 {{/task}}
56
57 {{task reference="/Tasks/Task_42" status="InProgress"}}
58 Maxwell
59 {{/task}}
60
61 {{task reference="/Tasks/Task_20" status="InProgress"}}
62 JupyterHub
63 {{/task}}
64
65 {{task reference="/Tasks/Task_21" status="InProgress"}}
66 conda ?
67 {{/task}}
68
69 {{task reference="/Tasks/Task_36" status="InProgress"}}
70 explain install from channel instead of fixed environment, but can use environment file from example repository
71 {{/task}}
72 )))
73 {{/task}}
74 )))
75
76 \\
77
78 (% class="task-list" %)
79 (((
80 {{task reference="/Tasks/Task_22" status="InProgress"}}
81 distribution
82
83 (% class="task-list" %)
84 (((
85
86
87 {{task reference="/Tasks/Task_23" status="InProgress"}}
88 channel  (where to host?)
89 {{/task}}
90
91 {{task reference="/Tasks/Task_24" status="InProgress"}}
92 environment file (repository with examples)
93 {{/task}}
94 )))
95 {{/task}}
96 )))
97
98 \\
99
100 (% class="task-list" %)
101 (((
102 {{task reference="/Tasks/Task_25" status="InProgress"}}
103 Documentation
104
105 (% class="task-list" %)
106 (((
107
108
109 {{task reference="/Tasks/Task_26" status="InProgress"}}
110 here VS repository VS sphinx
111 {{/task}}
112 )))
113 {{/task}}
114 )))
115
116 \\
117
118 (% class="task-list" %)
119 (((
120 {{task reference="/Tasks/Task_27" status="InProgress"}}
121 Binder
122
123 (% class="task-list" %)
124 (((
125
126
127 {{task reference="/Tasks/Task_28" status="InProgress"}}
128 examples with Stefan
129 {{/task}}
130 )))
131 {{/task}}
132 )))
133
134 \\
135
136 (% class="task-list" %)
137 (((
138 {{task reference="/Tasks/Task_29" status="InProgress"}}
139 Screencast
140
141 (% class="task-list" %)
142 (((
143
144
145 {{task reference="/Tasks/Task_30" status="InProgress"}}
146 login in jupyter hub
147 {{/task}}
148
149 {{task reference="/Tasks/Task_31" status="InProgress"}}
150 login maxwell display
151 {{/task}}
152
153 {{task reference="/Tasks/Task_32" status="InProgress"}}
154 use slix
155 {{/task}}
156
157 {{task reference="/Tasks/Task_33" status="InProgress"}}
158 use hdfview plugin in jupterLab
159 {{/task}}
160
161 {{task reference="/Tasks/Task_34" status="InProgress"}}
162 create conda env with flashh5
163 {{/task}}
164 )))
165 {{/task}}
166 )))
167
168 \\
169
170 \\
171
172 ----
173
174 ==== under review ====
175
176 \\
177
178 {{code language="bash"}}
179 conda create -n flashh5 python=3.10 # 3.10 not necessary, but would prefer 3.8 or higher
180 source activate flashh5
181 conda install ipython numpy pandas #TODO: fix dependcies
182 conda install -c https://www.desy.de/~cpassow/condarepo/ flashh5
183
184 ## on jhub
185 conda install ipykernel
186 python -m ipykernel install --user --name flashh5 --display-name "flashh5"
187
188
189 ## to remove on jhub
190 ## delete from: /home/$USER/.local/share/jupyter/kernels/
191 {{/code}}
192
193 \\
194
195 {{code language="py" title="moved to repository?"}}
196 class RunDirectory:
197
198 def get_run_table(): # more or less information? RunComment | Number of Files | start & stop time ?
199 ...
200
201 def get_run(daq, run_number): # daq is not needed!
202
203 ...
204
205
206 class Run: # constructor optional without RunDirectory or use there self.path
207
208 def get_files():
209 ...
210
211 def get_channels(): # of file #1
212 ...
213
214 def get_start_time(): # better as attribute?
215 ...
216
217 def get_stop_time(): # which? | better as attribute?
218 ...
219
220 def to_df(daq_map): # to_df(daq_map, slice) slice=[0:4] -> throw Exception
221 ...
222
223 def to_series(channel):
224 ...
225
226 def to_array(channel):
227 ...
228 {{/code}}
229
230 {{code language="py" title="ideas"}}
231 run.to_df(daq_map)
232 run.to_series(daq_adr or daq_map) # on channel only?
233 run.to_array(daq_adr) # on channel only?
234
235 ## interesting?
236 # run.to_dask(daq_map)
237 # run.to_xarray(daq_map)
238 {{/code}}
239
240 \\
241
242 \\
243 {{/expand}}