Skip to Content

Wiki source code of DEMO - Working with FLASH data

Version 33.1 by sndueste on 2023-09-28 16:51
Show last authors
1 Experimental data is recorded as HDF files[link] on the GPFS file system[link]. The access rights[link to ACLs] are linked to the user's DESY account and can be managed by the PI via the GAMMA portal[link]. The experimental data can be downloaded via the GAMMA portal, but it is advised to use the DESY computing infrastructure. Access point are via ssh[link], Maxwell-Display Server[link] or JuyterHub[link]. We recommmend using the JupyterHub for data exploration and the SLURM resources[link] for high performances computing.
2
3 For simplified acccess we provide a conda module flashh5[link] which can be installed in a personal conda environment[link] on the Maxwell Cluster. Example on the usage can be found here [link - repo + binder]
4
5 \\
6
7 {{expand title="How to login JupyterHub"}}
8
9
10 {{view-file att--filename="tmp.mp4" height="250"/}}
11 {{/expand}}
12
13 \\
14
15 link to Repos and Fab
16
17 \\
18
19 \\
20
21 \\
22
23 {{expand}}
24 \\
25
26 ----
27
28 === TODO ===
29
30 (% class="task-list" %)
31 (((
32 {{task reference="/Tasks/Task_18" status="InProgress"}}
33 Short descriptions including Links:   → as Text\\
34
35 (% class="task-list" %)
36 (((
37
38
39 {{task reference="/Tasks/Task_19" status="InProgress"}}
40 GPFS
41 {{/task}}
42
43 {{task reference="/Tasks/Task_40" status="InProgress"}}
44 Access rights
45 {{/task}}
46
47 {{task reference="/Tasks/Task_41" status="InProgress"}}
48 Gamma Portal
49 {{/task}}
50
51 {{task reference="/Tasks/Task_42" status="InProgress"}}
52 Maxwell
53 {{/task}}
54
55 {{task reference="/Tasks/Task_20" status="InProgress"}}
56 JupyterHub
57 {{/task}}
58
59 {{task reference="/Tasks/Task_21" status="InProgress"}}
60 conda ?
61 {{/task}}
62
63 {{task reference="/Tasks/Task_36" status="InProgress"}}
64 explain install from channel instead of fixed environment, but can use environment file from example repository
65 {{/task}}
66 )))
67 {{/task}}
68 )))
69
70 \\
71
72 (% class="task-list" %)
73 (((
74 {{task reference="/Tasks/Task_22" status="InProgress"}}
75 distribution
76
77 (% class="task-list" %)
78 (((
79
80
81 {{task reference="/Tasks/Task_23" status="InProgress"}}
82 channel  (where to host?)
83 {{/task}}
84
85 {{task reference="/Tasks/Task_24" status="InProgress"}}
86 environment file (repository with examples)
87 {{/task}}
88 )))
89 {{/task}}
90 )))
91
92 \\
93
94 (% class="task-list" %)
95 (((
96 {{task reference="/Tasks/Task_25" status="InProgress"}}
97 Documentation
98
99 (% class="task-list" %)
100 (((
101
102
103 {{task reference="/Tasks/Task_26" status="InProgress"}}
104 here VS repository VS sphinx
105 {{/task}}
106 )))
107 {{/task}}
108 )))
109
110 \\
111
112 (% class="task-list" %)
113 (((
114 {{task reference="/Tasks/Task_27" status="InProgress"}}
115 Binder
116
117 (% class="task-list" %)
118 (((
119
120
121 {{task reference="/Tasks/Task_28" status="InProgress"}}
122 examples with Stefan
123 {{/task}}
124 )))
125 {{/task}}
126 )))
127
128 \\
129
130 (% class="task-list" %)
131 (((
132 {{task reference="/Tasks/Task_29" status="InProgress"}}
133 Screencast
134
135 (% class="task-list" %)
136 (((
137
138
139 {{task reference="/Tasks/Task_30" status="InProgress"}}
140 login in jupyter hub
141 {{/task}}
142
143 {{task reference="/Tasks/Task_31" status="InProgress"}}
144 login maxwell display
145 {{/task}}
146
147 {{task reference="/Tasks/Task_32" status="InProgress"}}
148 use slix
149 {{/task}}
150
151 {{task reference="/Tasks/Task_33" status="InProgress"}}
152 use hdfview plugin in jupterLab
153 {{/task}}
154
155 {{task reference="/Tasks/Task_34" status="InProgress"}}
156 create conda env with flashh5
157 {{/task}}
158 )))
159 {{/task}}
160 )))
161
162 \\
163
164 \\
165
166 ----
167
168 ==== under review ====
169
170 \\
171
172 {{code language="bash"}}
173 conda create -n flashh5 python=3.10 # 3.10 not necessary, but would prefer 3.8 or higher
174 source activate flashh5
175 conda install ipython numpy pandas #TODO: fix dependcies
176 conda install -c https://www.desy.de/~cpassow/condarepo/ flashh5
177
178 ## on jhub
179 conda install ipykernel
180 python -m ipykernel install --user --name flashh5 --display-name "flashh5"
181
182
183 ## to remove on jhub
184 ## delete from: /home/$USER/.local/share/jupyter/kernels/
185 {{/code}}
186
187 \\
188
189 {{code language="py" title="moved to repository?"}}
190 class RunDirectory:
191
192 def get_run_table(): # more or less information? RunComment | Number of Files | start & stop time ?
193 ...
194
195 def get_run(daq, run_number): # daq is not needed!
196
197 ...
198
199
200 class Run: # constructor optional without RunDirectory or use there self.path
201
202 def get_files():
203 ...
204
205 def get_channels(): # of file #1
206 ...
207
208 def get_start_time(): # better as attribute?
209 ...
210
211 def get_stop_time(): # which? | better as attribute?
212 ...
213
214 def to_df(daq_map): # to_df(daq_map, slice) slice=[0:4] -> throw Exception
215 ...
216
217 def to_series(channel):
218 ...
219
220 def to_array(channel):
221 ...
222 {{/code}}
223
224 {{code language="py" title="ideas"}}
225 run.to_df(daq_map)
226 run.to_series(daq_adr or daq_map) # on channel only?
227 run.to_array(daq_adr) # on channel only?
228
229 ## interesting?
230 # run.to_dask(daq_map)
231 # run.to_xarray(daq_map)
232 {{/code}}
233
234 \\
235
236 \\
237 {{/expand}}
238
239 \\