-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.xml
216 lines (206 loc) · 20.7 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Visual Clutter</title>
<link>https://jadenecke.github.io/</link>
<atom:link href="https://jadenecke.github.io/index.xml" rel="self" type="application/rss+xml" />
<description>Visual Clutter</description>
<generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>2020-2022 Jannis Denecke</copyright><lastBuildDate>Fri, 07 Jan 2022 00:00:00 +0000</lastBuildDate>
<image>
<url>img/map[gravatar:%!s(bool=false) shape:circle]</url>
<title>Visual Clutter</title>
<link>https://jadenecke.github.io/</link>
</image>
<item>
<title>City Maps</title>
<link>https://jadenecke.github.io/project/citymaps/</link>
<pubDate>Fri, 07 Jan 2022 00:00:00 +0000</pubDate>
<guid>https://jadenecke.github.io/project/citymaps/</guid>
<description><p>A script I wrote to generate artistic city maps from OpenStreeMap. The script downloads the data to disk and reloads it to memory, to finally create the plot. The script automatically adjusts (roughly, according to the 1° lat = 111,111m
<a href="https://gis.stackexchange.com/questions/2951/algorithm-for-offsetting-a-latitude-longitude-by-some-amount-of-meters" target="_blank" rel="noopener">rule</a>) the coordinates to your aspect ratio in order to avoid distortions of the map. I did not manage to remove the white space on the top and right side of the plot. In addition, the code developed from a specific city to general, so there are many inherited inconveniences, especially in terms of size scaling.</p>
<p>
<a href="https://gist.github.com/jadenecke/5252609ef06f1655f42d8cdcd8b0b51e" target="_blank" rel="noopener">Download the script here</a></p>
</description>
</item>
<item>
<title>City Maps</title>
<link>https://jadenecke.github.io/project/commute/</link>
<pubDate>Fri, 07 Jan 2022 00:00:00 +0000</pubDate>
<guid>https://jadenecke.github.io/project/commute/</guid>
<description><p>As preparation to approach the housing market of Munich, I wanted to create a map showing the duration it would take to commute via public transport for both my partner and I. For mapping, my
<a href="https://jadenecke.github.io/project/citymaps/" title="city maps code">city maps code</a> was used, overlaid by a heatmap. The data was queried for points on a grid and linearly interpolated to reduce the number of API queries. For this specific version, the map displays the maximum time to either of two stations and is capped of at 60 minutes.</p>
<p>APIs used, please be kind:</p>
<ul>
<li>MVG-API (undocumented, Munich public transport data,
<a href="https://pypi.org/project/mvg-api/" target="_blank" rel="noopener">some hints can be found here</a>)</li>
<li>
<a href="https://wiki.openstreetmap.org/wiki/API" target="_blank" rel="noopener">OSM-API</a></li>
</ul>
<p>The Code can be found here:
<a href="https://github.com/jadenecke/MunichMetroHeatMap" target="_blank" rel="noopener">Github Repo</a></p>
</description>
</item>
<item>
<title>Setup of the MINC toolbox (Medical Image NetCDF) with WSL 2 (Windows Subsystem for Linux) under Windows 10</title>
<link>https://jadenecke.github.io/2020/11/24/setup-of-the-minc-toolbox-medical-image-netcdf-with-wsl-2-windows-subsystem-for-linux-under-windows-10/</link>
<pubDate>Tue, 24 Nov 2020 00:00:00 +0000</pubDate>
<guid>https://jadenecke.github.io/2020/11/24/setup-of-the-minc-toolbox-medical-image-netcdf-with-wsl-2-windows-subsystem-for-linux-under-windows-10/</guid>
<description>
<script src="https://jadenecke.github.io/rmarkdown-libs/header-attrs/header-attrs.js"></script>
<p>This post is a summary of my experiences and difficulties with setting up MINC under WSL. MINC is a set of open-source tools for processing MRI files. It is developed at the McConnell Brain Imaging Centre, Montreal Neurological Institute (<a href="http://bic-mni.github.io/" class="uri">http://bic-mni.github.io/</a>). In order to run it on a Windows machine, one needs to set up a virtual machine (VM), in my case an Ubuntu distribution. In the following, I will provide the steps and resources required to have a convenient setup using the Windows Subsystem for Linux (WSL) 2.</p>
<div id="installing-wsl-2" class="section level2">
<h2>Installing WSL 2:</h2>
<p>Microsoft provides a detailed step-by-step tutorial for this:
- German: <a href="https://docs.microsoft.com/de-de/windows/wsl/install-win10" class="uri">https://docs.microsoft.com/de-de/windows/wsl/install-win10</a>
- English: <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10" class="uri">https://docs.microsoft.com/en-us/windows/wsl/install-win10</a></p>
<p>Please make sure to follow closely and also check the “Troubleshooting installation” section. I experienced two errors, one related to a visualization setting in the BIOS, the other regarding folder compression. Both (and some more) are explained in the troubleshooting section of the Microsoft article.</p>
</div>
<div id="installing-mobaxterm" class="section level2">
<h2>Installing MobaXterm:</h2>
<p>You need to set up an X11 server to on your windows machine to access the graphical interface of MINC. I recommend MobaXterm, but there are other tools available:</p>
<ul>
<li><a href="https://mobaxterm.mobatek.net/" class="uri">https://mobaxterm.mobatek.net/</a></li>
</ul>
<p>Download, install, and launch the software. The X11 server should be already running when you launch the software, indicated by the colored X symbol in the upper right hand corner. The software should also automatically detect your Ubuntu distribution. You can open a terminal to your Ubuntu VM and it will start the VM, even when it is stopped. You will need the terminal to carry out the next step.</p>
</div>
<div id="installing-minc" class="section level2">
<h2>Installing MINC:</h2>
<p>The full details are documented here, depending on the choice of your operating system:</p>
<ul>
<li><a href="http://bic-mni.github.io/" class="uri">http://bic-mni.github.io/</a></li>
</ul>
<p>You can download the files with windows, as the VM has read-access to your windows files (under <code>/mnt/</code>). You can also download the files with <code>curl</code> in your VM.</p>
</div>
<div id="set-launch-options" class="section level2">
<h2>Set Launch options:</h2>
<p>There are two commands that I would like to execute on every launch with this VM:</p>
<p>First, you need to set the X11 display address on your VM:</p>
<ul>
<li><code>export DISPLAY="$(/sbin/ip route | awk '/default/ { print $3 }'):0"</code></li>
</ul>
<p>Second, you need to source the MINC config. I think this is because it uses some common names and you don’t want to have them sourced all the time. Since this is a one purpose VM (MINC), we can source the toolbox at startup:</p>
<ul>
<li><code>source /opt/minc/1.9.18/minc-toolkit-config.sh</code> for bash</li>
<li><code>source /opt/minc/1.9.18/minc-toolkit-config.csh</code> for tcsh</li>
</ul>
<p>To add these commands to your launch options of your VM, right-click the WSL-Ubuntu (differs for other OSs) session in MobaXterm and select “Edit Session”. Next, select the “Advanced WSL settings” tab and add the commands to the startup routine.</p>
</div>
<div id="wrap-up" class="section level2">
<h2>Wrap up:</h2>
<p>Close your open terminals and start a fresh one for your VM. Check whether e.g. the <code>Display</code> command works for you and the X11 transmission works properly. You might want to set the X11 remote access setting from “on-demand” to “full” in your MobaXterm settings if you want to avoid confirming the X11 session all the time.</p>
</div>
</description>
</item>
<item>
<title>Training a Bidirectional Encoder Representations from Transformers (BERT) on a Cluster with R: Links and tips</title>
<link>https://jadenecke.github.io/2020/03/09/training-a-bidirectional-encoder-representations-from-transformers-bert-on-a-cluster-with-r-links-and-tips/</link>
<pubDate>Mon, 09 Mar 2020 00:00:00 +0000</pubDate>
<guid>https://jadenecke.github.io/2020/03/09/training-a-bidirectional-encoder-representations-from-transformers-bert-on-a-cluster-with-r-links-and-tips/</guid>
<description>
<p>This will merely be a short script of what my experience was and which errors I encountered. Or see it as notes in the case that I ever have to do this again.</p>
<p>I won’t give any links providing theoretical background, because I think everybody has different needs and any search engine should do a better job with this than me. There are tons of sources out there with different highlights, so just look around.</p>
<div id="code-tutorial" class="section level2">
<h2>Code Tutorial:</h2>
<ul>
<li><a href="https://blogs.rstudio.com/tensorflow/posts/2019-09-30-bert-r/" class="uri">https://blogs.rstudio.com/tensorflow/posts/2019-09-30-bert-r/</a></li>
</ul>
<p>This is the only tutorial I could find for R, but there are more python versions available.</p>
</div>
<div id="software-prerequisites" class="section level2">
<h2>Software prerequisites:</h2>
<p>The cluster I work on provides modules that can be easily loaded. In case of R it comes with no packages at all and in case of python only with some base modules. Therefore the next two links will explain how to install those:</p>
<p>For R:</p>
<ul>
<li><a href="https://thecoatlessprofessor.com/programming/r/working-with-r-on-a-cluster/" class="uri">https://thecoatlessprofessor.com/programming/r/working-with-r-on-a-cluster/</a></li>
</ul>
<p>They key part starts at “Setup a local R library for installing and loading R Packages” but the section “Passing arguments” is also very interesting, if you want to launch you script with some parameters, e.g. if you want to specify different learning rates, but don’t want to copy &amp; paste multiple scripts to run them parallel.</p>
<p>For python:</p>
<ul>
<li><a href="https://www.pik-potsdam.de/members/linstead/guides/python-on-the-cluster/installing-your-own-python-modules-on-the-cluster" class="uri">https://www.pik-potsdam.de/members/linstead/guides/python-on-the-cluster/installing-your-own-python-modules-on-the-cluster</a></li>
</ul>
<p>I used this tutorial with “Option 2. virtualenv and pip”. The instructions are deprecated, because I think they favor a conda virtualenv over a python virtualenv now. I have no knowledge to judge the appropriateness of either, but the python virtualenv worked great for me and was easy to install and understand.</p>
</div>
<div id="cuda-and-tensorflow-and-cudnn" class="section level2">
<h2>CUDA and Tensorflow (and cuDNN):</h2>
<p><em>This is only relevant if you want to train on a GPU. If you only train on a CPU just install the newest version of tensorflow</em></p>
<p>This gave the most troubles. You want to be very accurate about the versions of all three and have them matched, otherwise you enter a whole new world of frustration. Here is a list of which versions are compatible:</p>
<ul>
<li><a href="https://www.tensorflow.org/install/source#tested_build_configurations" class="uri">https://www.tensorflow.org/install/source#tested_build_configurations</a></li>
</ul>
<p>CUDA came as a module on my cluster, and I was able to choose different versions. I chose the 10.1 version. Installing tensorflow was trouble-less using the python virtualenv and pip. CuDNN took me a while to figure out, because it wasn’t installed with the CUDA version on the cluster. I assume this is due to some license issues, the reason being you have to apply for the NVIDIA developer program to download them. The easiest and non-invasive way to include the cuDNN files I found in an Stack Overflow thread after banging my head against a wall for a whole day:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/41494585/setting-ld-library-path-from-inside-r" class="uri">https://stackoverflow.com/questions/41494585/setting-ld-library-path-from-inside-r</a></li>
</ul>
<p>You will know that you have to perform this step if you see an error message like this:</p>
<pre class="r"><code>ImportError: libcudnn.so.7: cannot open shared object file: No such file or directory</code></pre>
<p>Basically you want to include every file with <em>dyn.load()</em> that tensorflow is complaining about. For me it was only this <em>libcudnn.so.7</em> file.</p>
</div>
<div id="keras" class="section level2">
<h2>Keras:</h2>
<p>Keras for R itself worked without much troubles, the only bug I also encountered was the mysterious and strangely appearing and disappearing</p>
<pre class="r"><code> Error in py_get_attr_impl(x, name, silent) :
AttributeError: module &#39;kerastools&#39; has no attribute &#39;progbar&#39;</code></pre>
<p>bug. This apparently is a real bug and fixed in the new version of keras:</p>
<ul>
<li><a href="https://github.com/rstudio/keras/issues/992" class="uri">https://github.com/rstudio/keras/issues/992</a></li>
</ul>
<p>For the time being you can either install the development version of keras, which I was hesitant about, not wanting to introduce more dependency issues, or just run</p>
<pre class="r"><code>try(k_constant(1), silent=TRUE)
try(k_constant(1), silent=TRUE)</code></pre>
<p>This will trigger the error on the first try and apparently load whatever is needed so it works on the second try. Again this is a bug that will be fixed in the upcoming version (current version: 2.2.5.0)</p>
</div>
<div id="saving-loading-and-predicting-the-bert-model" class="section level2">
<h2>Saving, loading, and predicting the BERT model:</h2>
<p>There are different means of saving and loading a model with keras:</p>
<ul>
<li>save_model_tf(): This function saves the model as multiple files in a folder and you have to specify the only the folder, if you want to load the model. I had no luck with this. There always was an error message when I tried to load the model but i also didn’t try to hard, as I found another way that worked for me.</li>
<li>save_model_weights_tf/hdf5(): As far as I understand, this only saves the matrix with the weights, so you need to specify and compile the whole model as it was trained, when you want to load weights into the model.</li>
<li>save_model_hdf5(): This was the function I ended up using, although it also wasn’t very straight forward. It saves the model in a single file (or multiple, depending on the size). The slightly finicky details are as follows:</li>
</ul>
<p>You want to load the model with the following command:</p>
<pre class="r"><code>bert &lt;- keras::load_model_hdf5(&quot;path/to/file.hdf5&quot;),
custom_objects = k_bert$get_custom_objects())</code></pre>
<p>Therefore, you need to load the model (which you have to do anyway to tokenize your to be predicted data). All the code to load the model you can find in the BERT for R tutorial. As BERT uses custom layers not included with keras, you have to pass them as custom objects to the load function. Otherwise this error will appear:</p>
<pre class="r"><code>Error in py_call_impl(callable, dots$args, dots$keywords) :
ValueError: Unknown layer: TokenEmbedding </code></pre>
<p>Then, at least for me, the model also failed to auto-compile on load, so I had to compile it by hand. Fortunately this worked by the same means as in the training tutorial. You have to specify the target length, learning rate, batch size, and epochs, but that it plain worked for me.</p>
<pre class="r"><code>c(decay_steps, warmup_steps) %&lt;-% k_bert$calc_train_steps(
targetsPred %&gt;% length(),
batch_size=bch_size,
epochs=epochs
)
bert %&gt;% compile(
k_bert$AdamWarmup(decay_steps=decay_steps,
warmup_steps=warmup_steps, lr=learning_rate),
loss = &#39;binary_crossentropy&#39;,
metrics = &#39;accuracy&#39;
)</code></pre>
<p>The data you want to predict needs to be tokenized as well. To achieve this, just repeat the same steps as with the training data, until the <em>concat</em> list is made. For the <em>k_bert$calc_train_steps()</em> function the length of the data must be specified. I just generated a <em>LABEL_COLUMN</em> in my dataframe with NAs and kept the rest of the code the same. If you do it this way, you also don’t have to modify the <em>tokenize_fun()</em> function. For my convenience I modified the function anyway, to accept two strings for the data and the label column:</p>
<pre class="r"><code>tokenize_fun = function(dataset, DATA_COLUMN, LABEL_COLUMN) {
c(indices, target, segments) %&lt;-% list(list(),list(),list())
pb = txtProgressBar(min = 0, max = nrow(dataset), initial = 0,style = 3)
for (i in 1:nrow(dataset)) {
setTxtProgressBar(pb,i)
c(indices_tok, segments_tok) %&lt;-% tokenizer$encode(dataset[[DATA_COLUMN]][i],
max_len=seq_length)
indices = indices %&gt;% append(list(as.matrix(indices_tok)))
target = target %&gt;% append(dataset[[LABEL_COLUMN]][i])
segments = segments %&gt;% append(list(as.matrix(segments_tok)))
}
return(list(indices, segments, target))
}</code></pre>
<p>Generating predictions from the model is quite simple, just use the <em>predict()</em> function with the model, and the tokenized data:</p>
<pre class="r"><code>predict(bert, concat, verbose = 1)</code></pre>
<p>The regular <em>predict()</em> function has no verbosity, so there is no default. I highly recommend setting it as prediction might take a while. You might even want to do the prediction on a cluster as well.</p>
</div>
<div id="some-advice-for-using-a-cluster" class="section level2">
<h2>Some advice for using a cluster:</h2>
<p>This is probably some pretty natural stuff for anyone who already worked with a cluster, but as this was my first experience with one, I want to state a single thing that I wish I had known earlier:</p>
<p>I am not sure, whether every cluster offers this functionality, but at my place I was able to queue an interactive session instead of a job. After some waiting, this provides you with a shell interface to a node. Troubleshooting is so much easier if you don’t have to wait an hour for your job to be scheduled and then see it fail within seconds because you made a typo. Just run your script with <em>Rscript</em> and don’t forget to include many</p>
<pre class="r"><code>cat(paste(Sys.time(), &quot;Loading some data or whatever...\n&quot;))</code></pre>
<p>to see where your stuff fails. And it will fail.</p>
</div>
</description>
</item>
</channel>
</rss>