Jupyter notebook

Instructions for launching Jupyter notebook (NB) from cluster. NB should be launched from compute nodes. These instructions is for Mac users.

Some basic concepts

• Your Mac can only directly connect to the login nodes, but not compute node.
• If you want to launch NB from computes nodes, you have to setup connections between your Mac and the login node (forward tunnel), as well as between the login node and the compute node (reverse tunnel).
• Please setup your own port number, as well as SSL credentials (see below).

Setups

• Install Jupyter
• Start by creating a jupyter config file (~/.jupyter/jupyter_notebook_config.py) if one doesn't already exist using:

jupyter notebook --generate-config

• Create SSL credentials using:

openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout ~/.jupyter/mycert.key -out ~/.jupyter/mycert.pem

• Next, open Python and use the following to generate a password:

In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:14488ced102d:c675686a6f3ff455a355cfc2a6faa75683744a89'

• Open the jupyter config file (~/.jupyter/jupyter_notebook_config.py), and make the following changes:

1. c.NotebookApp.ip = '0.0.0.0'.
2. c.NotebookApp.port = 7700 (replacing 7700 with a new, unused port number).
3. c.NotebookApp.open_browser = False.
4. c.NotebookApp.password = u'sha1:14488ced102d:c675686a6f3ff455a355cfc2a6faa75683744a89' (replacing with your hash from above).
5. c.NotebookApp.certfile = u'/users/<username>/.jupyter/mycert.pem' (replacing username with your username).
6. c.NotebookApp.keyfile = u'/users/<username>/.jupyter/mycert.key' (replacing username with your username).

Launch NB from compute node (example)

1. From sug-login1, submit an interactive job qsub -I -q <your queue> (Note: if you want to keep the interactive job alive, you have to use tmux to create a session on the login node before submitting job. You can't use screen, because you'll need screen inside the compute node later)
2. Within app / compute node, create a screen to launch NB jupyter-notebook (or jupyter lab), this gives you the NB url
3. Create another screen Ctrl+a c to setup reverse tunnel ssh -NR <port#>:localhost:<port#> sug-login1 (has to be the same login node as in step 5 below. Also use your own port number)
4. Detach Ctrl+a d
5. From your Mac, setup forward tunnel ssh -NL <port#>:localhost:<port#> <username>@sug-login1.hgsc.bcm.edu (has to be the same login node as in step 3 above)
6. Now you can access your NB at https://localhost:<port#> (use your own port number, of course). Note that you need to use https and a password to access your notebooks over the web.
7. Run notebook forever for free (!!! Warning !!! you do not see this trick from here). After you setup everything and make sure it's up and running, you can just simply kill your job, that's it! The only thing you need to do is after you lose internet connection, you have to redo the forward tunnel from your local computer, but your work still exist and running on the compute node. (!!! Important !!! Please remember which node you launched your notebook, because in case you accidentally kill your notebook from a terminal on the notebook, you have to log back to that specific compute node in order to deactivate / recycle your port id, and it's not fun to try to figure out where you've launched your notebook if you lose track)