Using Jupyter Notebook

We discourage the use of Jupyter Notebook on the supercomputers. Supercomputers are setup to process jobs in a batch fashion, while Jupyter Notebook is designed to be interactive. In short, it wastes resources that could be used by other users.

We suggest you use Jupyter Notebook on your own system during the development stage, then, once you’re ready to run it on large datasets, convert it to straight Python and run it on the supercomputers in the traditional manner. We are happy to help you get this setup.

However, we realize some of you will want to use Jupyter Notebooks despite our warnings, so below are instructions on how to do so.

Catalpa is the only system that we support running Jupyter Notebook on.

First Time

First, send an email to assist@mcsr.olemiss.edu requesting permission to use Jupyter Notebooks. This can take up to one business day.

Each Time

To start Jupyter Notebook, login to Catalpa via hpcwoods. Then, start an interactive PBS session similar to below:

qsub -I -l walltime=8:00:00 -l ncpus=1 -l mem=32gb

Modify the command above if you need more resources, but don’t request more than 1 CPU unless you know what you’re doing.

The walltime parameter means that your PBS session will be automatically killed after 8 hours. This is to ensure that you don’t accidentally leave your Jupyter Notebook session running when it’s not in use. If you need it to run longer than 8 hours, modify this parameter. However, realize that if you walk off and leave Jupyter Notebook running, it is keeping other users from using the system.

Next, load the python module and start jupyter-notebook:
module load python
jupyter-notebook --no-browser --ip=130.74.110.5

If you need to load a specific Anaconda environment, do it after loading the Python module, but before starting jupyter-notebook.

This will produce several lines of output, including one that looks like this:
http://130.74.110.5:8888/?token=2e3b8a29a23e1e62a380aace9bfba58ad3bc638616e83424

If Catalpa were directly on the Internet, you would be able to copy and paste this URL into your browser and start computing. But it’s not, so you need to do a couple of extra steps.

SSH Tunnel

Note the port number from the URL above. Generally, it will be 8888, but it could vary. This is the port number that Jupyter Notebook is listening on.

From your local system, run the below, changing both instances of the port number if needed, and substituting your username:
ssh -N -L 8888:catalpa:8888 username@hpcwoods.olemiss.edu

Mac and Linux users have SSH installed by default. Windows users will need to setup SSH forwarding via PuTTY.

Connect via browser

Now paste the URL from before into your local browser, but change the IP address (130.74.110.5) to localhost:
http://localhost:8888/?token=2e3b8a29a23e1e62a380aace9bfba58ad3bc638616e83424

Your local browser should now be able to connect to the Jupyter Notebook server running on Catalpa, via the SSH tunnel that you have setup.

Shutdown

When you’re done with your Jupyter Notebook session, start the shutdown process by closing your browser tab or window.

Then, in your Catalpa terminal window, press Control-C twice in quick succession to shutdown the Jupyter Notebook server.

Then, type “exit” to exit the interactive PBS session to release the CPUs and memory to be used by other users.

Finally, in your local terminal window, press Control-C to stop the SSH tunnel.