JupyterHub

A JupyterHub server is available at https://mc2.ua.pt/jupyter. JupyterHub brings the power of notebooks to groups of users. It gives users access to computational environments and resources without the burden of installation and maintenance tasks.

Kernel Management

Viewing installed kernels

Installed kernels can be listed on the Jupyter Launching page, or using the command-line in a terminal window. On the Launcing page we have a layout like the one depicted below.

Figure: Launch page of JupyterHub listing the available kernels.

On the command line simply use the jupyter kernelspec command:

[user@hn]$ jupyter kernelspec list
Available kernels:
  python3    /opt/jupyterhub/share/jupyter/kernels/python3

The above python3 kernel (displayed as Python 3 (ipykernel)), is a default that available system-wide should you have no additional kernels installed.

Add a kernel using a Conda environment

For the sake of example we are creating a Conda environment test enclosing a Jupyter kernel with the same name (test), which will be displayed on the Lauching page as Kernel Test.

1. Create a new Conda environment containing the ipykernel package (mandatory) plus any other packages you want:

[user@hn]$ conda create --name test ipykernel <other-packages...>

Press [y] to accept the installation of the packages, and wait a bit.

2. Activate and register the Conda environment as a Jupyter kernel named test and presenting itself as Kernel Test on the Jupyter web service:

[user@hn]$ conda activate test
(test) [user@hn]$ python -m ipykernel install --user --name test --display-name 'Kernel Test'

The --user option means that the kernel will be visible for the current user only.

3. Verify that the Kernel has been installed

[user@hn]$ jupyter kernelspec list
Available kernels:
  python3    /home/user/.conda/envs/test/share/jupyter/kernels/python3
  test       /home/user/.local/share/jupyter/kernels/test

The kernelspec test is written in JSON to kernel.json in the installation directory:

# /home/user/.local/share/jupyter/kernels/test/kernel.json
{
 "argv": [
  "/home/user/.conda/envs/test/bin/python",
  "-Xfrozen_modules=off",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Kernel Test",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

Add a kernel using a Python virtual environment

1. Create and activate a virtual environment:

[user@hn]$ python3 -m venv ~/.virtualenvs/test
[user@hn]$ source ~/.virtualenvs/test/bin/activate

2. Verify that you are working with the correct Python

(test) $[user@hn] which python
~/.virtualenvs/test/bin/python

3. Install ipykernel in the virtual environment:

(test) $[user@hn] python -m pip install ipykernel

4. Register the virtual environment as a Jupyter kernel (same as for Conda):

(test) [user@hn]$ python -m ipykernel install --user --name test --display-name 'Kernel Test'

5. Verify that the Kernel has been installed

[user@hn]$ jupyter kernelspec list
Available kernels:
  python3    /home/user/.conda/envs/test/share/jupyter/kernels/python3
  test       /home/user/.local/share/jupyter/kernels/test

Setting environment variables on a kernel

You can define environment variables visible to kernels by passing the --env option to the ipykernel install command upon kernel creation:

[user@hn]$ conda activate test
(test) [user@hn]$ python -m ipykernel install --user --name test --display-name 'Kernel Test' --env COMPUTE /mnt/beegfs/compute/user --env LD_LIBRARY_PATH $HOME/lib:\${LD_LIBRARY_PATH}

The value of $HOME is substituted on the command line since it was not escaped, but since ${LD_LIBRARY_PATH} was escaped with \ and that text is preserved in the kernelspec configuration file for substitution at kernel launch time:

# /home/user/.local/share/jupyter/kernels/test/kernel.json
{
 "argv": [
  "/home/user/.conda/envs/test/bin/python",
  "-Xfrozen_modules=off",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Kernel Test",
 "language": "python",
 "metadata": {
  "debugger": true
  },
  "env": {
    "COMPUTE": "/mnt/beegfs/compute/user",
    "LD_LIBRARY_PATH": "/home/user/lib:${LD_LIBRARY_PATH}"
 }
}

Remove a kernel

To uninstall/remove a Kernel named test from Jupyter, make sure that the Conda environment is deactivated. Then execute the kernelspec remove command:

(test)[user@hn]$ conda deactivate test
[user@hn]$ jupyter kernelspec remove test
Kernel specs to remove:
  test                  /home/user/.local/share/jupyter/kernels/test
Remove 1 kernel specs [y/N]: <Press 'y'>
Removed /home/user/.local/share/jupyter/kernels/test

You still need to remove the conda environment if you wish to wipe out any trace of the Kernel:

[user@hn]$ conda remove --name test --all

If you created the kernel on a Python virtual environment, just deactivate the environment, remove the kernel, and wipe out the venv directory tree by hand:

(test) [user@hn]$ deactivate
[user@hn]$ jupyter kernelspec remove test
[user@hn]$ rm -rf ~/.virtualenvs/test