Embracing the Messiness in Search of Epic Solutions

Enabling Python VirtualEnv in JupyterLab

Posted

in

,

This post illustrates how you can enable Python virtualenv in GCP JupyterLab so that you can organize your .ipynb files to use different virtual environments to keep track of Python package dependencies.

PROBLEM

  • You are using GCP JupyterLab.
  • You want to adhere to the Python development best practices by not polluting the global environment with your Python packages so that you can generate a cleaner “pip freeze” in the future.
  • You want each Notebook file (.ipynb) to have its own environment so that you can run them with different package versions.
  • You configured a Python virtual environment, but “pip install” from the Notebook file still installs the packages in the global environment.
  • You are fed up.

SOLUTION

Configuring Virtual Environments

In the JupyterLab Notebook’s terminal, create an empty directory to organize all virtual environments.

mkdir virtualenv

Ensure ipykernel is installed. This is used to create new IPython kernels.

python3 -m pip install ipykernel

For each new virtual environment, run the following commands to perform these steps:

  • Get into the base virtual environment directory.
  • Define a new virtual environment name. Replace [NEW_ENV_NAME] with a new name.
  • Create new Python virtual environment.
    • The –system-site-packages option ensures you can still use the “data-sciencey” packages that come pre-installed with the GCP JupyterLab Notebook within your new virtual environment.
  • Jump into the newly created virtual environment.
  • Create a new IPython kernel.
  • Exit from virtual environment.
cd virtualenv
VENV=[NEW_ENV_NAME] # Update this!
python3 -m venv $VENV --system-site-packages
source $VENV/bin/activate
python -m ipykernel install --user --name=$VENV
deactivate

Configuring Notebook File

Create a new Notebook file (.ipynb).

In Select Kernel dialog, select the kernel that you created. In this example, there are 2 new virtual environments (“smurfs” and “thundercats”).

Selecting a kernel in JupyterLab

To perform a simple test, install a new package.

%pip install pandas==1.3.0

IMPORTANT: You need to use IPython’s Magics (literally speaking) to ensure the packages are installed in the virtual environment.

  • %pip = This uses the pip package manager within the current kernel. Magic!
  • ! pip = This uses the pip package manager from the underlying OS. No magic!

From the menu bar, select Kernel -> Restart Kernel and Clear All Outputs… . Always restart the kernel when new packages are installed with %pip.

Kernel -> Restart Kernel and Clear All Outputs in JupyterLab

Inspect the package version. This should show the version you have just installed.

import pandas
print(pandas.__version__)

To verify this actually works, create another Notebook file pointing to another kernel. In this file, install the same package but with different version.

Testing Python Virtualenv in JupyterLab

Comments

Leave a Reply