Python packages

• Common steps
• Using pip
• Virtual environment (recommended method)
• Conda

Python is widely used on the cluster, and several different versions/subversions are provided as modules, as part of the central software stack. Users just need to load the module for the specific version they need before using it. Several of the most used python packages (e.g. numpy, scipy, …) are also installed as modules as part of the central software stack, and available as modules as well.
However, users often come with specific needs for a particular package that is not of general interest/relevance, or more commonly for a specific package version that works for a specific python subversion. This kind of special case is not easily managed in a centralized software stack, and therefore users are encouraged to install the package locally.
This procedure is almost the same that can be used on a local Linux computer, but there are a few important differences. For this reason, we provide here some guidelines for the most common cases.

Please ensure you use a recent Python version (>=3.9). Python 2 is deprecated and not maintained anymore.

General common steps

The main differences with respect to installing packages on a local Linux machine, are:

• Different Python versions are available through modules
• Users do not have administrator (root) privileges;

First step is to choose the python version that you need. Look into the available modules, and load the module that you prefer.

NOTE: specific packages only works with some specific python versions.

NOTE: you cannot mix packages for different python versions.

The lack of administrator privileges is not a problem.

Using pip

NOTE: please use virtual environments as a standard!

pip can be used to install packages in the user’s HOME directory. The user only need to specify the flag --user

To give an example,  to install the package scikit-learn under python 3.10.13

module load python3/3.10.13
python3 -m pip install --user scikit-learn

Then start python with the command

python3

Please notice:

• You need to first load the python module
• for python 3.x.y  the commands are python3 and pip3
• python is pointing to the python-2.X which comes with the Scientific Linux 7.X – you usually don’t want this
• Please do not use pip3 install ... you should get used to using python3 -m pip ...

The packages installed in this way are installed in your HOME directory under, for python 3.x

~/.local/lib/python3.x/site-packages/

Virtual environments

Using virtual environments lets you have different stacks of software packages in different folders. Please search the web for additional information regarding virtual environments.

First, select the Python version you want, and load the corresponding module. Here we choose python 3.10.13. If you use versions older than 3.6, the instructions to create the virtual environment may differ.

module load python3/3.10.13

Then create a virtual environment. We choose the name venv_1

python3 -m venv venv_1

That’s all. Now, activate the environment with the command

source venv_1/bin/activate

When the environment is active the shell notes this to the left of the cursor. It will look something like this:

(venv_1) ~
gbarlogin1$

This indicates the virtual environment name in parenthesis.

Note that both the commands python and python3 point to the python version of your original module. Now, when the environment is active, you can install all the packages you need with the normal python3 -m pip install ... syntax, for example (note --user should NOT be added)

python -m pip install scikit-learn

All those packages will be installed locally under the directory

venv_1/lib/python3.10/site-packages/

When you have finished to use the environment, just deactivate it with the command

deactivate

NOTE: when you want to use the environment, activating the environment is not enough. You have to load also the python module you had used to create the environment.
For convenience, you can add, at the top of the venv_1/bin/activate file (adapt to the name of your environment), the line module load python3/3.10.13 (adapt to the version you used). This way, sourcing the environment will automatically load the dependencies of the python interpreter, all in one step

NOTE: virtual environments are made to isolate an environment from another, so they should be used to install everything you need inside it. It is possible to combine packages installed as modules as system level, and packages installed in the environment. However, this needs some fixes in the PYTHONPATH variable.

NOTE: NEVER USE the flag --user inside a virtual environment. It will install the packages locally, but outside the virtual environment.