Python¶

When using Python, you may find you need to use various libraries (e.g., numpy for numerical analysis or matplotlib for plotting). Installing and managing these different libraries and their dependencies can be problematic, especially when you run into conflicts. Conda and Mamba are package managers that help you create and navigate "environments" to help automatically handle these conflicts. These environments can help you keep the python package versions needed for your different projects separate, which helps resolve dependency conflicts. To learn more, we recommend this introduction to conda or this conda tutorial. Mamba is, for the most part, a drop-in replacement for Conda, and is often much faster and better able to solve complex dependencies: Mamba documentation. In the following tutorial, we will demonstrate how to install and use Mamba, but if you already have a conda installation via miniconda or another distribution, you can likely just replace mamba with conda in the commands.

Installing Conda/Mamba¶

For managing python environments on the cluster, we recommend following the install instruction for Unix-like platforms provided by Miniforge:

wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

The following prompt will ask whether you accept the license, if the default installation location (typically in your home directory) is alright (it is), and whether you want to initialize conda each time you log in (generally a good idea). Once the install is finished, re-sourcing your bashrc will add the conda/mamba commands to your path:

. ~/.bashrc

At this point you should see the environment, (base), appear at the front of your terminal prompt:

(base)[yourusername@borah-login ~]$

This lets you know that your "base" environment is active.

Info

Using mamba install while your base environment is active will install packages into the base environment.

It is good practice to not install any packages in your base environment but instead to create separate environments.

Creating an environment¶

Now that you've installed mamba/conda, let's create an environment.

Warning

Don't create conda environments or install packages on the login node. You can tell which node you're on by looking at your terminal prompt. If your prompt shows you are on the login node (e.g., [username@borah-login]$), make sure to check out an interactive session using the command dev-session before installing.

The general command to create an environment is as follows:

mamba create -n ENVIRONMENTNAME -c CHANNEL PACKAGE1 PACKAGE2

The environment name can be whatever you like, the channel is one from Anaconda.org (common channels are conda-forge or bioconda), and the package names are whatever packages you want in that environment. For example:

The following command creates an environment called "my-env" that pulls from the conda-forge channel with the packages matplotlib and numpy:

mamba create -n my-env -c conda-forge matplotlib numpy

Once this environment is created, it can be activated using the following command:

mamba activate my-env

Conda/Mamba are powerful tools with many different options, to learn more check out the conda user guide.

Creating an environment to work with the GPU¶

Notes about conda builds and virtual packages¶

Many python packages distribute builds which can make use of the GPU through the CUDA api. In order to build an environment which can use the GPU, conda either needs to be able to detect the CUDA version on the system or to be manually instructed what CUDA version to use so that it can download the correct python package build. Conda does this detection through virtual packages. You can see what virtual packages conda has by running conda info.

For example if we run conda info on a GPU node:

conda info

active environment : base

...

virtual packages : __archspec=1=cascadelake
                   __conda=24.11.3=0
                   __cuda=12.4=0
                   __glibc=2.17=0
                   __linux=3.10.0=0
                   __unix=0=0

We can see in the output (highlighted above) that conda detects a virtual CUDA package.

It is also helpful to understand a little about the structure of a conda package. When you pull a package from a conda channel, the naming convention is PACKAGENAME-VERSION-BUILD for example numpy-2.4.2-py313hfc84e54_1 is the package, NumPy, version 2.4.2 built for Python 3.13 (we can ignore the rest of the build string).

A GPU-capable build will often have "gpu" or "cuda" in the build tag. For example, you might see the following output during the environment creation process if you are pulling tensorflow or pytorch built with CUDA:

pytorch                         2.5.1  cuda126_mkl_py313_h33c0e77_310
...
tensorflow                     2.17.0  cuda120py312h02ad488_203

Building a GPU-capable environment¶

First, check out an interactive session to prevent the conda environment creation step from getting killed on the login node:
```
gpu-session
```
If this command is taking a while, it might mean all the available nodes are in use, so you can also try gpu-session-l40 or dev-session.

Info

If you use dev-session, which starts an interactive session on a node without GPU, you'll need to run export CONDA_OVERRIDE_CUDA="12.4" before creating your environment.
Create your new environment specifying a "cuda" or "gpu" build:
```
mamba create -n my-gpu-env "tensorflow=*=cuda*"
```
The above command tells conda to grab the package "tensorflow", any version, and any build that starts with "cuda".
Activate your environment and confirm that your package was installed correctly:
```
mamba activate my-gpu-env
```
To check if PyTorch can use the GPU:
```
python -c "import torch; print(torch.cuda.is_available())"
```
To check if TensorFlow can use the GPU:
```
python -c "import tensorflow as tf; print(tf.test.is_built_with_cuda())"
```
If your pytorch/tensorflow installation is built with cuda, both of those lines should print "True".

And that's it! Your python environment is ready to use the GPU.

Submitting jobs that use python in an environment¶

Following is an example script to submit a python job to the scheduler.

conda-slurm.sh

#!/bin/bash
#SBATCH -J python         # job name
#SBATCH -o log_slurm.o%j  # output and error file name (%j expands to jobID)
#SBATCH -n 1              # total number of tasks requested
#SBATCH -c 48             # CPU cores per task
#SBATCH -N 1              # number of nodes you want to run on
#SBATCH -p bsudfq         # queue (partition)
#SBATCH -t 12:00:00       # run time (hh:mm:ss) - 12.0 hours in this example.

# Activate the environment
# Replace environmentName with your environment name
. ~/.bashrc
conda activate environmentName

# Your code goes here
# Replace mypythonscript.py with the script you want to run
python mypythonscript.py

Using an environment with Open OnDemand¶

Open OnDemand provides a graphical interface to the cluster. The OnDemand interface for Borah can be accessed at ondemand.boisestate.edu.

In order to use your environment in a Jupyter Notebook through OnDemand, you'll need to install some additional packages. With the environment you want to use activated, install ipykernel:

mamba install ipykernel

Then run ipykernel to create the custom Jupyter kernel: (replace ENVIRONMENT_NAME with the environment name and PYTHON ENV NAME with the name you will select for the kernel)

python -m ipykernel install --user --name ENVIRONMENT_NAME --display-name "PYTHON ENV NAME"

Then navigate to the Jupyter Notebook App on ondemand.boisestate.edu:

Navigate to the Jupyter Notebook App

Once your Jupyter session starts, select the kernel you just made (It will be listed under the name you put in PYTHON ENV NAME the example below shows a kernel named "climate"):

Select the right Jupyter kernel

Moving your conda installation to scratch¶

Conda environments can get quite large and can exceed your home directory quota. In this situation you may want to move your conda installation to your scratch. You can relocate your miniforge3 directory to your scratch space using the following steps:

Make a miniforge3 directory in your scratch space:
```
mkdir ~/scratch/miniforge3
```
Copy over your existing data. (This may take several minutes if your miniforge3 directory is large.):
```
rsync -aAvP ~/miniforge3/ ~/scratch/miniforge3
```
Remove your current miniforge3 directory:
```
rm -rf ~/miniforge3
```
Create a link to your new miniforge3 directory:
```
ln -s ~/scratch/miniforge3 ~/miniforge3
```

And that's it—you can continue using conda as before!