Forward compatibility for newer CUDA toolkits

Overview

Some Python packages are built against newer CUDA toolkits than those available in the system CUDA driver. In such cases, the CUDA Forward Compatibility package (cuda-compat) allows applications to use newer CUDA runtime libraries without requiring a system-wide driver upgrade.

On Ibex, this can be installed inside a Conda environment without administrator privileges.

Prerequisites

Before proceeding, ensure that Miniforge (Conda) is installed on Ibex.

For installation instructions, refer to How to Setup Conda on Ibex Guide to get started.

Installation

1. Allocate a GPU node and start an interactive session:

srun -N1 --gres=gpu:a100:1 --ntasks=4 --time=1:0:0 --pty bash

2. Activate your Miniforge installation:

source /ibex/user/$USER/miniforge/bin/activate

3. Create a new Conda environment:

conda create --name envcuda13

4. Activate the environment:

conda activate envcuda13

5. Install the CUDA Forward Compatibility package together with Python and pip:

conda install conda-forge::cuda-compat==13.0.0 python pip

6. Add the CUDA compatibility libraries to LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=$CONDA_PREFIX/cuda-compat/:$LD_LIBRARY_PATH

7. Install PyTorch:

pip install torch torchvision

Verification

Verify that PyTorch detects the GPU and is using the expected CUDA toolkit:

python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"

Expected output:

True 13.0

Notes

• The cuda-compat package provides forward compatibility libraries for newer CUDA toolkits.

• This installation is isolated within the Conda environment and does not modify the system CUDA installation.

• The LD_LIBRARY_PATH environment variable must include the cuda-compat directory before launching CUDA applications.

• For more information on CUDA Forward Compatibility, refer to the official NVIDIA documentation: https://docs.nvidia.com/deploy/cuda-compatibility/latest/forward-compatibility.html