PyTorch: RuntimeError: Library libcublas.so.11 is not found or cannot be loaded

Verified FixPython 3.10+PyTorch 2.2+, CUDA 11/12Silo: pytorch

Quick Fix / Solution Rapide

This error occurs when Python tries to run GPU matrix operations in PyTorch but cannot locate the NVIDIA cuBLAS shared library libcublas.so.11. Install nvidia-cublas-cu11 via pip or add site-packages/nvidia/cublas/lib to LD_LIBRARY_PATH.

Root Cause Analysis

This error occurs on Linux systems when PyTorch attempts to execute CUDA matrix multiplications (such as torch.matmul, linear layers, or convolutions) on an NVIDIA GPU, but the dynamic linker fails to load the NVIDIA Basic Linear Algebra Subprograms library (libcublas.so.11 or libcublasLt.so.11).

Root Cause 1: Missing nvidia-cublas-cu11 Package in pip Wheels

In modern PyTorch pip wheels, NVIDIA CUDA runtime libraries are distributed as separate modular pip packages (e.g. nvidia-cublas-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11). If these packages were not installed or were accidentally uninstalled, PyTorch fails at runtime when initializing the GPU backend.

Root Cause 2: Missing LD_LIBRARY_PATH Configuration on Linux

The Linux dynamic linker (ld.so) searches system library paths (/usr/lib, /usr/local/cuda/lib64). When CUDA libraries reside inside a Python virtualenv (.venv/lib/python3.x/site-packages/nvidia/cublas/lib), the dynamic linker will not find them unless LD_LIBRARY_PATH includes the virtualenv's nvidia directory.

Root Cause 3: CUDA Toolkit Version Mismatch (CUDA 11 vs CUDA 12)

If you install a PyTorch build compiled for CUDA 11 (cu118) on a system with CUDA 12 libraries (libcublas.so.12), the binary specifically requests libcublas.so.11 and aborts.

Root Cause 4: Container Environment Missing NVIDIA Container Toolkit

Running Docker containers without passing the --gpus all flag or without the NVIDIA Container Toolkit prevents the container from accessing host GPU driver libraries.

Reproduction Code (MCVE)

Example: Bug Reproduction

# Simulating dynamic library loader failure for libcublas.so.11
class MockCUDALibraryLoader:
    def load_library(self, lib_name: str):
        if lib_name == "libcublas.so.11":
            raise RuntimeError(
                "RuntimeError: Library libcublas.so.11 is not found or cannot be loaded. "
                "Ensure nvidia-cublas-cu11 is installed and LD_LIBRARY_PATH includes the library path."
            )

loader = MockCUDALibraryLoader()
loader.load_library("libcublas.so.11")

Solution 1: Install NVIDIA CUDA Runtime Packages via pip

Install the required nvidia-cublas-cu11 package or reinstall PyTorch with the complete CUDA runtime bundle.

Example: Recommended Solution

# Solution 1: Terminal commands to install complete CUDA runtime bundle
commands = [
    "pip install nvidia-cublas-cu11 nvidia-cudnn-cu11",
    "pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118"
]

print("Installation commands:")
for c in commands:
    print(f"  $ {c}")

assert len(commands) == 2

Solution 2: Export LD_LIBRARY_PATH to Include Virtualenv NVIDIA Libraries

Add the virtual environment's nvidia library directories to LD_LIBRARY_PATH in your shell configuration.

Example: Alternative Solution

import os

# Solution 2: LD_LIBRARY_PATH configuration pattern
virtualenv_nvidia_path = "/home/user/project/.venv/lib/python3.10/site-packages/nvidia/cublas/lib"
current_ld = os.environ.get("LD_LIBRARY_PATH", "")

updated_ld = f"{virtualenv_nvidia_path}:{current_ld}".strip(":")
print("Configured LD_LIBRARY_PATH export:")
print(f'export LD_LIBRARY_PATH="{updated_ld}"')

assert virtualenv_nvidia_path in updated_ld

In Docker containers, never run pip install torch without verifying the base image. Always use an official NVIDIA CUDA base image (such as nvidia/cuda:11.8.0-runtime-ubuntu22.04) or install the matching PyTorch wheel.

Note de reproductibilité : Cette erreur dépend des bibliothèques C++ système Linux, de la présence des runtimes NVIDIA CUDA et de la configuration des variables d'environnement LD_LIBRARY_PATH.

Contrast libcublas.so with libcudnn.so: libcublas handles matrix multiplication and linear algebra; libcudnn handles neural network operations like convolutions and activations.

Previous Next