RuntimeError: CUDA error: device-side assert triggered in PyTorch

Verified FixPython 3.10+PyTorch 2.0+ / CUDASilo: pytorch

Quick Fix / Solution Rapide

Set environment variable CUDA_LAUNCH_BLOCKING=1 to pinpoint the exact line, and verify all target labels fall within [0, num_classes - 1].

Root Cause Analysis

This error occurs when a CUDA kernel assertion fails asynchronously on the GPU hardware, typically caused by indexing a tensor, embedding, or loss function with values outside valid memory bounds.

1. Asynchronous Execution and Masked Tracebacks

PyTorch dispatches CUDA kernel calls asynchronously to maximize hardware throughput. When an illegal operation occurs on the GPU (such as accessing embedding index 100 in an embedding table of size 50), the GPU triggers an asynchronous device-side assert. Because CPU execution has already progressed past the offending line, Python reports the error on a subsequent CUDA call, masking the true origin of the bug.

2. Common Causes of Device-Side Asserts

The two most common triggers are: (1) classification labels exceeding the loss function class count (e.g. label 5 passed to a 3-class classifier); and (2) negative or out-of-bounds token IDs passed to nn.Embedding.

3. Isolating the Bug with Synchronous Mode

To locate the exact failing line of code, enable synchronous CUDA execution by running your script with the environment variable CUDA_LAUNCH_BLOCKING=1 or running the model on CPU.

4. Note de Reproductibilité

Cette erreur dépend de l'état du matériel GPU, du pilote CUDA et de la configuration de synchronisation des threads asynchrones. La reproduction exacte sur GPU nécessite un environnement accéléré compatible CUDA.

Reproduction Code (MCVE)

Example: Bug Reproduction

import torch
import torch.nn as nn

# Simulation: Out-of-bounds target index (5 is >= 3 classes)
criterion = nn.CrossEntropyLoss()
logits = torch.randn(2, 3)  # 3 classes: valid indices 0, 1, 2
invalid_targets = torch.tensor([1, 5], dtype=torch.long)  # Index 5 is out of bounds

# Raises IndexError / RuntimeError when evaluated
loss = criterion(logits, invalid_targets)

Solution 1: Validate and Clamp Target Index Bounds

Ensure all target class indices strictly adhere to 0 <= target < num_classes before calling loss functions.

Example: Recommended Solution

import torch
import torch.nn as nn

num_classes = 3
criterion = nn.CrossEntropyLoss()
logits = torch.randn(2, num_classes)
raw_targets = torch.tensor([1, 5], dtype=torch.long)

# Validate and clamp target indices to ensure valid range [0, num_classes - 1]
valid_targets = torch.clamp(raw_targets, 0, num_classes - 1)
loss = criterion(logits, valid_targets)
print('Loss calculated safely:', loss.item())

Solution 2: Add Defensive Shape & Range Assertions

Add explicit assertion guards in your training pipeline to fail loudly with clear error messages before GPU dispatch.

Example: Alternative Solution

import torch
import torch.nn as nn

def safe_loss_step(logits, targets, num_classes):
    assert targets.min() >= 0, f'Negative target label found: {targets.min()}'
    assert targets.max() < num_classes, f'Target label {targets.max()} exceeds num_classes {num_classes}'
    criterion = nn.CrossEntropyLoss()
    return criterion(logits, targets)

logits = torch.randn(4, 5)
targets = torch.tensor([0, 1, 2, 3], dtype=torch.long)
loss = safe_loss_step(logits, targets, num_classes=5)
print('Safe step loss:', loss.item())

After a device-side assert is triggered, the CUDA context is corrupted and entering an unrecoverable state. Any subsequent PyTorch CUDA call in the same Python session will immediately raise CUDA driver error: device-side assert triggered even on valid code. You must restart the Python interpreter/kernel after fixing the bug. Always verify zero-indexed labels: if your classes are labeled 1 to 10 in a CSV, map them to 0 to 9 before feeding them into CrossEntropyLoss.

Previous Next