PyTorch: CUDA error: device-side assert triggered on Colab and Cloud GPUs

Verified FixPython 3.10+PyTorch 2.2+Silo: pytorch

Quick Fix / Solution Rapide

This error occurs when Python executes CUDA kernel assertions caused by out-of-bounds class labels in loss functions (e.g. CrossEntropyLoss) or invalid embedding indices. Set CUDA_LAUNCH_BLOCKING=1 to identify the exact line and check target indices.

Root Cause Analysis

This error occurs when Python executes an asynchronous CUDA GPU kernel that encounters an invalid memory access or failed assertion (such as accessing a tensor index outside valid dimension bounds) during training on Google Colab or cloud GPUs.

Root Cause 1: Target Class Labels Exceeding num_classes in Loss Functions

The most common cause is passing target labels y to nn.CrossEntropyLoss() or nn.NLLLoss() where label values fall outside [0, num_classes - 1]. For example, in a 10-class classification task (classes 0 through 9), having a target label 10 or a negative label (other than -100 for ignore_index) will trigger a GPU kernel assertion violation.

Root Cause 2: Asynchronous CUDA Kernel Execution and Delayed Error Reporting

Because CUDA kernel launches in PyTorch are asynchronous, Python does not halt at the exact line of code that triggered the bad memory access. Instead, the crash surfaces several lines later during a tensor synchronization point (such as loss.item(), loss.backward(), or optimizer.step()), obscuring the true origin.

Root Cause 3: Out-of-Bounds Vocabulary Indices in nn.Embedding

In NLP transformer models, feeding token IDs to an nn.Embedding(vocab_size, d_model) layer where token IDs $\ge ext{vocab_size}$ or $< 0$ triggers a device-side assertion.

Root Cause 4: Permanent CUDA Context Corruption

Once a device-side assertion triggers, the entire CUDA context on the GPU becomes permanently corrupted. In Google Colab or Jupyter notebooks, subsequent cells will continue failing with CUDA error: device-side assert triggered until the Python runtime or kernel is completely restarted.

Reproduction Code (MCVE)

Example: Bug Reproduction

# Simulating an out-of-bounds target index triggering a device-side assert in classification loss
def validate_classification_targets(logits_shape: tuple, targets: list):
    num_classes = logits_shape[1]
    for idx, target in enumerate(targets):
        if target < 0 or target >= num_classes:
            raise RuntimeError(
                f"CUDA error: device-side assert triggered. "
                f"Target label {target} at index {idx} is out of bounds for num_classes={num_classes} "
                f"(valid range is [0, {num_classes - 1}])."
            )

# Simulating 5-class model output with an invalid target class '5'
batch_logits_shape = (4, 5)  # batch_size=4, num_classes=5 (classes 0..4)
invalid_targets = [0, 2, 4, 5]  # '5' is out of bounds!
validate_classification_targets(batch_logits_shape, invalid_targets)

Solution 1: Enable CUDA_LAUNCH_BLOCKING and Verify Target Index Bounds

Set the environment variable CUDA_LAUNCH_BLOCKING=1 before importing torch to force synchronous kernel execution and validate target bounds with assertions.

Example: Recommended Solution

import os

# Solution 1: Synchronous debugging and target validation pattern
# Set before torch operations:
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"

def safe_compute_loss_inputs(num_classes: int, targets: list[int]):
    # Assert bounds before dispatching to GPU kernels
    min_t, max_t = min(targets), max(targets)
    assert min_t >= 0, f"Target labels must be non-negative, found: {min_t}"
    assert max_t < num_classes, f"Target label {max_t} exceeds num_classes-1 ({num_classes-1})"
    return True

# Validated targets within range [0, 4] for 5 classes
valid_targets = [0, 1, 3, 4]
assert safe_compute_loss_inputs(5, valid_targets) is True
print("Target bounds verified successfully for CUDA dispatch.")

Solution 2: Clamp or Remap Labels Before Loss Computation

Sanitize target labels using torch.clamp or remap categorical labels using Scikit-Learn LabelEncoder to ensure contiguous zero-indexed classes.

Example: Alternative Solution

# Solution 2: Label remapping and validation
raw_category_ids = [101, 102, 105, 101, 102]

# Map arbitrary category IDs to contiguous indices 0..K-1
unique_cats = sorted(list(set(raw_category_ids)))
cat_to_idx = {cat: idx for idx, cat in enumerate(unique_cats)}

zero_indexed_targets = [cat_to_idx[cat] for cat in raw_category_ids]
print("Remapped 0-indexed targets:", zero_indexed_targets)
assert max(zero_indexed_targets) < len(unique_cats)

Whenever you encounter this error in Google Colab, you MUST restart the Colab runtime (Runtime -> Restart Session) before re-running your code. Once CUDA asserts on hardware, all subsequent CUDA calls in that Python process will fail with the same error, even if you fixed the bug in your code.

Note de reproductibilité : Cette assertion GPU dépend de l'état matériel du GPU, de la configuration du driver CUDA et de l'activation du mode asynchrone par défaut dans PyTorch.

Contrast CUDA error: device-side assert triggered with CUDA out of memory: OOM indicates the GPU VRAM capacity is exhausted by batch sizes/model parameters; a device-side assert indicates illegal memory access or failed algorithmic invariant.

Previous Next