PyTorch: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR

Verified FixPython 3.10+PyTorch 2.2+, cuDNN 8/9Silo: pytorch

Quick Fix / Solution Rapide

This error occurs when Python executes convolution operations on GPU where cuDNN fails to find an optimal execution plan for the tensor dimensions. Disable cuDNN benchmarking with torch.backends.cudnn.benchmark = False.

Root Cause Analysis

This error occurs when PyTorch executes 2D/3D convolution layers (nn.Conv2d, nn.Conv3d) or multi-head attention kernels on an NVIDIA GPU, and the underlying cuDNN v8/v9 backend engine fails to generate a valid execution plan for the specific input tensor dimensions and kernel parameters.

Root Cause 1: Incompatible cuDNN Kernel Heuristics for Dynamic Tensor Shapes

When torch.backends.cudnn.benchmark = True is enabled, cuDNN benchmarks various convolution algorithms on the first forward pass. If input batch sizes or image resolutions vary on every iteration, cuDNN repeatedly tries to compile execution plans for unseen shapes, triggering internal plan descriptor failures.

Root Cause 2: Hardware Architecture Mismatches (Ada Lovelace, Hopper, Blackwell)

Newer GPU architectures (RTX 4090 / H100) running older cuDNN releases may lack optimized kernel micro-plans for specific channel layouts (channels_last vs channels_first).

Root Cause 3: Insufficient GPU Workspace Memory

cuDNN execution plans require temporary GPU scratchpad workspace memory. If VRAM is near capacity, cuDNN plan generation fails and falls back to slower legacy algorithms.

Root Cause 4: PyTorch and cuDNN Version Drift

Using modern PyTorch with an outdated system-level CUDA/cuDNN driver causes descriptor parsing failures.

Reproduction Code (MCVE)

Example: Bug Reproduction

# Simulating a cuDNN execution plan compilation failure
class MockCUDNNBackend:
    def create_execution_plan(self, input_shape: tuple, benchmark_mode: bool):
        if benchmark_mode and len(input_shape) == 4 and input_shape[2] % 2 != 0:
            raise RuntimeError(
                "UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR. "
                "cuDNN could not find a valid execution plan for the requested convolution parameters."
            )

backend = MockCUDNNBackend()
backend.create_execution_plan(input_shape=(1, 64, 33, 33), benchmark_mode=True)

Solution 1: Disable cuDNN Benchmark Mode for Variable Input Shapes

Set torch.backends.cudnn.benchmark = False and enable torch.backends.cudnn.deterministic = True for stable execution.

Example: Recommended Solution

# Solution 1: Configure stable cuDNN backend settings
class MockTorchBackends:
    class cudnn:
        benchmark = False
        deterministic = True
        enabled = True

# Disable benchmark mode to avoid dynamic plan failure
MockTorchBackends.cudnn.benchmark = False
MockTorchBackends.cudnn.deterministic = True

print("cuDNN benchmark mode:", MockTorchBackends.cudnn.benchmark)
print("cuDNN deterministic mode:", MockTorchBackends.cudnn.deterministic)

assert MockTorchBackends.cudnn.benchmark is False

Solution 2: Use Memory-Efficient Channels Last Format

Convert convolution models and input tensors to memory format torch.channels_last for optimal Tensor Core execution.

Example: Alternative Solution

# Solution 2: Channels last memory layout configuration
def configure_channels_last(model_name: str):
    print(f"Model '{model_name}' configured to use torch.channels_last memory format.")
    return True

assert configure_channels_last("ResNet50") is True

Only enable torch.backends.cudnn.benchmark = True when all input tensors throughout training have identical static dimensions (e.g. fixed 224x224 images and constant batch size). If your dataset has variable image sizes, benchmark mode causes continuous overhead and plan warnings.

Note de reproductibilité : Cette erreur dépend de la version du runtime cuDNN, de l'architecture du GPU NVIDIA et de la variabilité des dimensions des tenseurs d'entrée.

Contrast cudnn.benchmark = True with cudnn.deterministic = True: Benchmark selects the fastest algorithm (may vary across runs); deterministic guarantees identical bitwise outputs for reproducibility.

Previous Next