RuntimeError: Expected all tensors to be on the same device in PyTorch

Verified FixPython 3.10+PyTorch 2.0+Silo: pytorch

Quick Fix / Solution Rapide

Transfer all tensors and model parameters to the same target device using .to(device) before mathematical operations.

Root Cause Analysis

This error occurs when Python tries to execute a mathematical operation or neural network layer between two PyTorch tensors that reside on different hardware memory devices (such as CPU host memory and GPU CUDA device memory).

1. PyTorch Explicit Memory Management

PyTorch does not perform implicit data transfers between different computing devices. Memory allocated on CPU host RAM cannot be directly referenced by GPU compute kernels without an explicit DMA memory transfer. When an operator like torch.add() or nn.CrossEntropyLoss() receives tensors on differing devices, PyTorch fails fast with RuntimeError: Expected all tensors to be on the same device.

2. Common Scenarios in Deep Learning Pipelines

This error typically arises during training loops when: (1) model parameters have been moved to GPU with model.to(device), but incoming DataLoader batches remain on CPU; (2) target labels or mask tensors are instantiated inside forward() without passing device=x.device; or (3) multi-GPU models dispatch operations across cuda:0 and cuda:1 without proper synchronization.

3. Best Practices for Device Synchronization

Define a single dynamic device variable at the top of your training script: device = torch.device('cuda' if torch.cuda.is_available() else 'cpu'). Ensure both the model and every batch (inputs, labels) invoke .to(device, non_blocking=True).

4. Debugging Device Allocations

Print tensor.device and next(model.parameters()).device before executing forward passes to identify offending CPU/GPU mismatches immediately.

Reproduction Code (MCVE)

Example: Bug Reproduction

import torch

# Tensor 1 on CPU, Tensor 2 on meta device (simulating cross-device mismatch)
t1 = torch.tensor([1.0, 2.0], device='cpu')
t2 = torch.tensor([3.0, 4.0], device='meta')

# RuntimeError: Expected all tensors to be on the same device
result = t1 + t2

Solution 1: Move All Tensors to a Common Target Device

Define a single device variable and transfer all participating tensors and models to that device using .to(device).

Example: Recommended Solution

import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Move all tensors to the common device
t1 = torch.tensor([1.0, 2.0]).to(device)
t2 = torch.tensor([3.0, 4.0]).to(device)

result = t1 + t2
print(f'Computed result on {device}:', result)

Solution 2: Create Dynamic Tensors Matching Module Parameter Device

Inside custom nn.Module classes, always inherit the device of existing parameter weights when creating intermediate tensors.

Example: Alternative Solution

import torch
import torch.nn as nn

class NormalizedLinear(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.linear = nn.Linear(in_features, out_features)

    def forward(self, x):
        # Create buffer on the exact device and dtype of the layer weights
        bias_offset = torch.zeros_like(x, device=self.linear.weight.device)
        return self.linear(x + bias_offset)

model = NormalizedLinear(4, 2)
input_data = torch.randn(1, 4)
output = model(input_data)
print('Forward pass successful:', output.shape)

A common beginner mistake is calling model.to(device) (which mutates the model in-place) and expecting tensor.to(device) to also mutate in-place. Tensors are immutable regarding device placement: tensor = tensor.to(device) is mandatory. Contrast Expected all tensors to be on the same device with Input type and weight type should be the same, which represents data type (dtype) mismatches rather than hardware device mismatches. In multi-GPU DistributedDataParallel (DDP) setups, ensure each process binds exclusively to its local rank device torch.cuda.set_device(local_rank).

Previous Next