ValueError: Input contains NaN, infinity or a value too large for dtype(float64)

Verified FixPython 3.10+Scikit-Learn 1.4+Silo: scikit-learn

Quick Fix / Solution Rapide

Impute or remove NaN and infinite values before model training using sklearn.impute.SimpleImputer or df.fillna().

Root Cause Analysis

This error occurs when Python tries to train or transform a Scikit-Learn estimator on a dataset that contains missing values (np.nan), infinite floats (np.inf, -np.inf), or floating-point numbers that exceed IEEE 754 float64 limits.

1. Mathematical Assumptions of Classical Estimators

Most Scikit-Learn algorithms (such as LogisticRegression, LinearRegression, SVM, and KMeans) rely on underlying linear algebra libraries (BLAS/LAPACK) that compute dot products and matrix inverses. Mathematical matrix operations cannot propagate undefined NaN or inf values without corrupting gradient and loss calculations.

2. Common Sources of NaN and Inf in Pipelines

Missing values arise from unrecorded survey fields, outer table joins, or failed data conversions. Infinite values typically stem from unhandled division by zero (e.g. calculating price ratios where denominator is zero) or logarithmic transformations of zero.

3. Resolving Missing Values with Imputers

Scikit-Learn provides SimpleImputer (mean, median, most_frequent, or constant strategy) and KNNImputer to replace missing values systematically.

4. Pipeline Integration

Encapsulate imputers directly inside an sklearn.pipeline.Pipeline to prevent data leakage between training and validation splits.

Reproduction Code (MCVE)

Example: Bug Reproduction

import numpy as np
from sklearn.linear_model import LogisticRegression

# Dataset containing NaN values
X = np.array([[1.0, 2.0], [np.nan, 4.0], [5.0, 6.0]])
y = np.array([0, 1, 0])

# ValueError: Input contains NaN, infinity or a value too large for dtype('float64')
clf = LogisticRegression()
clf.fit(X, y)

Solution 1: Impute Missing Values with SimpleImputer

Apply SimpleImputer to replace NaN values with the mean or median of the training features.

Example: Recommended Solution

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

X = np.array([[1.0, 2.0], [np.nan, 4.0], [5.0, 6.0]])
y = np.array([0, 1, 0])

# Impute NaN values with feature mean
imputer = SimpleImputer(strategy='mean')
X_clean = imputer.fit_transform(X)

clf = LogisticRegression()
clf.fit(X_clean, y)
print('Model trained successfully. Coefficients:', clf.coef_)

Solution 2: Replace Infinite Values and Clean using NumPy

Use np.isfinite and np.nan_to_num to replace positive and negative infinite values with finite upper bounds.

Example: Alternative Solution

import numpy as np
from sklearn.linear_model import Ridge

X = np.array([[1.0, 2.0], [np.inf, 4.0], [5.0, 6.0]])
y = np.array([10.0, 20.0, 30.0])

# Replace inf with max finite float and nan with zero
X_clean = np.nan_to_num(X, nan=0.0, posinf=1e5, neginf=-1e5)

model = Ridge()
model.fit(X_clean, y)
print('Ridge model fitted successfully:', model.intercept_)

A dangerous practice is calling df.dropna() on the full dataset before splitting into train/test sets, which creates data distribution shifts. Always fit imputers exclusively on the training set (imputer.fit(X_train)) and transform both train and test sets (imputer.transform(X_test)). Contrast this ValueError with HistGradientBoostingClassifier, which supports native NaN handling without explicit imputation. Always inspect np.isnan(X).sum() and np.isinf(X).sum() during exploratory data analysis.

Previous Next