ValueError: Cannot convert non-finite values (NA or inf) to integer in Pandas

Verified FixPython 3.10+Pandas 2.0+Silo: pandas

Quick Fix / Solution Rapide

Use Pandas' nullable integer dtype .astype('Int64') (capitalized) or fill missing values with .fillna(0) before casting to native int.

Root Cause Analysis

This error occurs when Python tries to convert a pandas column containing missing (NaN, pd.NA) or infinite (inf) values into a standard NumPy integer dtype, but native integer types cannot represent non-finite values.

Cause 1: Missing Values (np.nan) in Integer Columns

np.nan is a special float value — not a missing-value sentinel that standard integer types can represent. Integer columns in NumPy have no concept of 'missing': every memory cell must hold a valid integer. Calling df['col'].astype(int) on a column containing NaN raises ValueError: Cannot convert non-finite values (NA or inf) to integer.

Cause 2: Infinite Values (np.inf / -np.inf) from Division

Dividing numeric columns by zero produces np.inf or -np.inf. Infinite floats cannot be represented as integers, triggering the same ValueError upon casting.

Cause 3: Ingestion of Nullable Database Columns

SQL columns with NULL or CSV columns with missing integers are automatically upcast to float64 by pandas upon loading. Attempting to force them back to int causes casting failures.

Reproduction Code (MCVE)

Example: Bug Reproduction

import pandas as pd
import numpy as np

df = pd.DataFrame({'user_id': [101.0, np.nan, 103.0]})
# ValueError: Cannot convert non-finite values (NA or inf) to integer
df['user_id'] = df['user_id'].astype(int)

Solution 1: Use Pandas Nullable Integer Extension Dtype ('Int64')

Use Pandas' built-in nullable integer dtype 'Int64' (capitalized 'I'). It supports missing values natively using pd.NA without converting the column to float.

Example: Recommended Solution

import pandas as pd
import numpy as np

df = pd.DataFrame({'user_id': [101.0, np.nan, 103.0]})
# Use nullable Int64 dtype to preserve missing values natively
df['user_id'] = df['user_id'].astype('Int64')
print(df)
print(f'Dtype: {df["user_id"].dtype}')

Solution 2: Impute Missing Values and Replace Infs Before Casting

Replace infinite values and impute NaN with a defined sentinel value (such as 0 or -1) before casting to primitive integer types.

Example: Alternative Solution

import pandas as pd
import numpy as np

df = pd.DataFrame({'user_id': [101.0, np.nan, np.inf]})
# Replace inf with nan, then impute with sentinel value
clean_ids = df['user_id'].replace([np.inf, -np.inf], np.nan).fillna(0).astype(int)
print(clean_ids)

Common Mistakes & Edge Cases

1. int64 vs Int64: The Capitalization Difference

'int64' (lowercase) refers to NumPy's native 64-bit integer, which crashes on NaN.
'Int64' (capitalized) refers to Pandas' nullable integer extension type, which supports pd.NA seamlessly.

2. Contrasting ValueError vs TypeError

Attempting .astype(int) on non-finite values raises ValueError during data conversion. In contrast, evaluating mathematical operations between pd.NA and unsupported types raises TypeError.

3. Safe Pipeline Ingestion

Specify dtype={'user_id': 'Int64'} directly in pd.read_csv() or pd.read_sql() to prevent columns from ever being cast to float64 during ingestion.

Previous Next