ParserError: Error tokenizing data in Pandas

Verified FixPython 3.10+Pandas 2.0+Silo: pandas

Quick Fix / Solution Rapide

This error occurs when CSV rows contain more delimiters than expected by the header. Use on_bad_lines='skip' or specify the correct delimiter with sep=';'.

Root Cause Analysis

This error occurs when Python tries to parse a delimited CSV file using pd.read_csv(), but encounters rows containing more delimiter fields or columns than defined by the header specification.

Cause 1: Inconsistent Number of Delimiters Per Row

When a CSV file has 2 columns defined in its header line, but row 2 contains extra unescaped commas (e.g. '2,Bob,extra_field'), the parser expects 2 tokens but receives 3, raising pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 3, saw 3.

Cause 2: Wrong Delimiter Parameter (sep=';')

European CSV files generated by Excel frequently use semicolons ; as separators instead of commas ,. If sep=';' is omitted, pandas treats entire lines as single strings until hitting malformed rows.

Cause 3: Unescaped Quotes in Free-Text Fields

Comments or product descriptions containing unescaped quotation marks disrupt line tokenization.

Reproduction Code (MCVE)

Example: Bug Reproduction

import io
import pandas as pd

csv_content = 'id,name\n1,Alice\n2,Bob,extra_field\n3,Carol'
# ParserError: Error tokenizing data
pd.read_csv(io.StringIO(csv_content))

Solution 1: Skip Corrupted Lines with on_bad_lines='skip'

Use on_bad_lines='skip' (or 'warn') in modern Pandas (2.0+) to bypass malformed records automatically during file ingestion.

Example: Recommended Solution

import io
import pandas as pd

csv_content = 'id,name\n1,Alice\n2,Bob,extra_field\n3,Carol'
# Safely bypass malformed rows
df = pd.read_csv(io.StringIO(csv_content), on_bad_lines='skip')
print(df)

Solution 2: Specify the Correct Delimiter and Quoting Parameters

Configure sep=';' or quoting=csv.QUOTE_MINIMAL when ingesting semicolon-separated or heavily quoted data.

Example: Alternative Solution

import io
import pandas as pd

csv_semicolon = 'id;name;city\n1;Alice;Paris\n2;Bob;London'
df = pd.read_csv(io.StringIO(csv_semicolon), sep=';')
print(df)

Common Mistakes & Edge Cases

1. Legacy error_bad_lines vs on_bad_lines

In Pandas 1.3+ and standard in Pandas 2.0+, error_bad_lines=False and warn_bad_lines=False were deprecated and removed in favor of on_bad_lines='skip' or on_bad_lines='warn'.

2. Custom Callables for on_bad_lines

You can pass a custom function to on_bad_lines=custom_handler to log or quarantine offending rows into a separate audit file before skipping them.

3. Contrasting ParserError vs EmptyDataError

ParserError occurs when token counts are inconsistent. EmptyDataError occurs when pd.read_csv() is called on a completely empty 0-byte file.

Previous Next