UnicodeDecodeError: utf-8 codec cant decode byte in Python
This error occurs when reading a file encoded in cp1252/latin1 as UTF-8. Pass the explicit encoding parameter open(file, encoding='latin1') or 'utf-8-sig'.
Root Cause Analysis
This error occurs when Python tries to decode a sequence of raw bytes into a UTF-8 string using the open() function or str.decode(), but the byte sequence contains byte values that do not conform to valid UTF-8 encoding rules.
Cause 1: Ingesting Files Encoded in Windows-1252 or Latin-1 (ISO-8859-1)
Files created by Windows applications (such as Excel CSV exports) commonly use legacy encodings like cp1252 or iso-8859-1. Characters like accented vowels (é, à) or smart quotes are represented by single high-byte values (e.g. 0xe9) that are invalid start bytes in UTF-8, triggering UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 4: invalid continuation byte.
Cause 2: Default System Encodings Differ Across Operating Systems
In Python versions prior to Python 3.15, calling open('file.txt') without an encoding argument uses the OS locale default (cp1252 on Windows, utf-8 on Linux/macOS), causing scripts to succeed on Mac but crash on Windows.
Cause 3: Reading Binary Files in Text Mode
Opening binary formats (PDFs, images, compiled objects) using 'r' instead of 'rb' causes the text decoder to fail when encountering arbitrary binary bytes.
Reproduction Code (MCVE)
raw_bytes = b'Donn\xe9es client'
# Decoding ISO-8859-1 byte (0xe9) as UTF-8 raises UnicodeDecodeError
raw_bytes.decode('utf-8')
Solution 1: Specify the Explicit Encoding Parameter
Provide the exact encoding (such as encoding='latin1' or encoding='cp1252') when opening files or reading datasets with open() or pd.read_csv().
raw_bytes = b'Donn\xe9es client'
# Decode with matching legacy encoding
decoded_text = raw_bytes.decode('latin1')
print(f'Decoded string: {decoded_text}')
Solution 2: Handle UTF-8 Byte Order Marks with encoding='utf-8-sig'
When reading CSV files from Microsoft Excel containing a UTF-8 BOM (\ufeff), pass encoding='utf-8-sig' to automatically strip the header mark.
bom_bytes = b'\xef\xbb\xbfid,name\n1,Alice'
cleaned_text = bom_bytes.decode('utf-8-sig')
print(f'BOM stripped header: {cleaned_text.splitlines()[0]}')
Common Mistakes & Edge Cases
1. Always Specify encoding='utf-8' Explicitly
Never write bare open('file.txt', 'r'). Always write open('file.txt', 'r', encoding='utf-8') to ensure cross-platform reproducibility across Windows, Linux, and macOS.
2. Error Handling Strategies (errors='replace' / 'ignore')
When parsing corrupted log streams where dropping bad characters is acceptable, pass errors='replace' (replaces bad bytes with ?) or errors='ignore'.
3. Contrasting UnicodeDecodeError vs UnicodeEncodeError
UnicodeDecodeError: Fails when converting rawbytes$\rightarrow$str(reading input).UnicodeEncodeError: Fails when convertingstr$\rightarrow$ rawbytes(writing output to an ASCII stream).