Python Requests: Resolving json.decoder.JSONDecodeError in response.json()

Verified FixPython 3.10+Requests 2.31+, Python 3.10+Silo: requests

Quick Fix / Solution Rapide

This error occurs when Python tries to parse an empty response or an HTML error page as JSON via response.json(). Inspect response.status_code and response.text before decoding.

Root Cause Analysis

This error occurs when Python tries to deserialize a non-JSON HTTP response body into a Python dictionary or list using response.json() or json.loads().

Root Cause 1: Parsing Empty Response Bodies (Status 204 or Empty 200)

When an API returns an empty response body (such as HTTP 204 No Content, HTTP 202 Accepted, or a 200 OK with Content-Length: 0), response.text is an empty string "". Calling .json() on an empty string causes Python's standard json decoder to raise json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).

Root Cause 2: Server Returning HTML Error Pages Instead of JSON

When an upstream server encounters an error (such as HTTP 500 Internal Server Error, 502 Bad Gateway, or 404 Not Found), the reverse proxy (Nginx, Cloudflare, Apache) often generates an HTML error page (<html><body>502 Bad Gateway</body></html>). When requests attempts to parse this HTML text as JSON, the parser fails immediately at the first < character with JSONDecodeError: Expecting value: line 1 column 1 (char 0).

Root Cause 3: API Responses with Preceding Whitespace or UTF-8 BOM

Some legacy APIs prepend a Byte Order Mark (\xef\xbb\xbf) or extraneous non-printable control characters to JSON payloads. In certain parsing configurations, unstripped BOM prefixes can cause decoding failures.

Root Cause 4: API Rate Limit and Captcha Interceptions

When rate limits are exceeded, security gateways intercept requests and return JavaScript challenge pages or Captchas instead of the requested JSON resource. Attempting to parse the challenge script raises decoding exceptions.

Reproduction Code (MCVE)

Example: Bug Reproduction

import json

# Simulating response.json() on an empty string or HTML error page
raw_response_text = ""

# Attempting to decode empty text triggers json.decoder.JSONDecodeError
json.loads(raw_response_text)

Solution 1: Inspect Status Code and Response Text Before Parsing

Verify that response.status_code indicates success and that response.text.strip() contains content before calling .json().

Example: Recommended Solution

import json

# Solution 1: Validate response status and content before decoding
response_text = '{"status": "ok", "items": ["alpha", "beta", "gamma"]}'
response_status_code = 200

if response_status_code == 200 and response_text.strip():
    parsed_data = json.loads(response_text)
    print("Decoded JSON payload successfully:")
    print(parsed_data)
else:
    print(f"Server returned non-JSON or status {response_status_code}")

assert isinstance(parsed_data, dict)
assert len(parsed_data["items"]) == 3

Solution 2: Implement Safe JSON Parsing with Exception Fallback

Wrap JSON decoding in a dedicated helper function that catches json.JSONDecodeError and returns structured fallback data or logs the raw HTML text.

Example: Alternative Solution

import json

def safe_extract_json(response_text: str, default_fallback=None):
    if not response_text or not response_text.strip():
        return default_fallback
    try:
        return json.loads(response_text)
    except json.JSONDecodeError as decode_err:
        print(f"Warning: Failed to decode response as JSON: {decode_err}")
        return default_fallback

# Test valid JSON decoding
valid_payload = safe_extract_json('{"success": true, "code": 200}')
print("Valid payload parsed:", valid_payload)

# Test graceful fallback on empty response
empty_payload = safe_extract_json("", default_fallback={"error": "Empty body"})
print("Empty payload fallback:", empty_payload)
assert empty_payload["error"] == "Empty body"

A very common pitfall is checking if response.json(): without wrapping it in a try...except block or checking response.ok. If the server returns a 500 HTML page, response.json() will raise an exception before the if condition can even evaluate. Always check if response.ok and response.text: or wrap the call in try...except requests.exceptions.JSONDecodeError.

Another edge case is endpoint redirects: if an API endpoint redirects (301 Moved Permanently or 302 Found) to a login page, requests follows the redirect by default and receives HTML login markup.

Contrast JSONDecodeError with KeyError: JSONDecodeError occurs when the raw text cannot be parsed into a JSON structure, whereas KeyError occurs after successful parsing when trying to access a dictionary key that does not exist in the JSON payload.

Previous Next