Catch HTTP Errors When Using the Requests Module in Python 3

Verified FixPython 3.10+Requests 2.31+Silo: requests

Quick Fix / Solution Rapide

Call response.raise_for_status() inside a try/except requests.exceptions.HTTPError: block to intercept 4xx and 5xx responses.

Root Cause Analysis

This error occurs when Python tries to execute an HTTP request using the requests library and encounters an unsuccessful HTTP status code (4xx client error or 5xx server error) when response.raise_for_status() is called.

1. Non-Raising Behavior of Default Requests

By default, requests.get() or requests.post() does NOT raise an exception when a server returns an HTTP 404 (Not Found), 401 (Unauthorized), or 500 (Internal Server Error). Instead, requests populates the response.status_code and response.ok attributes. An explicit call to response.raise_for_status() is required to turn bad HTTP status codes into exceptions.

2. Unhandled `HTTPError` Exceptions

When response.raise_for_status() encounters a status code in the 400–599 range, it raises requests.exceptions.HTTPError. If the calling function does not enclose the call in a try/except block, the entire Python program crashes.

3. The Hierarchy of Requests Exceptions

HTTPError, ConnectionError, and Timeout all inherit from requests.exceptions.RequestException. Catching only standard Exception or omitting connection handling leaves applications vulnerable to network dropouts.

4. API Error Payloads and Diagnostics

Modern REST APIs often return structured JSON payloads explaining why a 400 Bad Request occurred. Crashing without inspecting response.text prevents developers from diagnosing why the request was rejected.

Reproduction Code (MCVE)

Example: Bug Reproduction

import requests

# Simulating an HTTP 404 response that calls raise_for_status() without try/except
response = requests.Response()
response.status_code = 404
response.url = 'https://api.example.com/missing-endpoint'
response.raise_for_status()

Solution 1: Wrap raise_for_status() in Specific try/except Blocks

Catch requests.exceptions.HTTPError specifically to handle API error codes cleanly without terminating execution.

Example: Recommended Solution

import requests

response = requests.Response()
response.status_code = 404
response.url = 'https://api.example.com/missing-endpoint'

try:
    response.raise_for_status()
except requests.exceptions.HTTPError as http_err:
    print(f'HTTP Error intercepted: {http_err}')
    print(f'Status Code: {response.status_code}')

Solution 2: Inspect response.ok Attribute Before Processing

Check the boolean response.ok attribute (True for status codes < 400) to branch logic without raising exceptions.

Example: Alternative Solution

import requests

def fetch_resource():
    response = requests.Response()
    response.status_code = 404
    if not response.ok:
        print(f'API returned non-200 status: {response.status_code}')
        return None
    return response

result = fetch_resource()
print(f'Handled cleanly, result: {result}')

A common mistake is wrapping requests.get() in try/except requests.exceptions.HTTPError: without calling response.raise_for_status(). Because requests.get() only raises on network failure, 404 or 500 responses will bypass the except block completely and pass bad HTML into .json(). Always call response.raise_for_status(). Another critical edge case is catching requests.exceptions.RequestException too broadly at low levels: always log the response body (response.text) before re-raising or returning fallback data. Contrast this error with requests.exceptions.ConnectionError, which occurs before any HTTP handshake can be completed.

Previous Next