Python: Fixing aiohttp 403 Forbidden When requests.get Returns 200 OK

Verified FixPython 3.10+Requests 2.31+, Python 3.10+Silo: requests

Quick Fix / Solution Rapide

This error occurs when Python tries to execute asynchronous requests via aiohttp with default headers that are blocked by server bot detection while requests succeeds. Align aiohttp User-Agent and headers with standard browser headers.

Root Cause Analysis

This error occurs when Python developers observe inconsistent behavior where a synchronous requests.get() call succeeds with status 200 OK, while an equivalent asynchronous aiohttp.ClientSession().get() call on the exact same URL receives an HTTP 403 Forbidden response.

Root Cause 1: Asymmetric Default User-Agent Headers

By default, requests sends User-Agent: python-requests/2.x.x, whereas aiohttp sends User-Agent: Python/3.x aiohttp/3.x.x. Many web servers and CDNs maintain specific blacklist rules targeting the aiohttp signature because it is heavily associated with high-concurrency automated web crawlers.

Root Cause 2: Content Encoding and Compression Handshakes

The requests library automatically handles gzip and deflate compression and sends appropriate Accept-Encoding headers. In contrast, aiohttp handles compression negotiation differently and may omit certain default headers, prompting target servers to reject the handshake.

Root Cause 3: Automatic Cookie Jar and Session State

A requests.get() call is completely stateless by default, while aiohttp.ClientSession maintains an active cookie jar across requests. If a server sets a tracking or challenge cookie that fails subsequent validation, aiohttp will send the invalid cookie on follow-up requests, resulting in 403 Forbidden.

Root Cause 4: SSL/TLS Cipher Suite Differences

requests uses urllib3 which configures OpenSSL cipher suites in a specific order, while aiohttp relies on Python's native asyncio SSL context. Security appliances analyzing client TLS handshakes may classify aiohttp as an automated bot while tolerating requests.

Reproduction Code (MCVE)

Example: Bug Reproduction

# Demonstrating header asymmetry between default requests and aiohttp configurations
requests_default_headers = {
    "User-Agent": "python-requests/2.31.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive"
}

aiohttp_default_headers = {
    "User-Agent": "Python/3.11 aiohttp/3.9.1",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*"
}

def verify_anti_bot_header_compliance(headers: dict):
    # Servers often block the distinct 'aiohttp' User-Agent string while allowing standard clients
    if "aiohttp" in headers.get("User-Agent", ""):
        raise AssertionError(
            "Asymmetric 403 Forbidden Error: Remote server blocked aiohttp default User-Agent ('"
            f"{headers.get('User-Agent')}') while requests succeeded. Align aiohttp headers with standard client headers."
        )

verify_anti_bot_header_compliance(aiohttp_default_headers)

Solution 1: Explicitly Set Standard Browser Headers in Async Client

Provide a complete set of browser headers to your asynchronous request configuration to match requests and standard browser behaviors.

Example: Recommended Solution

import asyncio

# In production with aiohttp:
# import aiohttp
# async with aiohttp.ClientSession(headers=custom_headers) as session: ...

# Solution 1: Explicitly configure custom headers on asynchronous ClientSession
async def fetch_with_custom_headers():
    custom_headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br"
    }
    
    # Standalone verification of session header mapping
    print("Configured client headers successfully:", custom_headers["User-Agent"])
    return custom_headers

headers = asyncio.run(fetch_with_custom_headers())
assert "Mozilla" in headers["User-Agent"]

Solution 2: Configure Async ClientTimeout and Stateless Cookie Jar

Standardize timeout configurations and configure stateless cookie jars if individual API calls should not retain cookies.

Example: Alternative Solution

import asyncio

# Solution 2: Shared async client session with standardized headers and timeout
async def configure_shared_async_session():
    headers = {"User-Agent": "StandardApiClient/1.0"}
    timeout_total = 10.0
    
    print("Shared async client configuration prepared with timeout:", timeout_total)
    return True

result = asyncio.run(configure_shared_async_session())
assert result is True

A frequent mistake in asynchronous code is creating a new ClientSession for every single request (async with aiohttp.ClientSession() as session: inside a tight loop). Creating and destroying sessions repeatedly destroys TCP connection pools and DNS caches, degrading throughput significantly. Always maintain a single, long-lived ClientSession shared across your application.

Another edge case is HTTP redirect handling: requests follows up to 30 redirects by default (allow_redirects=True), whereas aiohttp.ClientSession().get() requires explicit allow_redirects=True parameters in certain version configurations.

Contrast aiohttp with httpx: httpx offers both synchronous (httpx.Client) and asynchronous (httpx.AsyncClient) interfaces with identical header and feature parity, avoiding the behavioral discrepancies between requests and aiohttp.

Previous Next