Python Requests: Avoiding 403 Forbidden When Downloading Files and Assets

Verified FixPython 3.10+Requests 2.31+Silo: requests

Quick Fix / Solution Rapide

This error occurs when Python tries to download hosted assets without providing expected Referer and browser User-Agent headers. Include Referer headers matching the host origin and download files in chunks using stream=True.

Root Cause Analysis

This error occurs when Python tries to download binary files, documents, or media assets (such as PDFs, ZIP archives, or images) from a remote web server that enforces anti-hotlinking rules or direct download restrictions.

Root Cause 1: CDN Hotlink Protection Policies

Many web hosts and Content Delivery Networks (CDNs) enforce hotlink protection to prevent third-party scripts from consuming their bandwidth. The CDN inspects the incoming HTTP Referer header: if the Referer is missing or does not match an approved origin domain, the server rejects the download with a 403 Forbidden status code.

Root Cause 2: Automated Downloader Fingerprinting

Direct calls to requests.get(url) send Python's default User-Agent string. Many asset repositories, academic archives, and media hosts actively block non-browser User-Agents to prevent bulk scraping and automated asset mirroring.

Root Cause 3: Signed URL Expiration and Parameter Tampering

Modern file storage platforms (such as Amazon S3, Google Cloud Storage, or Azure Blob Storage) use pre-signed URLs with cryptographic signature parameters (AWSAccessKeyId, Signature, Expires). If the pre-signed URL has expired or if special characters in query parameters were improperly encoded, the storage server responds with 403 Forbidden.

Root Cause 4: Missing Cookie-Based Session Verification

Download endpoints frequently require that a user has already visited the parent page or logged in, establishing a session cookie. Making an isolated GET request without supplying previously issued session cookies will trigger a permission denial.

Reproduction Code (MCVE)

Example: Bug Reproduction

import requests

# Simulating direct file download blocked by remote hotlink protection or CDN anti-bot rules
class MockFileDownloadResponse:
    status_code = 403
    def raise_for_status(self):
        raise requests.exceptions.HTTPError(
            "403 Client Error: Forbidden for url: https://cdn.example.org/files/document.pdf - "
            "Direct asset download blocked due to missing Referer and browser headers."
        )

res = MockFileDownloadResponse()
res.raise_for_status()

Solution 1: Emulate Browser Referer and Stream File in Chunks

Supply the parent page URL in the Referer header and stream the download in chunks using stream=True to prevent memory exhaustion.

Example: Recommended Solution

import requests

# Solution 1: Provide Referer and User-Agent headers to satisfy CDN hotlink rules
download_headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
    "Referer": "https://example.org/downloads",
    "Accept": "application/pdf,application/octet-stream,*/*"
}

# Demonstrating chunked stream download pattern
def stream_download_simulation(headers: dict):
    print("Stream download initialized with headers:", headers)
    chunks = [b"%PDF-1.5", b" content_chunk_1", b" content_chunk_2"]
    total_size = sum(len(c) for c in chunks)
    return total_size

size = stream_download_simulation(download_headers)
print(f"Downloaded {size} bytes successfully.")
assert size > 0

Solution 2: Use requests.Session with Download Streaming Context

Use a persistent session to first load the landing page and acquire cookies, then download the target asset using response.iter_content().

Example: Alternative Solution

import requests
import io

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Referer": "https://example.org/files/"
})

# Mocking the streaming write pattern
buffer = io.BytesIO()
mock_chunks = [b"Chunk 1 ", b"Chunk 2 ", b"Chunk 3"]
for chunk in mock_chunks:
    buffer.write(chunk)

print("Downloaded buffer size:", buffer.tell(), "bytes")
assert buffer.tell() == sum(len(c) for c in mock_chunks)

A common trap when downloading large files with Python requests is omitting stream=True. Without stream=True, requests loads the entire response body into system RAM immediately. Downloading large datasets (such as 2GB ZIP archives) will quickly trigger MemoryError or crash the operating system process. Always use with requests.get(url, stream=True) as r: and iterate over r.iter_content(chunk_size=8192).

Note de reproductibilité : Le blocage varie selon les politiques de sécurité du CDN hébergeant le fichier, la validité temporelle des URL pré-signées et les restrictions de hotlinking du site source.

Contrast 403 Forbidden with 404 Not Found: 404 indicates the requested file URL does not exist on the server, whereas 403 confirms the file exists but the server refuses permission due to missing credentials, referrer headers, or CDN anti-scraping rules.

Previous Next