HTTP 403 Forbidden Error when Web Scraping with Python

Verified FixPython 3.10+requests 2.x / urllibSilo: scraping

Quick Fix / Solution Rapide

Add realistic browser User-Agent and Accept headers to your requests.get(url, headers=headers) call, or manage cookies via requests.Session().

Root Cause Analysis

This error occurs when Python tries to scrape web pages via HTTP requests, but the target web server actively denies access and returns an HTTP 403 Forbidden status code because it detected an automated bot (such as python-requests default User-Agent headers or missing browser headers).

Why Web Servers Block Automated Scrapers

When requests.get('https://example.com') is called without custom headers, Python sends an HTTP request with the default header User-Agent: python-requests/2.31.0. Modern web servers and Web Application Firewalls (such as Cloudflare, Akamai, and AWS WAF) automatically block known bot User-Agents to prevent scraping and abuse.

Key Detection Vectors for 403 Forbidden

Default User-Agent Header: Immediate rejection of strings containing python-requests, urllib, or aiohttp.
Missing Standard Browser Headers: Browsers always send headers like Accept-Language, Accept-Encoding, Sec-Fetch-Dest, and Sec-Ch-Ua.
Aggressive Request Rate: Sending dozens of requests per second from a single IP address triggers automated rate-limiting.
Missing Session Cookies / CSRF Tokens: Sites requiring initial handshake cookies before serving content pages.
TLS Fingerprinting (JA3 / JA4): Advanced anti-bot firewalls analyzing the cryptographic TLS handshake characteristics.

Reproduction Code (MCVE)

Example: Bug Reproduction

import requests

# Simulating an HTTP 403 Forbidden response check
response = requests.Response()
response.status_code = 403
response.url = 'https://example-protected-job-portal.com/api/jobs'
response.raise_for_status()

Solution 1: Set Realistic Browser Headers with `requests.Session`

Configure standard browser User-Agent, Accept, and Accept-Language headers on a persistent session object.

Example: Recommended Solution

import requests

def get_authenticated_session() -> requests.Session:
    session = requests.Session()
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'gzip, deflate, br',
        'DNT': '1',
        'Upgrade-Insecure-Requests': '1'
    })
    return session

session = get_authenticated_session()
print(f'Prepared session with User-Agent: {session.headers["User-Agent"][:45]}...')

Solution 2: Implement Exponential Backoff and Rate Limiting Delays

Add delays between scraping requests and use retry adapters to handle transient blocks politely.

Example: Alternative Solution

import time
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

session = requests.Session()
retries = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retries)
session.mount('https://', adapter)
print('Configured resilient HTTP session with exponential backoff adapter.')

Note de reproductibilité

La reproductibilité de l'erreur 403 Forbidden varie selon les politiques de sécurité (Cloudflare, Akamai, WAF) et les mécanismes de limitation de débit du serveur web cible.

Edge Cases & Advanced Scraping Ethics

Always check the website's robots.txt file (e.g. https://example.com/robots.txt) to respect crawling directives and terms of service. For sites protected by JavaScript challenges (Cloudflare Turnstile), simple header modifications with requests may be insufficient; consider browser automation tools like Playwright or Selenium with stealth plugins.

Previous Next