Scraping: HTTP Error 404 and NoneType When Scraping HTML Tables with BeautifulSoup

Verified FixPython 3.10+Requests 2.31+, BeautifulSoup4 4.12+Silo: scraping

Quick Fix / Solution Rapide

This error occurs when Python tries to scrape a table from a URL that returns an HTTP 404 Not Found response. Call response.raise_for_status() before passing response.text to BeautifulSoup.

Root Cause Analysis

This error occurs when Python web scraping scripts make an HTTP request to a target webpage that responds with an HTTP 404 Not Found status code, and then pass the 404 error page markup into BeautifulSoup, which fails to find the expected data tables.

Root Cause 1: Passing 404 HTML Error Pages into BeautifulSoup

When a web server returns a 404 Not Found status code, requests.get(url) does not raise an exception by default. It returns a Response object containing the web server's 404 error page HTML (e.g. <html><body>404 Page Not Found</body></html>). When code immediately calls soup.find('table'), BeautifulSoup returns None. Calling .find_all('tr') on None then raises AttributeError: 'NoneType' object has no attribute 'find_all'.

Root Cause 2: Broken Pagination and Stale URLs

Scrapers that iterate through numeric pagination (e.g. f'https://example.com/data?page={page}') often overshoot the final page, hitting a 404 Not Found or empty results page on subsequent iterations.

Root Cause 3: Dynamic JavaScript-Rendered Tables (SPA)

Single Page Applications (React, Vue, Angular) load tables asynchronously via client-side JavaScript. When requests.get() fetches the raw initial HTML, the <table> element has not yet been rendered in the DOM, returning an empty container.

Root Cause 4: URL Slugs Modified Upstream

Target websites frequently update URL structures or rename slug paths. Without robust status code validation, scrapers fail silently or crash on downstream data extraction.

Reproduction Code (MCVE)

Example: Bug Reproduction

import requests

# Simulating scraping an invalid or expired URL that returns 404 Not Found
class Mock404ScrapeResponse:
    status_code = 404
    text = "<html><body><h1>404 Not Found</h1><p>The requested URL was not found on this server.</p></body></html>"
    def raise_for_status(self):
        raise requests.exceptions.HTTPError(
            "404 Client Error: Not Found for url: https://example.com/reports/financial_table_2023.html"
        )

res = Mock404ScrapeResponse()
# Without validation, soup.find('table') would return None; raise_for_status catches the 404
res.raise_for_status()

Solution 1: Validate Response with raise_for_status() and Guard Table Lookup

Call response.raise_for_status() to catch HTTP errors immediately and check if table is not None: before extracting rows.

Example: Recommended Solution

import requests

# Solution 1: Robust scraping pattern with HTTP status validation
def scrape_table_safely(html_content: str, status_code: int):
    if status_code != 200:
        raise ValueError(f"HTTP Error {status_code}: Unable to fetch webpage.")
    
    # Standalone simulation of table extraction
    if "<table>" in html_content:
        return ["Row 1 Data", "Row 2 Data"]
    return []

# Simulating valid table HTML
valid_html = "<html><body><table><tr><td>Sample Data</td></tr></table></body></html>"
data = scrape_table_safely(valid_html, 200)

print("Extracted rows:", data)
assert len(data) == 2

Solution 2: Graceful Error Handling and Pagination Termination

Wrap requests in a structured try...except requests.exceptions.HTTPError block to terminate pagination gracefully when reaching end-of-catalog 404s.

Example: Alternative Solution

import requests

def fetch_page_records(page_num: int):
    # Simulating pagination boundary check
    if page_num > 3:
        return None  # End of pages reached
    return [{"id": page_num, "value": f"Record_{page_num}"}]

all_records = []
for page in range(1, 10):
    records = fetch_page_records(page)
    if records is None:
        print(f"Pagination completed at page {page}. Total collected: {len(all_records)}")
        break
    all_records.extend(records)

assert len(all_records) == 3

If the webpage returns HTTP 200 OK but soup.find('table') still returns None, inspect the raw HTML with print(response.text[:500]). In most modern websites, the table is rendered dynamically via client-side JavaScript (Fetch/XHR). Open your browser's Developer Tools (Network tab -> Fetch/XHR), find the raw JSON API endpoint used by the frontend, and query that API directly with requests.get() instead of scraping HTML.

Note de reproductibilité : Cette erreur dépend de la disponibilité de la page distante, de la stabilité des routes d'URL du site source et des politiques de redirection du serveur.

Contrast HTTP 404 Not Found with HTTP 403 Forbidden: 404 means the requested path does not exist on the server; 403 means the path exists but access is blocked by firewall, bot detection, or missing authorization.

Previous Next