Scraping/Playwright: Resolving TimeoutError and Exporting Clean CSV Datasets

Verified FixPython 3.10+Playwright 1.42+Silo: scraping

Quick Fix / Solution Rapide

This error occurs when Python waits for a Playwright selector that never renders within the timeout period. Use page.wait_for_selector(..., timeout=10000) with try/except and write records safely to CSV with csv.DictWriter.

Root Cause Analysis

This error occurs when Python scripts using Microsoft Playwright for asynchronous web scraping attempt to locate dynamic DOM elements (page.wait_for_selector or page.locator.click()), but the target webpage delays loading, encounters bot protection, or changes DOM selectors, exceeding the default 30,000ms timeout.

Root Cause 1: Dynamic Client-Side Rendering Delays

Modern Single Page Applications (SPAs) load content via asynchronous API calls. If network latency is high or the server throttles requests, Playwright's default selector wait expires, raising playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded.

Root Cause 2: Flawed Selector Logic on Variable Layouts

Using rigid full XPath selectors (e.g. /html/body/div[2]/div[1]/...) breaks when advertisements or dynamic banners appear, preventing Playwright from finding the element.

Root Cause 3: Unhandled Rate Limiting / Cloudflare Challenges

When target websites detect automated Playwright browser instances, they intercept the navigation with Cloudflare Turnstile or Captcha pages. The scraper continues waiting for product selectors that never load.

Root Cause 4: CSV Serialization Data Loss on Process Crash

Scraping hundreds of pages and saving records only at the very end causes total data loss if a single TimeoutError crashes the script midway.

Reproduction Code (MCVE)

Example: Bug Reproduction

# Simulating Playwright element selector timeout when target element is not loaded
class MockPlaywrightPage:
    def wait_for_selector(self, selector: str, timeout_ms: int = 5000):
        # Simulating selector timeout failure
        if selector == "#dynamic-listing-table":
            raise TimeoutError(
                f"TimeoutError: Timeout {timeout_ms}ms exceeded while waiting for selector '{selector}'. "
                "The target element failed to render within the allotted time."
            )

page = MockPlaywrightPage()
page.wait_for_selector("#dynamic-listing-table", timeout_ms=3000)

Solution 1: Use Safe Selector Waits with Fallback and Batch CSV Writing

Wrap selector lookups in try...except TimeoutError and stream extracted items immediately to CSV using csv.DictWriter.

Example: Recommended Solution

import csv
import io

# Solution 1: Incremental CSV writing pattern
records = [
    {"id": "ITEM-101", "name": "Mechanical Keyboard", "price": 89.99},
    {"id": "ITEM-102", "name": "Wireless Mouse", "price": 49.50}
]

output_buffer = io.StringIO()
fieldnames = ["id", "name", "price"]
writer = csv.DictWriter(output_buffer, fieldnames=fieldnames)
writer.writeheader()

for item in records:
    writer.writerow(item)

csv_output = output_buffer.getvalue()
print("Exported CSV content:")
print(csv_output)
assert "Mechanical Keyboard" in csv_output

Solution 2: Configure Realistic Browser Context and Stealth Options

Configure human-like viewport dimensions, user-agent, and locale in Playwright browser contexts.

Example: Alternative Solution

# Solution 2: Playwright context options simulation
browser_context_options = {
    "viewport": {"width": 1920, "height": 1080},
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "locale": "en-US",
    "timezone_id": "America/New_York"
}

print("Configured Playwright context parameters:")
for k, v in browser_context_options.items():
    print(f"  {k}: {v}")

assert browser_context_options["viewport"]["width"] == 1920

Never use time.sleep() in Playwright scripts. Use page.wait_for_load_state('networkidle') or page.locator('selector').wait_for(state='visible') to await elements reactively without wasting execution time.

Contrast Playwright with Selenium: Playwright has native auto-waiting (waits for elements to be actionable before clicking); Selenium requires explicit WebDriverWait.

Previous Next