Scraping: ModuleNotFoundError: No module named bs4 Despite Appearing in pip list

Verified FixPython 3.10+BeautifulSoup4 4.12+Silo: scraping

Quick Fix / Solution Rapide

This error occurs when Python executes a script using an interpreter different from the one where beautifulsoup4 was installed. Install packages using python -m pip install beautifulsoup4 to ensure the active Python interpreter receives the package.

Root Cause Analysis

This error occurs when Python executes a web scraping script and encounters import bs4 or from bs4 import BeautifulSoup, but the running Python interpreter cannot locate the package despite pip list displaying beautifulsoup4 on the terminal.

Root Cause 1: Disconnected pip and python Executable Binaries

In modern operating systems, multiple Python versions and virtual environments frequently coexist (e.g. system Python, Homebrew Python, Anaconda, pyenv, VS Code virtualenv). Running a global pip list or pip install beautifulsoup4 executes the first pip found in the system PATH, which may belong to Python 3.10, while the terminal command python script.py or the IDE runner executes a different Python 3.12 binary that does not have bs4 installed in its site-packages.

Root Cause 2: Package Name vs Import Module Name Confusion

The PyPI distribution package name is beautifulsoup4, whereas the Python import statement uses import bs4. Running pip install bs4 installs a legacy dummy package or fails on certain registries, while running pip install beautifulsoup4 properly installs the bs4 module.

Root Cause 3: Misconfigured IDE Interpreter Selection

In IDEs like VS Code or PyCharm, the selected Python interpreter in the status bar may point to a different virtual environment than the terminal shell where the developer ran pip install.

Root Cause 4: Inactive Virtual Environments

Executing scripts after opening a new terminal session without activating the virtual environment (source venv/bin/activate or venv\Scripts\activate) forces the script to run against system Python where project dependencies are absent.

Reproduction Code (MCVE)

Example: Bug Reproduction

import sys

# Simulating interpreter execution where bs4 is not in the active sys.path
def check_bs4_import():
    if "bs4" not in sys.modules and not any("bs4" in p for p in sys.path):
        raise ModuleNotFoundError(
            "ModuleNotFoundError: No module named 'bs4'. "
            f"Executed with Python binary: {sys.executable}. "
            "Ensure you install via 'python -m pip install beautifulsoup4' for this interpreter."
        )

check_bs4_import()

Solution 1: Install with Explicit python -m pip and Verify Interpreter

Always run python -m pip install beautifulsoup4 using the exact python executable that runs your script to guarantee proper site-packages installation.

Example: Recommended Solution

import sys
import os

# Solution 1: Verify current interpreter and site-packages path
print("Active Python Executable:", sys.executable)
print("Python Version:", sys.version.split()[0])

# To install BeautifulSoup into THIS exact interpreter, run:
# f"{sys.executable} -m pip install beautifulsoup4"
print("Command to run in terminal:")
print(f'"{sys.executable}" -m pip install beautifulsoup4')

assert os.path.exists(sys.executable)

Solution 2: Ensure Virtual Environment is Properly Activated

Create a clean project virtual environment and verify that sys.prefix != sys.base_prefix indicating an active virtualenv.

Example: Alternative Solution

import sys

# Solution 2: Check if running inside an active virtual environment
is_virtualenv = sys.prefix != getattr(sys, "base_prefix", sys.prefix)

print("Running inside virtual environment:", is_virtualenv)
print("Prefix path:", sys.prefix)

# Standalone simulation demonstrating proper import structure
sample_html = "<html><body><h1>Scraping Guide</h1></body></html>"
print("HTML ready for parser processing:", len(sample_html), "bytes")

Always check the Python interpreter path in VS Code by opening the Command Palette (Ctrl+Shift+P / Cmd+Shift+P) and selecting Python: Select Interpreter. Ensure the selected interpreter path matches the path of your active .venv.

Note de reproductibilité : Cette erreur d'environnement dépend de la configuration du système d'exploitation, des variables d'environnement PATH, de l'état d'activation des environnements virtuels (venv, conda) et de la sélection de l'interpréteur dans l'IDE.

Contrast beautifulsoup4 with bs4: beautifulsoup4 is the official package name on PyPI, while bs4 is the internal Python package name imported in code (from bs4 import BeautifulSoup).

Previous Next