Unhandled asyncio.CancelledError in Python Background Tasks

Verified FixPython 3.10+Python Native AsyncIO 3.11+Silo: asyncio

Quick Fix / Solution Rapide

Intercept asyncio.CancelledError in background workers for cleanup, or use asyncio.TaskGroup for structured concurrency in Python 3.11+.

Root Cause Analysis

This error occurs when an asyncio.Task is cancelled via task.cancel() or an enclosing timeout expires, and the coroutine propagates asyncio.CancelledError without handling cleanup or re-raising it improperly.

1. How Task Cancellation Works in Python

When task.cancel() is called on an active asyncio Task, the event loop injects an asyncio.CancelledError exception into the coroutine at its current await suspension point. If the task does not catch the exception, it terminates immediately.

2. In Python 3.8+, CancelledError Inherits from BaseException

In modern Python (3.8+), asyncio.CancelledError inherits from BaseException rather than Exception. Standard except Exception: blocks do NOT catch cancellation, allowing CancelledError to propagate unhindered.

3. Suppressing CancelledError Improperly

Catching CancelledError and continuing execution without re-raising or exiting prevents orchestrators from stopping tasks gracefully.

4. Background Tasks Fire-and-Forget Garbage Collection

Creating tasks via asyncio.create_task() without retaining a reference causes the Python garbage collector to destroy the task mid-execution.

Reproduction Code (MCVE)

Example: Bug Reproduction

import asyncio

async def background_worker():
    await asyncio.sleep(10)

async def main():
    task = asyncio.create_task(background_worker())
    await asyncio.sleep(0.01)
    task.cancel()
    # Awaiting cancelled task raises CancelledError
    await task

asyncio.run(main())

Solution 1: Handle Cancellation Cleanly with try...except CancelledError

Catch asyncio.CancelledError to perform cleanup (closing files, releasing locks), then re-raise or exit.

Example: Recommended Solution

import asyncio

async def robust_worker():
    try:
        print('Worker running...')
        await asyncio.sleep(10)
    except asyncio.CancelledError:
        print('Worker received cancellation: cleaning up resources.')
        raise  # Re-raise to signal clean cancellation

async def main():
    task = asyncio.create_task(robust_worker())
    await asyncio.sleep(0.01)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print('Task cancelled cleanly.')

asyncio.run(main())

Solution 2: Use asyncio.TaskGroup for Structured Concurrency (Python 3.11+)

Use asyncio.TaskGroup to automatically manage task lifecycles, cancellation propagation, and error grouping.

Example: Alternative Solution

import asyncio

async def worker(n):
    await asyncio.sleep(0.01)
    return f'Worker {n} complete'

async def main():
    async with asyncio.TaskGroup() as tg:
        t1 = tg.create_task(worker(1))
        t2 = tg.create_task(worker(2))
    print(f'Results: {t1.result()}, {t2.result()}')

asyncio.run(main())

A common mistake is storing running tasks in local variables that go out of scope. Always store background tasks in a global set (background_tasks.add(task); task.add_done_callback(background_tasks.discard)) to prevent premature garbage collection. Edge cases occur during FastAPI shutdown: FastAPI cancels background tasks during lifespan exit. Ensure cleanup code does not call blocking time.sleep(). Contrast this error with TimeoutError, which is raised when a deadline expires rather than an explicit .cancel() call.

Previous Next