Free-threaded CPython can starve a thread reattaching during repeated stop-the-world GC

Bug report

While hardening free-threaded Python support in numba-cuda, we found a workload that could wedge when a stress worker repeatedly called gc.collect() in a tight loop while other threads were doing CUDA dispatch work. Removing only the tight manual-GC worker made the workload pass.

I reduced this to a no-third-party CPython reproducer. The reduced case is strongest on a debug free-threaded CPython main build, where the extra free-threaded GC validation work makes the stop-the-world window long enough to reproduce reliably.

import faulthandler
import gc
import importlib
import os
import sys
import threading
import time

faulthandler.dump_traceback_later(25, exit=True)

modules = (
    "abc", "argparse", "collections", "contextlib",
    "decimal", "enum", "functools", "heapq",
    "importlib", "inspect", "itertools", "json",
    "math", "operator", "random", "re",
    "statistics", "threading", "types", "weakref",
)
for name in modules:
    importlib.import_module(name)

print("python", sys.version, flush=True)
print("abiflags", getattr(sys, "abiflags", ""), flush=True)
print("gil", getattr(sys, "_is_gil_enabled", lambda: None)(), flush=True)
print("debug", hasattr(sys, "gettotalrefcount"), flush=True)
print("pid", os.getpid(), flush=True)

stop = threading.Event()
count = 0

def collect():
    global count
    while not stop.is_set():
        gc.collect()
        count += 1

thread = threading.Thread(target=collect, name="gc-collector")
started = time.monotonic()
thread.start()

time.sleep(5.0)
stop.set()
thread.join(10.0)

elapsed = time.monotonic() - started
print(f"alive={thread.is_alive()} count={count} elapsed={elapsed:.6f}",
      flush=True)

raise SystemExit(1 if thread.is_alive() else 0)

Build used for the reproducer:

./configure --with-pydebug --disable-gil --without-ensurepip
make -j

Observed on CPython main debug/free-threaded:

Python 3.16.0a0 free-threading build
abiflags td
GIL disabled
debug True

I reproduced this on a B200 Linux system with 224 CPUs at the host level. I have not yet reproduced it on a smaller local workstation. The test process was running under a Slurm allocation that reported 28 CPUs available to the process and 224 CPUs on the system.

The script regularly times out via the faulthandler watchdog. The Python-level traceback shows the GC worker in gc.collect() and the main thread apparently still at time.sleep(). A native stack shows the more precise state:

gc-collector:
  validate_refcounts() / validate_gc_objects()
  gc_visit_heaps()
  deduce_unreachable_heap()
  gc_collect_internal()
  _PyGC_Collect()
  gc.collect()

main thread:
  _PyParkingLot_Park()
  tstate_wait_attach()
  _PyThreadState_Attach()
  PyEval_RestoreThread()
  pysleep()
  time.sleep()

At the time of the stall, GDB showed the interpreter stop-the-world state as:

requested = true
world_stopped = true
thread_countdown = 0
requester = GC worker thread state
main thread state = _Py_THREAD_SUSPENDED

The suspected issue is that after start_the_world() moves suspended threads back to detached and unparks them, a thread running a tight gc.collect() loop can immediately request another stop-the-world pause and re-suspend a just-unparked thread before that thread gets to attach. The result is starvation of a thread trying to return from a detached operation such as time.sleep().

This was not reproduced in my minimized matrix on:

a non-debug free-threaded CPython main build
conda-forge Python 3.14.6t

However, the original larger stress workload was first encountered while testing Python 3.14t free-threaded package support, and the reduced CPython main debug build gives a clean way to expose the progress bug.

I have a local CPython patch that tracks thread states waiting to attach after a stop-the-world suspension and prevents a subsequent stop-the-world requester from immediately re-suspending those attach waiters. With that patch:

the minimized reproducer passes
the broader pure-Python GC stress matrix passes
test_free_threading.test_gc passes under debug and non-debug free-threaded builds
test_free_threading.test_gc test_threading -v passes under the debug free-threaded build

I will prepare a PR with the fix and regression test once this issue exists.

CPython versions tested on:

3.14
3.16
CPython main branch

Operating systems tested on:

Linux

Linked PRs

Hey, this seems like an interesting example of a thread starvation problem.

I take some time to dive into this problem, and based on my understanding, the root cause is a race in start_the_world / park_detached_threads: a thread woken from SUSPENDED → DETACHED can be re-parked before it gets a chance to re-attach, because the next gc.collect() call fires immediately, and park_detached_threads sees it as plain DETACHED.

I have some personal suggestions on this problem fix, maybe introduce DETACHED_WAKING (4) as a transient state:

start_the_world transitions SUSPENDED → DETACHED_WAKING instead of DETACHED.
park_detached_threads sees DETACHED_WAKING → sets _PY_EVAL_PLEASE_STOP_BIT on the eval breaker without parking. The thread gets a chance to attach first, then suspends cooperatively via _Py_HandlePending().
tstate_try_attach now accepts DETACHED_WAKING → ATTACHED in addition to DETACHED → ATTACHED, so Py_END_ALLOW_THREADS succeeds immediately after a wakeup.

Side: Based on the description, this issue seems to occur only on HPC systems. I haven’t had a chance to compile it on an HPC system yet, but I haven’t encountered this problem on the M chip. I’ll try to reproduce the issue on a machine with a 128-core CPU and an RTX-6000, and I’ll report back to you later.

Thanks — yes, that is the release/re-park race I am seeing. I considered
representing the wakeup entirely in the thread-state value.

The complication with mapping every resumed SUSPENDED tstate to
DETACHED_WAKING is that some tstates remain detached because their OS thread
is still blocked in sleep or I/O and is not yet trying to attach. If the next
STW refuses to park one of those tstates, it can wait until that operation
returns; an eval-breaker bit cannot run while the thread remains detached.

PR #152826 instead distinguishes tstates parked from DETACHED, then marks a
waiter only when tstate_wait_attach() actually observes that state. A later
STW skips only active attach waiters. Passive detached tstates remain
parkable, and the normal uncontended attach path remains the existing CAS. A
state-only representation of an active waiter could also work, but it would
need another state and transition rather than marking every woken tstate.

@colesbury

@colesbury

pythongh-151518: Avoid STW starvation of attaching threads

pythongh-151518: Track attach waiters in the thread state

pythongh-151518: Simplify the GC fairness regression

pythongh-151518: Exercise STW fairness directly

pythongh-151518: Avoid STW starvation of attaching threads

pythongh-151518: Track attach waiters in the thread state

pythongh-151518: Simplify the GC fairness regression

pythongh-151518: Exercise STW fairness directly

pythongh-151518: Preserve attach waiters when resuming BRC suspensions

pythongh-151518: Strengthen the STW attach fairness regression

pythongh-151518: Restrict STW test pauses to free-threaded builds

pythongh-151518: Wake only registered attach waiters

gh-151518: Avoid STW starvation of attaching threads (#152826)

[3.14] pythongh-151518: Avoid STW starvation of attaching threads

Navigation Menu

Repository navigation

Free-threaded CPython can starve a thread reattaching during repeated stop-the-world GC #151518

Bug report

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Free-threaded CPython can starve a thread reattaching during repeated stop-the-world GC #151518

Description

Bug report

Linked PRs

Activity

zangjiucheng commented on Jun 22, 2026

tpn commented on Jul 1, 2026

nascheme commented on Jul 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions