PyThreadState Sometimes Used for Different Threads for Subinterpreters

[ericsnowcurrently]

Bug report

In a few places, we treat PyInterpreterState.threads.head (AKA PyInterpreterState_ThreadHead()) for subinterpreters as an unused thread state that we can use in a one-off manner:

• _PyInterpreterState_IDDecref() (Python/pystate.c)
• _call_in_interpreter() (Python/pystate.c) - fixed in 3.13+ (gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data #109556)
• _run_script_in_interpreter() (Modules/_xxsubinterpreter.c)
• interp_destroy() (Modules/_xxsubinterpreter.c)

The problem is that each thread state should correspond to a single OS thread, and for each OS thread there should be at most one thread state per interpreter. Using PyInterpreterState.threads.head like that violates those assumptions, which can cause problems. ¹

Also, some code assumes PyInterpreterState.threads.head is the current thread state, when it might not be:

• interpreter_clear() (Python/pystate.c)
• ...

We should fix this by always using the thread state that corresponds to the current thread (and create one if missing). For efficiency, we can use a thread-local (via PyThread_tss_*) to store each interpreter's thread state for that thread.

Linked PRs

• gh-109860: Use a New Thread State When Switching Interpreters, When Necessary #110245
• [3.12] gh-109860: Use a New Thread State When Switching Interpreters, When Necessary (gh-110245) #110709

Footnotes

1. Currently, this mostly doesn't affect the main interpreter. Ideally, the main interpreter wouldn't actually be treated specially. See (Mostly) Stop Special-casing the Main Interpreter #109857. ↩