Our internal headers are far too interlinked

[markshannon]

Adding a small feature like #131198 (comment) becomes a complicated puzzle of moving functions and structs between header files to avoid cycles.
I was ultimately defeated in this case.

This is a bit silly. We should break up these dependencies, by breaking headers into structs and code.
Code depends on data, and headers that depend on some structs, but not the code, can import only the struct definitions. Breaking headers into smaller, more self contained chunks would also help.

As an example of the problem, to use _PyThreadState_GET() one must import pycore_pystate.h but that imports pycore_runtime.h which imports nineteen other "core" header files!

Linked PRs

• GH-131238: Core header refactor #131250
• gh-131238: Add explicit include to pycore headers #131257
• GH-131238: More refactoring of core header files #131351
• gh-131238: Convert pycore_pystate.h static inline to functions #131352
• gh-131238: Remove pycore_runtime.h from pycore_pystate.h #131356
• gh-131238: Remove many includes in pycore_interp.h #131472
• gh-131238: Remove more includes from pycore_interp.h #131480
• gh-131238: Use pycore_interp_structs.h header #131481
• gh-131238: Move pycore_obmalloc.h include to pycore_runtime_structs.h #131482
• gh-131238: Remove pycore_lock.h includes #131483
• gh-131238: Cleanup pycore_runtime.h includes #131486
• gh-131238: Remove includes from pycore_interp.h #131495
• gh-131238: Add pycore_interpframe.h to PYTHON_HEADERS #131545
• gh-131238: Remove pycore_object_deferred.h from pycore_object.h #131549
• gh-131238: Add pycore_interpframe_structs.h header #131553
• gh-131238: Move _Py_VISIT_STACKREF() to pycore_stackref.h #131560
• gh-131238: Add missing pycore_function.h includes for JIT compiler #131571

[vstinner]

One solution to reduce dependencies would be to convert some static inline and macros to regular functions to move the implementation details from header files to the C code (.c files).

As an example of the problem, to use _PyThreadState_GET() one must import pycore_pystate.h but that imports pycore_runtime.h which imports nineteen other "core" header files!

In Python 3.11, _PyThreadState_GET() uses the runtime API:

static inline PyThreadState*
_PyThreadState_GET(void)
{
    return _PyRuntimeState_GetThreadState(&_PyRuntime);
}

In Python 3.12, it's no longer the case:

static inline PyThreadState*
_PyThreadState_GET(void)
{
#if defined(HAVE_THREAD_LOCAL) && !defined(Py_BUILD_CORE_MODULE)
    return _Py_tss_tstate;
#else
    return _PyThreadState_GetCurrent();
#endif
}

But there is still code which access the PyRuntimeState API:

static inline int
_Py_IsMainInterpreterFinalizing(PyInterpreterState *interp)
{
    return (_PyRuntimeState_GetFinalizing(&_PyRuntime) != NULL &&
            interp == &_PyRuntime._main_interpreter);
}

#define HEAD_LOCK(runtime) \
    PyThread_acquire_lock((runtime)->interpreters.mutex, WAIT_LOCK)
#define HEAD_UNLOCK(runtime) \
    PyThread_release_lock((runtime)->interpreters.mutex)

[markshannon]

The problem is that the structs and functions are mixed together, which means that any function that refers to a struct in another file introduces cycles. Breaking the headers in struct and function pairs breaks most of the dependencies

[vstinner]

Another problem is that the _PyRuntimeState structure is a giant beast. I never understood the rationale for putting ALL global variables in a single place. For example, _Py_ID() requires to get access to the whole _PyRuntimeState structure since singletons are stored there. It means pulling all includes to build this structure. But I don't know if it would make sense to move back globals were they were before to split again _PyRuntimeState.