_socket extension C capsule cannot be visited by the GC and so _socket.socket type may stay alive longer than expected

[vstinner]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]

A clear and concise description of the bug:

The following script will never delete _socket.socket type:

import _socket

# uncomment to workaround the bug
#_socket.CAPI = None

_socket = None

import sys
del sys.modules['_socket']

import gc
gc.collect()

print("exit")

Example on a Python debug build:

$ ./python -X showrefcount script.py 
exit
[408 refs, 260 blocks]

The GC cannot visit the strong reference to the type stored in _socket.CAPI capsule (C API). In Python, creating a heap type creates a reference cycle. For example, a type MRO contains the type. Methods also contain a strong reference to their type.

$ ./python
Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
>>> import _socket
>>> t=_socket.socket
>>> t.__mro__[0] is t
True

See also:

• https://pythondev.readthedocs.io/garbage_collector.html#reference-cycles
• https://vstinner.github.io/debug-python-refleak.html
• https://vstinner.github.io/subinterpreter-leaks.html

Linked PRs

• gh-108240: Fix a reference cycle in _socket.CAPI capsule #108241
• gh-108240: Add _PyCapsule_SetTraverse() internal function #108339
• gh-108240: _PyCapsule_SetTraverse() rejects NULL callbacks #108417
• gh-108240: Add pycore_capsule.h internal header file #108596

[vstinner]

I wrote PR #108241 to fix the issue for the _socket extension: avoid strong references in the capsule object.

[vstinner]

_curses extension is affected by the same bug: _curses._C_API capsule stores a strong reference to _curses.window heap type.

Extensions not affected by the issue, using borrowed references:

• _datetime.datetime_CAPI stores borrowed references to _datetime heap types
• pyexpat.expat_CAPI does not store types
• unicodedata._ucnhash_CAPI does not store types
• _testcapi capsules don't store types

[vstinner]

By the way, commit 46366ca may have introduced a leak: see PR #107577.

[vstinner]

I closed issue #107789 as a duplicate of this issue.

[vstinner]

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

[Eclips4]

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

Probably related: #107994

[vstinner]

Probably related: #107994

Oh right, the _decimal extension has no capsule, so the refleak is unrelated to this issue.

[vstinner]

cc @erlend-aasland @corona10 @pablogsal: yet another complex GC issue involving extension multi-phase initialization and the garbage collector.

[vstinner]

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

[Eclips4]

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

You mean leak in the _curses is a known issue?

import importlib
import sys

def foo():
    name = "_curses"
    importlib.import_module(name)
    del sys.modules[name]


for _ in range(4):
    foo()

./python.exe -X showrefcount example.py
[1006 refs, 491 blocks]

[vstinner]

You mean leak in the _curses is a known issue?

Yes. It should be ported to multi-phase init and its static types should be converted to heap types.