regrtest should re-run failed tests in subprocesses

[vstinner]

Feature or enhancement

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Proposal:

Buildbots and GitHub Actions run the Python test suite with --verbose2 option to re-run failed tests in verbose mode. The current regrtest implementation is the re-run tests in the main process. It is not reliable: if a test crash or timeout, the main process cannot get back the control since the process is killed. Usually, there is no final report after the test is re-run, and the output is just "truncated".

I propose to re-run failed tests in subprocesses to make them more reliable:

• Each test file is run in a fresh process and so does not inherit the state of previously executed tests. It runs in a known and reproducible state: threads, signals, etc.
• The main process can kill the worker process if it hangs.
• If faulthandler kills a worker process on a timeout, the main control detects it and continues its regular work.
• Same if a worker process is killed because of a fatal error: Python Fatal Error, segmentation fault, etc.
• Regular regrtest features can be re-used, like detection of leaking temporary files.

I'm working on an implementation.

Linked PRs

• gh-108834: regrtest reruns failed tests in subprocesses #108839
• gh-108834: Cleanup libregrtest #108858
• gh-108834: regrtest --fail-rerun exits with code 5 #108896
• [3.12] gh-108834: Sync libregrtest with the main branch #108966