gh-134634: Use a pipe instead of a queue for Pool's change notifier#148050
Closed
yonatan-genai wants to merge 3 commits intopython:mainfrom
Closed
gh-134634: Use a pipe instead of a queue for Pool's change notifier#148050yonatan-genai wants to merge 3 commits intopython:mainfrom
yonatan-genai wants to merge 3 commits intopython:mainfrom
Conversation
…fier Pool.__init__ used a multiprocessing.SimpleQueue for its internal _change_notifier. SimpleQueue creates Lock objects backed by POSIX named semaphores (sem_open), which requires /dev/shm. The change notifier never crosses a process boundary, so named semaphores were never needed. Replace it with a _ChangeNotifier class backed by multiprocessing.connection.Pipe (os.pipe). Same interface, no sem_open dependency. <claude>
|
All commit authors signed the Contributor License Agreement. |
The new tests need @warnings_helper.ignore_fork_in_thread_deprecation_warnings() to avoid DeprecationWarning when creating Pool with the fork start method in a multi-threaded test process (Python 3.14+ strictness). <claude>
Author
|
The macOS / build and test (macos-26) failure is unrelated to this change — it's |
Member
|
https://devguide.python.org/getting-started/generative-ai/ I see no evidence of a person being involved here and thus doubt the CLA was signed by a human with legal authority to do so. |
Author
|
My username has been this way for several years… before Claude Code,
OpenClaw, etc. I signed the CLA.
…On Sat, 4 Apr 2026 at 7:23 Gregory P. Smith ***@***.***> wrote:
*gpshead* left a comment (python/cpython#148050)
<#148050 (comment)>
https://devguide.python.org/getting-started/generative-ai/
I see no evidence of a person being involved here and thus doubt the CLA
was signed by a human with legal authority to do so.
—
Reply to this email directly, view it on GitHub
<#148050 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARWBSXZ3SZD4PFGDOAM7UGL4UBBWFAVCNFSM6AAAAACXMA72ZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOBVGU4TANBWGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Author
|
I’d appreciate it if you could consider reopening… |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pool.__init__used amultiprocessing.SimpleQueuefor its internal_change_notifier, a mechanism that wakes up the worker handler thread when the pool state changes (close, terminate, cache empty).SimpleQueuecreates twoLockobjects backed by POSIX named semaphores viasem_open(), which requires/dev/shmto be mounted as tmpfs.The change notifier never crosses a process boundary. It is only accessed by threads within the parent process (the worker handler, result handler, and main thread). Named semaphores were never needed here.
This replaces the
SimpleQueuewith a_ChangeNotifierclass backed bymultiprocessing.connection.Pipe(which wrapsos.pipe()). It provides the same interface (put,get,empty,_reader) without requiringsem_open().Impact
Both
multiprocessing.Poolandmultiprocessing.pool.ThreadPoolfail on platforms without/dev/shm, even thoughThreadPoolonly uses threads and never needs inter-process synchronization. Affected platforms include AWS Lambda (no/dev/shm), Android and iOS (sem_open()unavailable), and Docker containers with restricted shared memory.Changes
Lib/multiprocessing/pool.py: Add_ChangeNotifierclass, replace the one line inPool.__init__that created the SimpleQueueLib/test/_test_multiprocessing.py: Three new tests — notifier type verification + end-to-end pool.map, notification drain under load, worker process recovery after SIGKILLMisc/NEWS.d/next/Library/: NEWS entryTesting
Full
test_multiprocessing_spawnsuite: 447 tests passed, 0 failures. Reference leak check passed.