[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, not concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>

Author: 3 people

Lib/concurrent/futures/process.py

@@ -524,11 +524,6 @@ def _terminate_broken(self, cause):
 
         self.call_queue._terminate_broken()
 
-        # gh-107219: Close the connection writer which can unblock
-        # Queue._feed() if it was stuck in send_bytes().
-        if sys.platform == 'win32':
-            self.call_queue._writer.close()
-
         # clean up resources
         self._join_executor_internals(broken=True)
 

Lib/multiprocessing/queues.py

@@ -164,6 +164,11 @@ def _terminate_broken(self):
         # gh-94777: Prevent queue writing to a pipe which is no longer read.
         self._reader.close()
 
+        # gh-107219: Close the connection writer which can unblock
+        # Queue._feed() if it was stuck in send_bytes().
+        if sys.platform == 'win32':
+            self._writer.close()
+
         self.close()
         self.join_thread()
 

Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b2TrqG.rst

@@ -0,0 +1,6 @@
+On Windows, closing the connection writer when cleaning up a broken
+:class:`multiprocessing.Queue` queue is now done for all queues, rather than
+only in :mod:`concurrent.futures` manager thread.
+This can prevent a deadlock when a ``multiprocessing`` worker process terminates
+without cleaning up.
+This completes the backport of patches by Victor Stinner and Serhiy Storchaka.