gh-136655: ensure cancelled futures are notified on process pool shutdown

[duaneg]

At present when a process pool executor shuts down it is cancelling pending work items, but failing to notify any waiting threads. Fix this.

See also gh-109934, which is a similar bug in the thread pool executor.

• Issue: ProcessPoolExecutor fails to notify when called with shutdown(cancel_futures=True) #136655

[YvesDup]

Your tests failed in MacOSX and Ubuntu, perhaps because there is a barrier that indefinitly waits for - It's strongly possible that a future is cancelled or never starts, and so on never calls the barrier.wait -
I wonder why you didn't choose your example from the problem. That would be simpler.

[duaneg]

Thanks, but I am aware the tests failed. I am currently investigating/debugging, but haven't had a lot of time to spend on it over the last few days. I've managed to reproduce the failure locally now, so will hopefully start to make progress.

I think there is something more subtle than just an indefinite barrier.wait going on with this failure. For a start, none of those waits should be indefinite: they should timeout after 1 second. Also the test only fails with a process pool executor using the "fork" start method, but it is the forkserver manager process that is blocking indefinitely in shutdown. Anyway, investigation is ongoing, so I'll hopefully know more soon.

The proposed unit test is functionally similar to the example from the issue, but attempts to use barriers instead of arbitrary sleep delays so as to work reliably independent of timing. Obviously it isn't there yet, but I'm not a fan of using sleeps to try and hit timing windows, so intend to persevere with this approach for now.

[duaneg]

Another interesting thing about this failure is that it seems to be new: it did not occur in the #134618 PR (the other fix in there "should" not be relevant to this failure). I've confirmed that now on my local system, and have bisected the introduction of the failure to #136796.

Which suggests a fix, and indeed adding a @warnings_helper.ignore_fork_in_thread_deprecation_warnings() decorator seems to have fixed it. I've updated the test in this PR (which should be silencing the warning, of course). I'm not sure why the test was hanging indefinitely rather than failing without the decorator, but the same happens with other tests in this class.