ProcessPoolExecutor executor manager thread crashes if task fails in some circumstances

[batterseapower]

Bug report

Bug description:

import time
from concurrent.futures import ProcessPoolExecutor

def fail():
    time.sleep(1)
    raise ValueError()

executor = ProcessPoolExecutor(max_workers=1, max_tasks_per_child=1)
try:
    print(executor.submit(fail).exception() is not None)
finally:
    executor.shutdown(wait=False)

If you run this, then the following is printed to the console:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "python3.11/concurrent/futures/process.py", line 364, in run
    executor._adjust_process_count()
  File "python3.11/concurrent/futures/process.py", line 759, in _adjust_process_count
    process_count = len(self._processes)
                    ^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()

If you change the code to using wait=True, or remove the max_tasks_per_child=1 option then the message is not printed.

Probably the fix is just to check if self.processes is None before accessing it at https://github.com/python/cpython/blob/main/Lib/concurrent/futures/process.py#L762

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits #152978

[gpshead]

Status update after the related gh-115634 / GH-140900: The reproducer above no longer triggers on main.
Two changes account for that:

1. ProcessPoolExecutor raises exception or hangs the process when shutdown(wait=False) and worker throws exception #132969 (gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222) added an early return to the spawn path when self._processes is None, which covers the common orderings of this race.
2. gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child #140900 moved worker replacement into _replace_dead_worker(), which returns before touching self._processes when the executor is shutting down with no pending work -- exactly the state this reproducer creates. _shutdown_thread is read under _shutdown_lock, which shutdown() holds while setting it, so that check is not itself racy.

But... the underlying race is narrowed rather than closed so I'm keeping this open:

shutdown() sets _shutdown_thread under _shutdown_lock, but the reference-clearing block afterwards (self._processes = None and friends) runs outside the lock. With wait=False the manager thread is still alive, so a worker exit racing that block can still, in principle, observe self._processes as a dict at the None-check and as None at a later use.
In _replace_dead_worker() when pending work remains, and in the submit()-path _adjust_process_count()/_spawn_process() pair, which predates all of this. An unhandled TypeError there kills the manager thread and strands any pending futures.

The likely ideal fix is lock discipline in shutdown(): move the clearing of manager-thread-visible state (at minimum self._processes) under _shutdown_lock, or stop clearing self._processes entirely -- unlike the queues it holds no file descriptors, and the fd-reduction comment that motivates the block does not apply to it.

(Claude Fable 5 drafted most of that - i'd word it a bit differently myself but it's appropriate input to actually fix the remaining bits... -gpshead)