3.15 profiling, can't use ProcessPoolExecutor.map() in a simple script

[dimaqq]

Bug report

Bug description:

# test/test_unit.py
import concurrent.futures
...

def exercise(units=11, rounds=2):
    with concurrent.futures.ProcessPoolExecutor() as executor:
        ...
            results = list(
                executor.map(
                    step,
                    [f"app/{i}" for i in range(units)],
                    [config] * units,
                    [local_app_data] * units,
                    [peers_data] * units,
                )
            )


def step(
    unit: str,
    config: Mapping[str, str | int | float | bool],
    local_app_data: Mapping[str, JSON],
    all_units_data: Mapping[ops.testing.UnitID, Mapping[str, JSON]],
) -> tuple[dict[str, JSON] | None, dict[str, JSON], str | None, str]:
    ...

if __name__ == "__main__":
    import sys

    exercise(int(sys.argv[1]), int(sys.argv[2]))

(jgol-peer) 🦐/c/j/peer (main)> sudo -E env PYTHONPATH=src python -m profiling.sampling test/test_unit.py 100 100
Profiler coordinator error: Error executing script '/code/jgol/peer/test/test_unit.py': Can't pickle <function step at 0x102b7aae0>: it's not found as __main__.step

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

• gh-140729: Add __mp_main__ as a duplicate for __main__ for pickle to work #140735
• gh-140729: Fix subprocess handling in test_process_pool_executor_pickle #141688
• gh-140729: profiling or tracing multiprocessing can cause error #141874
• gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process #144700
• gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process #144715
• [3.15] gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process (GH-144715) #153243
• [3.14] gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process (GH-144715) #153265

[picnixz]

cc @pablogsal

[yihong0618]

This is interesting
its very similar like viztracer issue which is gaogaotiantian/viztracer#423
in ProcessPoolExecutor() the __main__ may because __mp_main__

a simple reproducer

cat ttt.py

import concurrent.futures


def worker(x):
    return x * 2


if __name__ == "__main__":
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = list(executor.map(worker, [1, 2, 3]))
        print(f"Results: {results}")

➜  cpython git:(main) ✗ sudo ./python.exe -m profiling.sampling ttt.py 100 100
Profiler coordinator error: Error executing script '/Users/hyi/prs/cpython/ttt.py': Can't pickle <function worker at 0x109a2d170>: it's not found as __main__.worker
Captured 1000 samples in 0.10 seconds
Sample rate: 10000.00 samples/sec
Error rate: 0.60%
Profile Stats:
       nsamples   sample%  tottime (ms)    cumul%  cumtime (ms)  filename:lineno(function)
          0/606       0.0         0.000      67.3        60.600  _base.py:694(Executor.__exit__)
        604/604      67.0        60.400      67.0        60.400  threading.py:1126(Thread.join)
          0/604       0.0         0.000      67.0        60.400  process.py:865(ProcessPoolExecutor.shutdown)
          0/143       0.0         0.000      15.9        14.300  <frozen importlib._bootstrap>:1379(_find_and_load)
          0/137       0.0         0.000      15.2        13.700  <frozen importlib._bootstrap>:1350(_find_and_load_unlocked)
           0/78       0.0         0.000       8.7         7.800  <frozen importlib._bootstrap>:938(_load_unlocked)
           0/77       0.0         0.000       8.5         7.700  <frozen importlib._bootstrap_external>:759(_LoaderBasics.exec_module)
           0/59       0.0         0.000       6.5         5.900  <frozen importlib._bootstrap>:924(_load_unlocked)
           1/58       0.1         0.100       6.4         5.800  <frozen importlib._bootstrap_external>:855(SourceLoader.get_code)
          58/58       6.4         5.800       6.4         5.800  <frozen importlib._bootstrap>:491(_call_with_frames_removed)
           0/57       0.0         0.000       6.3         5.700  <frozen importlib._bootstrap_external>:1057(ExtensionFileLoader.create_module)
           0/57       0.0         0.000       6.3         5.700  <frozen importlib._bootstrap>:816(module_from_spec)
          50/50       5.5         5.000       5.5         5.000  <frozen importlib._bootstrap_external>:955(FileLoader.get_data)
          31/31       3.4         3.100       3.4         3.100  resource_tracker.py:114(ResourceTracker._stop_locked)
           0/31       0.0         0.000       3.4         3.100  resource_tracker.py:89(ResourceTracker._stop)

Legend:
  nsamples: Direct/Cumulative samples (direct executing / on call stack)
  sample%: Percentage of total samples this function was directly executing
  tottime: Estimated total time spent directly in this function
  cumul%: Percentage of total samples when this function was on the call stack
  cumtime: Estimated cumulative time (including time in called functions)
  filename:lineno(function): Function location and name

Summary of Interesting Functions:

Functions with Highest Direct/Cumulative Ratio (Hot Spots):
  1.000 direct/cumulative ratio, 67.0% direct samples: threading.py:(Thread.join)
  1.000 direct/cumulative ratio, 6.4% direct samples: <frozen importlib._bootstrap>:(_call_with_frames_removed)
  1.000 direct/cumulative ratio, 5.5% direct samples: <frozen importlib._bootstrap_external>:(FileLoader.get_data)

Functions with Highest Call Frequency (Indirect Calls):
  606 indirect calls, 67.3% total stack presence: _base.py:(Executor.__exit__)
  604 indirect calls, 67.0% total stack presence: process.py:(ProcessPoolExecutor.shutdown)
  143 indirect calls, 15.9% total stack presence: <frozen importlib._bootstrap>:(_find_and_load)

Functions with Highest Call Magnification (Cumulative/Direct):
  58.0x call magnification, 57 indirect calls from 1 direct: <frozen importlib._bootstrap_external>:(SourceLoader.get_code)

the patch can be like what viztracer did

diff --git a/Lib/profiling/sampling/_sync_coordinator.py b/Lib/profiling/sampling/_sync_coordinator.py
index 79e8858ca1..43a80b6adb 100644
--- a/Lib/profiling/sampling/_sync_coordinator.py
+++ b/Lib/profiling/sampling/_sync_coordinator.py
@@ -177,8 +177,14 @@ def _execute_script(script_path: str, script_args: List[str], cwd: str) -> None:
             source_code = f.read()
 
         # Compile and execute the script
+        import types
+        main_module = types.ModuleType("__main__")
+        main_module.__file__ = script_path
+        main_module.__builtins__ = __builtins__
+        sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module
+
         code = compile(source_code, script_path, 'exec')
-        exec(code, {'__name__': '__main__', '__file__': script_path})
+        exec(code, main_module.__dict__)
     except FileNotFoundError as e:
         raise TargetError(f"Script file not found: {script_path}") from e
     except PermissionError as e:

after the patch
cc @dimaqq

[mehetabalom181-lab]

Bug report

Bug description:

test/test_unit.py

import concurrent.futures
...

def exercise(units=11, rounds=2):
with concurrent.futures.ProcessPoolExecutor() as executor:
...
results = list(
executor.map(
step,
[f"app/{i}" for i in range(units)],
[config] * units,
[local_app_data] * units,
[peers_data] * units,
)
)

def step(
unit: @str,
config: Mapping[str, str | int | float | bool],
local_app_data: Mapping[str, JSON],
all_units_data: Mapping[ops.testing.UnitID, Mapping[str, JSON]],
) -> tuple[dict[str, JSON] | None, dict[str, JSON], str | None, str]:
...

if name == "main":
import sys

exercise(int(sys.argv[1]), int(sys.argv[2]))

(jgol-peer) 🦐/c/j/peer (main)> sudo -E env PYTHONPATH=src python -m profiling.sampling test/test_unit.py 100 100
Profiler coordinator error: Error executing script '/code/jgol/peer/test/test_unit.py': Can't pickle <function step at 0x102b7aae0>: it's not found as main.step

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

• gh-140729: Add mp_main as a duplicate for main for pickle to work #140735

[YvesDup]

The execution of ttt.py script fails when run via the cProfile module as below:

python -m cProfile /Users/yves/Desktop/Bugs/gh-140729.py
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/queues.py", line 115, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__main__' has no attribute 'worker'

and with the profile module too.

Based on my initial tests, the proposed fix is not valid for these two modules. Should we open a new issue ?

[yihong0618]

The execution of ttt.py script fails when run via the cProfile module as below:

python -m cProfile /Users/yves/Desktop/Bugs/gh-140729.py
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/queues.py", line 115, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__main__' has no attribute 'worker'

and with the profile module too.

Based on my initial tests, the proposed fix is not valid for these two modules. Should we open a new issue ?

using the same method can fix this?

[YvesDup]

using the same method can fix this?

I don`t know yet

[YvesDup]

@picnixz, @StanFromIreland I am wondering if a topic-profiling (if exists) or a topic-multiprocessing label should not be better than topic-subinterpreters.

[StanFromIreland]

I added topic-multiprocessing, FYI all labels are listed here. topic-profiling does not exist.