Parallel parsing sometimes crashes when compiled in free-threaded Python

[ilevkivskyi]

I don't have a way to reproduce this, but just noticed this random crash in testOutputJsonParallel:

Current thread's C stack trace (most recent call first):
  Binary file "/opt/hostedtoolcache/Python/3.14.5/arm64-freethreaded/lib/libpython3.14t.so.1.0", at _Py_DumpStack+0x4c [0xff4fb2f871ac]
  Binary file "/opt/hostedtoolcache/Python/3.14.5/arm64-freethreaded/lib/libpython3.14t.so.1.0", at +0xed7ac [0xff4fb2f8d7ac]
  Binary file "linux-vdso.so.1", at __kernel_rt_sigreturn+0x0 [0xff4fb3536938]
  Binary file "/lib/aarch64-linux-gnu/libc.so.6", at +0x87608 [0xff4fb2d67608]
  Binary file "/lib/aarch64-linux-gnu/libc.so.6", at gsignal+0x1c [0xff4fb2d1cb3c]
  Binary file "/lib/aarch64-linux-gnu/libc.so.6", at abort+0xf4 [0xff4fb2d07e00]
  Binary file "/lib/aarch64-linux-gnu/libc.so.6", at +0x35cc0 [0xff4fb2d15cc0]
  Binary file "/lib/aarch64-linux-gnu/libc.so.6", at __assert_perror_fail+0x0 [0xff4fb2d15d30]
  Binary file "/home/runner/work/mypy/mypy/fb62009ad6f7d3457273__mypyc.cpython-314t-aarch64-linux-gnu.so", at +0x2bde554 [0xff4fab4ee554]
  Binary file "/home/runner/work/mypy/mypy/fb62009ad6f7d3457273__mypyc.cpython-314t-aarch64-linux-gnu.so", at +0x6118dc [0xff4fa8f218dc]
  Binary file "/home/runner/work/mypy/mypy/fb62009ad6f7d3457273__mypyc.cpython-314t-aarch64-linux-gnu.so", at CPyArg_ParseStackAndKeywordsSimple+0x1b0 [0xff4fa8f22e9c]
  Binary file "/home/runner/work/mypy/mypy/fb62009ad6f7d3457273__mypyc.cpython-314t-aarch64-linux-gnu.so", at CPyPy_mypy___build___State___parse_file_inner+0x48 [0xff4fa90d8f98]
  Binary file "/opt/hostedtoolcache/Python/3.14.5/arm64-freethreaded/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x48c4 [0xff4fb30620a4]

It looks like some C-level assertion failure (because of a race condition?) FWIW I have never seen this issue in regular (non-free-threaded) Python.

cc @JukkaL (as you have been interested in making native parser more stable)

[JukkaL]

I encountered similar-looking test failure on 3.14 non-free-threaded build (Linux), using interpreted mypy:

____________________________ testOutputJsonParallel ____________________________
[gw7] linux -- Python 3.14.3 /home/jukka/venv/mypy/bin/python
data: /home/jukka/src/remote-mypy/test-data/unit/outputjson.test:46:
/home/jukka/src/remote-mypy/mypy/test/testoutput.py:23: in run_case
    test_output_json(testcase)
/home/jukka/src/remote-mypy/mypy/test/testoutput.py:45: in test_output_json
    out, err, returncode = api.run(mypy_cmdline)
                           ^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/api.py:72: in run
    return _run(
/home/jukka/src/remote-mypy/mypy/api.py:59: in _run
    main_wrapper(stdout, stderr)
/home/jukka/src/remote-mypy/mypy/api.py:73: in <lambda>
    lambda stdout, stderr: main(args=args, stdout=stdout, stderr=stderr, clean_exit=True)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/main.py:154: in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/main.py:244: in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/build.py:422: in build
    result = build_inner(
/home/jukka/src/remote-mypy/mypy/build.py:537: in build_inner
    graph = dispatch(sources, manager, stdout, connect_threads)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/build.py:4133: in dispatch
    raise OSError(f"Cannot connect to build worker(s): {', '.join(not_connected)}")
E   OSError: Cannot connect to build worker(s): 0, 1
----------------------------- Captured stdout call -----------------------------
Failed to establish connection with worker: No status file foundFailed to establish connection with worker: No status file found

----------------------------- Captured stderr call -----------------------------
/home/jukka/venv/mypy/bin/python: Error while finding module specification for 'mypy.build_worker' (ModuleNotFoundError: No module named 'mypy')
/home/jukka/venv/mypy/bin/python: Error while finding module specification for 'mypy.build_worker' (ModuleNotFoundError: No module named 'mypy')
______________________ testOutputJsonSyntaxErrorParallel _______________________
[gw7] linux -- Python 3.14.3 /home/jukka/venv/mypy/bin/python
data: /home/jukka/src/remote-mypy/test-data/unit/outputjson.test:62:
/home/jukka/src/remote-mypy/mypy/test/testoutput.py:23: in run_case
    test_output_json(testcase)
/home/jukka/src/remote-mypy/mypy/test/testoutput.py:45: in test_output_json
    out, err, returncode = api.run(mypy_cmdline)
                           ^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/api.py:72: in run
    return _run(
/home/jukka/src/remote-mypy/mypy/api.py:59: in _run
    main_wrapper(stdout, stderr)
/home/jukka/src/remote-mypy/mypy/api.py:73: in <lambda>
    lambda stdout, stderr: main(args=args, stdout=stdout, stderr=stderr, clean_exit=True)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/main.py:154: in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/main.py:244: in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/build.py:422: in build
    result = build_inner(
/home/jukka/src/remote-mypy/mypy/build.py:537: in build_inner
    graph = dispatch(sources, manager, stdout, connect_threads)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/jukka/src/remote-mypy/mypy/build.py:4133: in dispatch
    raise OSError(f"Cannot connect to build worker(s): {', '.join(not_connected)}")
E   OSError: Cannot connect to build worker(s): 0, 1
----------------------------- Captured stdout call -----------------------------
Failed to establish connection with worker: No status file foundFailed to establish connection with worker: No status file found

----------------------------- Captured stderr call -----------------------------
/home/jukka/venv/mypy/bin/python: Error while finding module specification for 'mypy.build_worker' (ModuleNotFoundError: No module named 'mypy')
/home/jukka/venv/mypy/bin/python: Error while finding module specification for 'mypy.build_worker' (ModuleNotFoundError: No module named 'mypy')
=========================== short test summary info ============================

[ilevkivskyi]

@JukkaL I think that is completely different. Why do you think it is similar?

[JukkaL]

The same test case testOutputJsonParallel is failing randomly. I forgot to mention that this was random. So it seems at least suspicious, though I don't know if the root cause is the same or even related, but it seems somewhat unlikely that there are two unrelated random failure modes for the same test case.

[ilevkivskyi]

The same test case testOutputJsonParallel is failing randomly

We have only two-three test cases that run in parallel mode without --mypy-num-workers specified for pytest (this is what parallel CI builds do). So same test failing is in no way an indication of anything.

[ilevkivskyi]

Anyway, your test failure seems like something is wrong with virtualenv and/or the test runner, the build workers fail to start because the module is not found. While the original traceback I posted looks like a genuine race condition:

• It happens in a thread
• It is a C-level assert failure in CPyArg_ParseStackAndKeywordsSimple which is stateful (because of CPyArg_Parser)

[JukkaL]

We have only two-three test cases that run in parallel mode without --mypy-num-workers specified for pytest (this is what parallel CI builds do). So same test failing is in no way an indication of anything.

Ok, that would explain these failures -- they appear to be unrelated after all.