test-single-executable-application* tests flaking on macOS

[richardlau]

Test

test-single-executable-application

Platform

macOS

Console output

not ok 4699 sequential/test-single-executable-application
  ---
  duration_ms: 11491.49500
  severity: fail
  exitcode: 1
  stack: |-
    Copied /Users/admin/build/workspace/node-test-commit-osx/nodes/osx13-x64/out/Release/node to /Users/admin/node-tmp/.tmp.4698/sea
    Injected /Users/admin/node-tmp/.tmp.4698/sea-prep.blob into /Users/admin/node-tmp/.tmp.4698/sea
    Signed /Users/admin/node-tmp/.tmp.4698/sea
    [process 11198]: --- stderr ---
    
    [process 11198]: --- stdout ---
    
    [process 11198]: status = null, signal = SIGSEGV
    /Users/admin/build/workspace/node-test-commit-osx/nodes/osx13-x64/test/common/child_process.js:98
        throw error;
        ^
    
    Error: - process terminated with status null, expected 0
    - process terminated with signal SIGSEGV, expected null
        at Object.<anonymous> (/Users/admin/build/workspace/node-test-commit-osx/nodes/osx13-x64/test/sequential/test-single-executable-application.js:54:1)
        at Module._compile (node:internal/modules/cjs/loade...

Build links

• https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx13-x64/66348/console

Additional information

According to

• CI Reliability 2025-08-20 reliability#1293
• CI Reliability 2025-08-19 reliability#1292

Several test-single-executable-application* tests have been failing on one or more pull requests on macOS.

• sequential/test-single-executable-application
• sequential/test-single-executable-application-assets
• sequential/test-single-executable-application-assets-raw
• sequential/test-single-executable-application-disable-experimental-sea-warning
• sequential/test-single-executable-application-exec-argv
• sequential/test-single-executable-application-exec-argv-empty
• sequential/test-single-executable-application-snapshot
• sequential/test-single-executable-application-snapshot-and-code-cache
• sequential/test-single-executable-application-snapshot-worker
• sequential/test-single-executable-application-use-code-cache

It looks like the injected binary is crashing.

cc @nodejs/single-executable

[joyeecheung]

They seem to have gone in the recent CI runs: https://ci.nodejs.org/job/node-test-commit-osx/

I once ran into this after a rebase which was strangely persistent, and then it went away after another rebase. #59276 (comment) I wonder if it has something to do with the ephemeral macOS hosts and/or caching?

[panva]

They seem to have gone in the recent CI runs: ci.nodejs.org/job/node-test-commit-osx

I once ran into this after a rebase which was strangely persistent, and then it went away after another rebase. #59276 (comment) I wonder if it has something to do with the ephemeral macOS hosts and/or caching?

I had the same experience in #59497

[joyeecheung]

They seem to have come back 5 days ago and then gone, I noticed some patterns in https://github.com/nodejs/reliability/blob/main/reports/2025-09-02.md

1. They happen in a short amount of time on a few jobs and then go away (in the last report they happened in 68906, 68910, 68914, 68916 and then disappeared)
2. In the CI it only happens on the ephemeral macOS hosts
3. They don't seem to happen on GitHub actions on macOS, and I haven't seen them happening locally on my macOS machine

Given the pattern I still feel that it may be an issue with caching. Another possibility is code signing (I think it also crashes with segfault if you don't sign it properly on macOS) though I don't know how that's set up in the CI machines...

[richardlau]

I have no idea if it's the same cause, but the tests are consistently failing on the new macos15-x64 builds.

e.g. https://ci.nodejs.org/job/node-test-commit-osx/66925/nodes=macos15-x64/#showFailuresLink

07:53:44     [process 31240]: --- stderr ---
07:53:44     dyld[31240]: failed to set up thread local variables for '/Users/admin/node-tmp/.tmp.4757/sea': unsupported thread-local, larger than 4GB

@nodejs/single-executable

[ryanaslett]

"Another possibility is code signing" -> the only VM's capable of code signing are the release machines, unless these tests are relying on an existing binary that already been signed, I think that can be ruled out.

[joyeecheung]

unless these tests are relying on an existing binary that already been signed, I think that can be ruled out.

The tests relied on signing the generated SEAs or else they won't run at all. The tests would skip running these generated binaries if code signing is not available and therefore would not crash at running them (note that the tests don't need to sign them with an identity, the command is merely codesign --sign - /path/to/sea, I think it doesn't need any setup - I've never set up my local machine for signing with an identity and it can sign the SEA for the tests with this command just alright). Although it does need the codesign tool to work properly in the machine, while I am not sure if the macs in the CI has that...

[joyeecheung]

The unsupported thread-local, larger than 4GB seems to appear quite often for macos x64. I wonder if it's a bug in postject that corrupts the executable during injection and dyld ends up thinking something is TLS even though it's not. Maybe updating the lief dependency in postject would do something considering it's been a bit outdated and there might be some bug fixes..