test: investigate async-hooks/test-signalwrap

[refack]

• Version: master
• Platform: aix61-ppc64
• Subsystem: test,async_hooks

not ok 39 async-hooks/test-signalwrap
  ---
  duration_ms: 0.240
  severity: fail
  stack: |-
    Mismatched onsigusr2 function calls. Expected exactly 2, actual 0.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/common/index.js:484:10)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/async-hooks/test-signalwrap.js:14:30)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
        at startup (bootstrap_node.js:158:16)
  ...

https://ci.nodejs.org/job/node-test-commit-aix/6994/nodes=aix61-ppc64/console

/cc @nodejs/async_hooks @nodejs/platform-aix

[gireeshpunathil]

Running a thousand times locally to see if it fails.

[refack]

Stress CI (the whole test/async-hooks/ suite [which runs in parallel]): https://ci.nodejs.org/job/node-stress-single-test/1314/nodes=aix61-ppc64/

[refack]

Stress CI (the whole test/async-hooks/ suite [which runs in parallel]): https://ci.nodejs.org/job/node-stress-single-test/1314/nodes=aix61-ppc64/

Stress returned no results 🤷‍♂️

[Trott]

Stress returned no results

You had previously landed something that has the async-hooks tests running parallel, right? If so, maybe try the stress test again under load by passing something like -j 96 --repeat 960? Just an idea, not a particularly informed guess or anything as I haven't looked at the test. It's kinda my standard first thing to try when I can't make a test that's failing on CI fail otherwise.

[refack]

You had previously landed something that has the async-hooks tests running parallel, right? If so, maybe try the stress test again under load by passing something like -j 96 --repeat 960? Just an idea, not a particularly informed guess or anything as I haven't looked at the test. It's kinda my standard first thing to try when I can't make a test that's failing on CI fail otherwise.

Worth a shot - https://ci.nodejs.org/job/node-stress-single-test/1315/nodes=aix61-ppc64/
Why 96?

[Trott]

Why 96?

I think there's a default shell setting on macOS that will stop things at 100, so I usually go for just under that as a first pass. And 96 instead of 97, 98, or 99 because it's kinda sorta almost a power of 2 (64 + 32), and we all prefer powers of 2, don't we?

[refack]

@Trott you are an evil genius 😉 this gave us a plethora of weirdness... No fails in async-hooks/test-signalwrap though.

A harness burp:

ok 869 async-hooks/test-callback-error
  ---
  duration_ms: 4.485
  ...
Exception in thread Thread-23:
Traceback (most recent call last):
  File "/opt/freeware/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/opt/freeware/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "tools/test.py", line 140, in RunSingle
    output = case.Run()
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 537, in Run
    "TEST_THREAD_ID": "%d" % self.thread_id
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 519, in RunCommand
    env)
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 752, in Execute
    pty_out = pty_out
  File "/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/tools/test.py", line 649, in RunProcess
    **rest
  File "/opt/freeware/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/opt/freeware/lib/python2.7/subprocess.py", line 1231, in _execute_child
    self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

ok 870 async-hooks/test-callback-error

A fail in callback-error (here we go again 🤦‍♂️ )

not ok 609 async-hooks/test-callback-error
  ---
  duration_ms: 4.97
  severity: fail
  stack: |-
    start case 1
    assert.js:586
    assert.ifError = function ifError(err) { if (err) throw err; };
                                                      ^
    
    Error: spawnSync /home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/out/Release/node EAGAIN
        at exports._errnoException (util.js:1020:11)
        at Object.spawnSync (internal/child_process.js:918:20)
        at spawnSync (child_process.js:548:24)
        at Object.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:51:17)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
  ...

A different callback-error error:

not ok 41 async-hooks/test-callback-error
  ---
  duration_ms: 7.592
  severity: fail
  stack: |-
    start case 1
    end case 1: 726.624ms
    start case 2
    end case 2: 428.448ms
    start case 3
    /home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:84
      child.stdout.on('data', (data) => {
                  ^
    
    TypeError: Cannot read property 'on' of undefined
        at Object.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-callback-error.js:84:15)
        at Module._compile (module.js:569:30)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:503:32)
        at tryModuleLoad (module.js:466:12)
        at Function.Module._load (module.js:458:3)
        at Function.Module.runMain (module.js:605:10)
        at startup (bootstrap_node.js:158:16)
        at bootstrap_node.js:575:3
  ...

[BridgeAR]

@refack if I understand your comment correct it seems like the original error just had a hiccup and other errors came up as being flaky instead?

[refack]

@BridgeAR It seems like stressing AIX gives many varied failures. But AFAICT the issue in the OP is still unresolved, but also unreproducible.

Personally I haven't seen any of those since. I'll run another stress test, and if nothing comes up consistently I think this can be closed.

https://ci.nodejs.org/job/node-stress-single-test/1424/nodes=aix61-ppc64/

[refack]

Well we got one of the original:

not ok 24 async-hooks/test-signalwrap
  ---
  duration_ms: 7.424
  severity: fail
  stack: |-
    Mismatched onsigusr2Again function calls. Expected exactly 1, actual 0.
        at Object.exports.mustCall (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/common/index.js:485:10)
        at process.onsigusr2 (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/async-hooks/test-signalwrap.js:48:34)
        at process.<anonymous> (/home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64/test/common/index.js:517:15)
        at emitNone (events.js:105:13)
        at process.emit (events.js:207:7)
        at Signal.wrap.onsignal (internal/process.js:198:44)
  ...

(1 out of 100 while testing with -j 96 async-hooks)