Performance regression after updating to V8 11.3

[anonrig]

I realized that after V8 11.3 the benchmarks changed a lot. As far as I know, Ada performs worse right now, and I'm researching the root cause of this.

Prior to V8 here is the benchmark CI result of Ada: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1307/

Comparing main branch before V8 11.3 merging with Ada branch (with V8 11.3)

➜  node git:(update-ada-v2) ✗ node-benchmark-compare url-comparing-v10-with-ada.csv
                                                                                         confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***     13.79 %       ±2.04% ±2.72% ±3.54%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***     12.54 %       ±2.60% ±3.46% ±4.51%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    102.67 %       ±2.20% ±2.93% ±3.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     91.53 %       ±2.84% ±3.79% ±4.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***    -37.76 %       ±1.57% ±2.09% ±2.73%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***    -36.73 %       ±1.49% ±1.98% ±2.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***     13.55 %       ±1.63% ±2.17% ±2.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***     12.03 %       ±3.29% ±4.38% ±5.70%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***      6.59 %       ±2.42% ±3.22% ±4.19%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***      5.79 %       ±1.66% ±2.22% ±2.90%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     25.56 %       ±2.65% ±3.53% ±4.60%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     22.16 %       ±2.59% ±3.44% ±4.48%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***     32.07 %       ±1.58% ±2.11% ±2.75%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     25.25 %       ±1.65% ±2.19% ±2.85%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***     11.55 %       ±3.28% ±4.39% ±5.76%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***      8.60 %       ±3.71% ±4.96% ±6.50%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     35.73 %       ±1.46% ±1.94% ±2.55%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***     14.72 %       ±2.33% ±3.12% ±4.09%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***     17.45 %       ±2.89% ±3.85% ±5.01%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***     15.45 %       ±1.94% ±2.59% ±3.39%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

Comparing current main branch with Ada branch (both on V8 11.3)

➜  node git:(update-ada-v2) ✗ node-benchmark-compare url.csv
                                                                                         confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***     11.85 %       ±1.05% ±1.40% ±1.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***      9.00 %       ±0.79% ±1.05% ±1.36%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    100.67 %       ±2.00% ±2.68% ±3.50%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     81.76 %       ±1.47% ±1.97% ±2.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***      7.12 %       ±1.26% ±1.68% ±2.19%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***      4.10 %       ±0.80% ±1.07% ±1.39%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***      9.33 %       ±0.91% ±1.21% ±1.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***      7.89 %       ±0.81% ±1.08% ±1.41%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***      5.47 %       ±1.31% ±1.75% ±2.29%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***      3.72 %       ±1.07% ±1.42% ±1.86%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     22.05 %       ±1.22% ±1.63% ±2.13%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     18.64 %       ±0.95% ±1.26% ±1.64%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***    103.22 %       ±1.39% ±1.86% ±2.43%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     88.36 %       ±1.07% ±1.42% ±1.86%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***      7.02 %       ±1.11% ±1.48% ±1.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***      4.21 %       ±1.08% ±1.44% ±1.88%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     29.17 %       ±1.28% ±1.71% ±2.24%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      5.96 %       ±1.49% ±1.99% ±2.62%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***     14.88 %       ±1.10% ±1.47% ±1.92%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***     11.53 %       ±0.95% ±1.26% ±1.64%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

cc @targos @nodejs/performance

[anonrig]

This is definitely not a regression on the recent changes I've did to Ada branch. Reverting all of those changes still didn't make a difference.

[anonrig]

Comparing Node 19.8.1 with current main branch

                                                                                         confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'                       1.51 %       ±2.04% ±2.73% ±3.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'                        2.44 %       ±2.56% ±3.41% ±4.44%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    -41.41 %       ±1.42% ±1.89% ±2.47%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***    -37.82 %       ±4.29% ±5.76% ±7.62%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***    -43.22 %       ±1.63% ±2.17% ±2.83%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***    -39.59 %       ±1.74% ±2.33% ±3.07%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'                        2.57 %       ±2.63% ±3.49% ±4.55%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                         1.21 %       ±2.30% ±3.06% ±3.98%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***    -42.30 %       ±1.20% ±1.61% ±2.11%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***    -39.76 %       ±1.11% ±1.48% ±1.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'                *      3.37 %       ±2.64% ±3.52% ±4.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'                **      3.22 %       ±2.23% ±2.99% ±3.92%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***    -35.73 %       ±1.65% ±2.20% ±2.87%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***    -33.85 %       ±1.75% ±2.33% ±3.04%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***    -40.80 %       ±3.16% ±4.24% ±5.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***    -38.83 %       ±1.20% ±1.60% ±2.08%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     -9.28 %       ±3.97% ±5.34% ±7.06%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      5.65 %       ±2.36% ±3.14% ±4.09%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                        -0.15 %       ±2.97% ±3.95% ±5.14%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                          0.58 %       ±3.00% ±3.99% ±5.20%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

[jasnell]

/cc @nodejs/v8 @targos

[anonrig]

I'm waiting on merging this PR in order to understand it better: #47340

[anonrig]

Once noticable change I saw on the V8 update is: we enabled turbofan compiler, and disabled v8_enable_static_roots

0d75c05#diff-d53f68b29a1c48c958c2e6779cc25c916a986357c6010dd01421c17adcf2f09bR282

[anonrig]

Ada microbenchmarks still show the same result (295ns/iter) prior to V8 update, but the node benchmarks show bad results.

cpu: Apple M1 Max
runtime: node v20.0.0-pre (arm64-darwin)

benchmark         time (avg)             (min … max)       p75       p99      p995
---------------------------------------------------- -----------------------------
noop          316.15 ps/iter     (300 ps … 11.49 ns)  312.5 ps  337.5 ps  345.8 ps
noop            4.08 ns/iter    (4.04 ns … 10.52 ns)   4.06 ns   4.55 ns   4.94 ns
basic domain  295.26 ns/iter (290.07 ns … 309.39 ns) 298.61 ns 307.27 ns 309.39 ns
ipv4          262.41 ns/iter (257.75 ns … 276.33 ns) 265.86 ns 270.04 ns 270.13 ns
ipv6          292.75 ns/iter (284.33 ns … 354.96 ns)  295.3 ns 326.28 ns 354.96 ns
complex       438.23 ns/iter  (433.1 ns … 451.02 ns) 441.98 ns 449.62 ns 451.02 ns

[mscdex]

we enabled turbofan compiler,

TurboFan has been enabled by default for years now ever since Crankshaft was removed. The addition of the v8_enable_turbofan flag in the V8 gypi is just making that more explicit, so there should be no effective difference.

[anonrig]

cc @joyeecheung

[anonrig]

My local benchmarks show really bad results, but benchmark CI seems to persist with the current results. I'm closing this issue and open a new one if the situation happens again.

CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1314/consoleText

url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***    106.73 %      ±19.04% ±25.55% ±33.70%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***    111.98 %      ±18.10% ±24.30% ±32.05%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    112.52 %      ±16.28% ±21.83% ±28.73%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     91.22 %      ±17.19% ±23.05% ±30.35%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***     31.72 %      ±12.53% ±16.68% ±21.72%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***     31.05 %      ±12.37% ±16.46% ±21.43%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***     83.54 %      ±15.31% ±20.49% ±26.90%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***     81.76 %      ±16.08% ±21.57% ±28.44%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***    137.30 %      ±18.20% ±24.43% ±32.23%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***    111.40 %      ±16.83% ±22.53% ±29.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     25.11 %       ±8.80% ±11.73% ±15.28%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     23.16 %       ±8.63% ±11.52% ±15.06%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***     41.12 %      ±14.86% ±19.79% ±25.79%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     37.21 %      ±12.93% ±17.23% ±22.48%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***    123.39 %      ±18.93% ±25.36% ±33.34%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***    103.11 %      ±17.88% ±23.97% ±31.54%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     35.04 %       ±2.25%  ±3.02%  ±3.98%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      9.76 %       ±4.21%  ±5.62%  ±7.36%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***    133.03 %      ±19.31% ±25.91% ±34.16%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***    113.69 %      ±18.69% ±25.08% ±33.08%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 40 comparisons, you can thus
expect the following amount of false-positive results:
  2.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.40 false positives, when considering a   1% risk acceptance (**, ***),
  0.04 false positives, when considering a 0.1% risk acceptance (***)