bench: add experimental node:bench module

[jasnell]

A new node:bench module, modeled closely after the node:test architecture.

import { bench, suite } from 'node:bench';

suite('URL', () => {
  const input = 'https://example.com/a?b=c';

  bench('construct', {
    samples: 30,
    params: { input: 'short' },
  }, (b) => {
    const operations = 10_000;

    b.start();
    for (let i = 0; i < operations; i++) {
      new URL(input);
    }
    b.end(operations);
  });
});

node --bench benchmark.mjs

The core functionality is here but there are still improvements that are needed (see #65631). The focus was on minimal functionality, not absolutely optimal functionality. But this gives a solid starting point for improvements.

Architecturally, this is intended to be a minimal benchmark primitive. It's useful on it's own but provides a base set of primitives that more full-featured benchmark tools can build on. Specifically, this is not a full replacement for things like bench-node, etc. It shouldn't be judged as such.

[nodejs-github-bot]

Review requested:

• @nodejs/loaders
• @nodejs/performance
• @nodejs/startup
• @nodejs/tsc

[codecov]

Codecov Report

❌ Patch coverage is 98.99920% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (bb5cffc) to head (69ca552).
⚠️ Report is 31 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/bench_runner/cli.js	97.95%	13 Missing ⚠️
lib/internal/bench_runner/harness.js	99.49%	1 Missing and 3 partials ⚠️
src/histogram.cc	93.61%	0 Missing and 3 partials ⚠️
lib/internal/bench_runner/benchmark.js	99.61%	2 Missing ⚠️
src/node_options-inl.h	85.71%	0 Missing and 2 partials ⚠️
src/node_options.cc	97.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65606      +/-   ##
==========================================
+ Coverage   90.06%   90.14%   +0.08%     
==========================================
  Files         754      764      +10     
  Lines      255747   258220    +2473     
  Branches    48318    48885     +567     
==========================================
+ Hits       230327   232769    +2442     
- Misses      16550    16557       +7     
- Partials     8870     8894      +24     

Files with missing lines	Coverage Δ
lib/bench.js	100.00% <100.00%> (ø)
lib/bench/reporters.js	100.00% <100.00%> (ø)
lib/internal/bench_runner/benchmarks_stream.js	100.00% <100.00%> (ø)
lib/internal/bench_runner/reporter/json.js	100.00% <100.00%> (ø)
lib/internal/bench_runner/reporter/spec.js	100.00% <100.00%> (ø)
lib/internal/bench_runner/runner.js	100.00% <100.00%> (ø)
lib/internal/bootstrap/realm.js	96.37% <100.00%> (+0.01%)	⬆️
lib/internal/histogram.js	96.22% <100.00%> (+0.11%)	⬆️
lib/internal/main/bench_runner.js	100.00% <100.00%> (ø)
src/histogram.h	72.72% <ø> (ø)
... and 9 more

... and 39 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[RafaelGSS]

How is it different from what we (@H4ad) proposed in #50768?

[jsumners-nr]

@RafaelGSS probably nothing except that hopefully now it can progress. My question would be: why not vendor https://www.npmjs.com/package/bench-node?

[jasnell]

... why not vendor ...

I initially considered vendoring one of the existing packages and opted against it in favor of a light weight, minimal, no-new-dependency approach that a more feature-rich tool like bench-node can build/iterate on. I don't consider these either/or options.

Node.js' own benchmarks demonstrate that the full set of features provided by bench-node aren't necessary for a minimal bench suite. node:bench is intended to meet the low end, minimal requirement. These don't need to compete.

	node:bench	bench-node
Role	Minimal built-in benchmark kernel	Full-featured userland benchmark tool
Core features	Samples, warmup, suites, hooks, raw results, reporters, CLI isolation	Automatic warmup, V8 plugins, workers, DCE checks, baselines
Reporting	Structured event stream with spec and JSON reporters	Text, chart, pretty, HTML, JSON, and CSV
Dependencies	No new runtime dependencies	Piscina (installed but not used??)

We can provide stable measurement and structured result primitives without needing to ship the larger user-facing feature set and these existing tools can continue serving their current users while optionally adopting the built-in foundation. Just like node:test hasn't replaced the more sophisticated test suites like vitest.

That said, this is just a starting point. I've never been one to say "It must be done this specific way!" so all options are on the table. What I want is a built-in node --bench ... mechanism, I don't necessarily care how we get there as long as it works.

[jsumners-nr]

Just like node:test hasn't replaced the more sophisticated test suites like vitest.

If I can't do it with node:test I just don't worry about it.

That said, this is just a starting point. I've never been one to say "It must be done this specific way!" so all options are on the table. What I want is a built-in node --bench ... mechanism, I don't necessarily care how we get there as long as it works.

I'm with you. I just think it's easier to deliver what folks would actually want out of the module by bringing something like bench-node in. Look at the logger attempt. Instead of bringing in pino (yes, Matteo and I were/are both against that) a whole new logger was written while trying to keep the learnings from pino. That failed. But the parts undici is used for seem to be working. I think the same approach can be used.

[jasnell]

... what folks would actually want out of the module ...

different folks "actually want" different things. You prove this point yourself with the "If I can't do it with node:test I just don't worry about it" comment. Node.js' own benchmarks don't require any of the extended features of bench-node

[jsumners-nr]

Node.js' own benchmarks don't require any of the extended features of bench-node

Maybe, but aren't standard library modules meant to provide utility to the community, not just the project itself?

[jasnell]

Maybe, but aren't standard library modules meant to provide utility to the community, not just the project itself?

Why not both?

I'm intentionally designing this for both cases. Standalone utility plus primitives that can be used by tools like bench-node.

[jasnell]

Large PR Review Guide:

Given that this is a large PR for a new subsystem... some guidance for review:

1. I would start with the bench.md and test/fixtures/bench-runner examples, and test/parallel/test-bench-* to get a sense of the overall How It Works and What It Is.
2. The design is intentionally very similar to the test runner design. Familiarity with that helps.
3. The intent here is not a full-featured bench tool like bench-node but a minimal kernel that can provide fundamental benchmark primitives suitable at the very least for our our /benchmark tooling and that full-featured bench tools can build on.
4. Once you've got an understanding of the what, look at the implementation in lib/internal
5. This PR provides the core bits, the follow-up PR here bench: additional work on node:bench #65631 includes refinements.

[ronag]

How does this compare to mitata?