diff --git a/CHANGES.md b/CHANGES.md index 29fd7c1a0..b9ed9f29b 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -291,8 +291,22 @@ To be released. discovery-aware `--dry-run` planning, and ships with a local benchmark fixture used by the scenario tests. [[#744], [#783], [#784]] + - Added `actor`, `object`, `fanout`, `failure`, and `mixed` scenario runners + to `fedify bench`. Read scenarios can now benchmark actor and object + document fetches, including authenticated GET requests; fanout scenarios + drive the benchmark trigger endpoint and wait for queue task drain; failure + scenarios report expected fault outcomes as successes; and mixed scenarios + run weighted child scenario blends. The `collection` scenario type remains + reserved but not executable. Fanout and remote failure scenarios can set + `sinkBase` to generate deterministic benchmark sink inbox URLs for targets + that keep `triggerSinks` allowlisting enabled. This change is published + as benchmark scenario schema version 2. [[#744], [#785], [#801], [#802]] + [#783]: https://github.com/fedify-dev/fedify/issues/783 [#784]: https://github.com/fedify-dev/fedify/issues/784 +[#785]: https://github.com/fedify-dev/fedify/issues/785 +[#801]: https://github.com/fedify-dev/fedify/pull/801 +[#802]: https://github.com/fedify-dev/fedify/pull/802 ### @fedify/fixture diff --git a/docs/manual/benchmarking.md b/docs/manual/benchmarking.md index 0c04a2387..d20b94e0f 100644 --- a/docs/manual/benchmarking.md +++ b/docs/manual/benchmarking.md @@ -94,9 +94,9 @@ delivery with the same `@fedify/fedify` signer a real peer uses, so the measured crypto cost is real. > [!NOTE] -> This version runs the `inbox` and `webfinger` scenario types. The scenario -> format can express the others (`actor`, `object`, `fanout`, `collection`, -> `failure`, and `mixed`), but they are not executed yet. Within the runnable +> This version runs the `inbox`, `webfinger`, `actor`, `object`, `fanout`, +> `failure`, and `mixed` scenario types. The `collection` scenario type is +> reserved by the suite format but is not executed yet. Within the runnable > types, a few options the format accepts are also not implemented yet and are > rejected up front with a clear message: > @@ -115,7 +115,7 @@ is a superset). The suite declares the `target`, shared `defaults`, the block of pass/fail thresholds: ~~~~ yaml -# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v1.json +# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v2.json version: 1 target: http://localhost:3000 defaults: @@ -161,7 +161,56 @@ list, deliveries are rotated across the recipients (and across the synthetic `actors` signing them), modeling a server that receives from many peers into many local inboxes. -[published schema]: https://json-schema.fedify.dev/bench/scenario-v1.json +[published schema]: https://json-schema.fedify.dev/bench/scenario-v2.json + +### Scenario types + +The runnable scenario types cover the main benchmark surfaces: + + - `inbox`: discovers recipient inboxes and sends signed `Create(Note)` + deliveries through the target's inbound ActivityPub path. + - `webfinger`: drives direct `/.well-known/webfinger` lookups on the target. + - `actor`: resolves actor URLs from the scenario recipients and fetches actor + documents. Set `authenticated: true` to sign those GET requests. + - `object`: fetches object URLs from `source`. Set `authenticated: true` to + sign those GET requests. + - `fanout`: posts to `/.well-known/fedify/bench/trigger` so the target calls + `sendActivity()` and drains its fanout/outbox queue to benchmark-owned sink + inboxes. The command starts those sink inboxes locally. A non-loopback + target therefore needs `--advertise-host` unless the scenario sets + `sinkBase` to a reachable `http://host:port/` URL. The target must either + allow the generated sink inboxes through `triggerSinks` or run with + `allowUnsafeTriggerRecipients` in a controlled benchmark environment. Use + `sinkBase` when you want those inboxes to be deterministic, for example + `http://127.0.0.1:9090/inbox/0` through + `http://127.0.0.1:9090/inbox/4` for `followers: 5`. + `fedify bench` does not switch the target's queue backend; run the same + suite against targets configured with the queue implementations you want to + compare. Fanout triggers are serialized while the runner observes queue + drain, so client latency includes time spent waiting for earlier fanout + drains under high-rate or concurrent load. Use `deliveryThroughput` and + `queueDrain` expectations for delivery performance, and keep request + latency expectations conservative for this scenario type. + - `failure`: records expected fault outcomes as successes. For this + scenario type, `successRate` means “the expected failure was observed,” + not “the HTTP request succeeded.” The `invalid-signature` and + `missing-actor` faults send malformed signed deliveries to a recipient + inbox. The `remote-404`, `remote-410`, `slow-inbox`, and `network-error` + faults post to the benchmark trigger endpoint with `sender`, so the target + uses its normal outbound delivery path against controlled benchmark-owned + sink inboxes. Like `fanout`, these remote failure faults need + `--advertise-host` for a non-loopback target unless `sinkBase` gives a + reachable, fixed sink base URL that the target's `triggerSinks` can + preconfigure. Remote failure deliveries are also serialized while the + runner waits for the target's queue to + observe the expected failure or retry signal, so request latency can include + earlier wait time when the configured load is concurrent or high-rate. + - `mixed`: runs referenced child scenarios concurrently, splitting the + `mixed` scenario's load by each entry's `weight`. The referenced + scenarios are named scenarios in the same suite and are still run as normal + suite entries when listed. The mixed result merges client-side request, + throughput, delivery throughput, latency, and error measurements; + server-side metric snapshots are not merged across child runners. ### Actors @@ -239,7 +288,7 @@ CI check. Keep CI gates on robust signals such as success rate, error counts, and gross throughput or latency floors; precise latency-percentile regression belongs in a controlled environment, not a shared CI runner. -[report schema]: https://json-schema.fedify.dev/bench/report-v1.json +[report schema]: https://json-schema.fedify.dev/bench/report-v2.json ### Safety @@ -347,6 +396,11 @@ allowlist. To bypass this guard for a controlled run, set `~FederationBenchmarkOptions.allowUnsafeTriggerRecipients` to `true` in the application configuration. +For `fanout` and remote `failure` scenarios, set a `sinkBase` value such as +`http://host:port/` in the scenario when the target keeps the safe default and +you need stable sink URLs for `triggerSinks`. With `followers: 5`, the runner +generates `/inbox/0` through `/inbox/4` under that base. + A successful trigger returns `202 Accepted`: ~~~~ json diff --git a/packages/cli/src/bench/__fixtures__/invalid/failure-missing-fault.yaml b/packages/cli/src/bench/__fixtures__/invalid/failure-missing-fault.yaml deleted file mode 100644 index 88f6e3abd..000000000 --- a/packages/cli/src/bench/__fixtures__/invalid/failure-missing-fault.yaml +++ /dev/null @@ -1,6 +0,0 @@ -# A failure scenario must declare at least one fault. -version: 1 -target: http://localhost:3000 -scenarios: - - name: broken - type: failure diff --git a/packages/cli/src/bench/__fixtures__/invalid/mixed-server-metric.yaml b/packages/cli/src/bench/__fixtures__/invalid/mixed-server-metric.yaml new file mode 100644 index 000000000..d2898921b --- /dev/null +++ b/packages/cli/src/bench/__fixtures__/invalid/mixed-server-metric.yaml @@ -0,0 +1,10 @@ +# Server-side metrics are not merged for mixed scenario results. +version: 1 +target: http://localhost:3000 +scenarios: + - name: realistic-blend + type: mixed + mix: + - { scenario: fanout-1k, weight: 1 } + expect: + queueDrain.p95: "< 2s" diff --git a/packages/cli/src/bench/__fixtures__/invalid/object-signature-expect-unauthenticated.yaml b/packages/cli/src/bench/__fixtures__/invalid/object-signature-expect-unauthenticated.yaml new file mode 100644 index 000000000..0dd4b5dda --- /dev/null +++ b/packages/cli/src/bench/__fixtures__/invalid/object-signature-expect-unauthenticated.yaml @@ -0,0 +1,9 @@ +# signatureVerification.* is only valid for authenticated object scenarios. +version: 1 +target: http://localhost:3000 +scenarios: + - name: object-fetch + type: object + source: http://localhost:3000/objects/1 + expect: + signatureVerification.p95: "< 10ms" diff --git a/packages/cli/src/bench/__fixtures__/reports/inbox-report.json b/packages/cli/src/bench/__fixtures__/reports/inbox-report.json index b952b1f5a..b7ca535f8 100644 --- a/packages/cli/src/bench/__fixtures__/reports/inbox-report.json +++ b/packages/cli/src/bench/__fixtures__/reports/inbox-report.json @@ -1,6 +1,6 @@ { - "$schema": "https://json-schema.fedify.dev/bench/report-v1.json", - "schemaVersion": 1, + "$schema": "https://json-schema.fedify.dev/bench/report-v2.json", + "schemaVersion": 2, "tool": { "name": "@fedify/cli", "version": "2.3.0" }, "environment": { "runtime": "deno", diff --git a/packages/cli/src/bench/__fixtures__/scenarios/all-types.yaml b/packages/cli/src/bench/__fixtures__/scenarios/all-types.yaml index c83a775d1..3e2324726 100644 --- a/packages/cli/src/bench/__fixtures__/scenarios/all-types.yaml +++ b/packages/cli/src/bench/__fixtures__/scenarios/all-types.yaml @@ -1,6 +1,6 @@ # yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v1.json -# Exercises every scenario type the format can express, even though only -# `inbox` and `webfinger` have runners in this version. +# Exercises every scenario type the format can express; `collection` is still +# reserved but not executable in this version. version: 1 target: http://localhost:3000 defaults: diff --git a/packages/cli/src/bench/__fixtures__/scenarios/authenticated-object-expect.yaml b/packages/cli/src/bench/__fixtures__/scenarios/authenticated-object-expect.yaml new file mode 100644 index 000000000..e591096e1 --- /dev/null +++ b/packages/cli/src/bench/__fixtures__/scenarios/authenticated-object-expect.yaml @@ -0,0 +1,10 @@ +# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v2.json +version: 1 +target: http://localhost:3000 +scenarios: + - name: signed-object-fetch + type: object + authenticated: true + source: http://localhost:3000/objects/1 + expect: + signatureVerification.p95: "< 10ms" diff --git a/packages/cli/src/bench/action.test.ts b/packages/cli/src/bench/action.test.ts index 3c87a6396..b58dd9733 100644 --- a/packages/cli/src/bench/action.test.ts +++ b/packages/cli/src/bench/action.test.ts @@ -3,6 +3,7 @@ import { mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import test from "node:test"; +import { serve } from "srvx"; import { spawnBenchmarkTarget } from "../../test/bench/fixture.ts"; import runBench, { withUserAgent } from "./action.ts"; import type { BenchCommand } from "./command.ts"; @@ -208,6 +209,104 @@ scenarios: } }); +test("runBench - dry run resolves actor handles with configured fetch", async () => { + const file = await writeSuite(`version: 1 +target: http://127.0.0.1:3000 +scenarios: + - name: actor-read + type: actor + recipient: "acct:alice@example.test" + load: { concurrency: 1 } + duration: 1ms +`); + let code = -1; + let output = ""; + let webfingerFetched = false; + let webfingerUserAgent: string | null = null; + await runBench(command({ scenario: file, dryRun: true }), { + exit: (c) => { + code = c; + }, + writeOutput: (c) => { + output = c; + return Promise.resolve(); + }, + log: () => {}, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + const headers = new Headers( + init?.headers ?? (input instanceof Request ? input.headers : undefined), + ); + if (url.pathname === "/.well-known/webfinger") { + webfingerFetched = true; + webfingerUserAgent = headers.get("user-agent"); + return Promise.resolve( + new Response( + JSON.stringify({ + subject: url.searchParams.get("resource"), + links: [{ + rel: "self", + href: "http://127.0.0.1:3000/users/alice", + }], + }), + { + headers: { "content-type": "application/jrd+json" }, + }, + ), + ); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.strictEqual(code, 0); + assert.strictEqual(webfingerFetched, true); + assert.strictEqual(webfingerUserAgent, "Fedify-bench-test/1.0"); + assert.match(output, /\/users\/alice/); +}); + +test("runBench - dry run gates object discovery before fetching", async () => { + const file = await writeSuite(`version: 1 +target: http://127.0.0.1:3000 +scenarios: + - name: object-crawl + type: object + source: + seed: "https://public.example/users/alice" + collection: outbox + limit: 1 + load: { concurrency: 1 } + duration: 1ms +`); + let code = -1; + let output = ""; + let publicFetched = false; + await runBench(command({ scenario: file, dryRun: true }), { + exit: (c) => { + code = c; + }, + writeOutput: (c) => { + output = c; + return Promise.resolve(); + }, + log: () => {}, + resolveTargetAddresses: resolvePublicHost, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.hostname === "public.example") { + publicFetched = true; + throw new Error("dry-run fetched an unsafe object source"); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.strictEqual(code, 0); + assert.strictEqual(publicFetched, false); + assert.match(output, /object discovery failed/); + assert.match(output, /Refusing to send benchmark read load/); +}); + test("runBench - unsafe override requires an explicit CLI target", async () => { const file = await writeSuite(`version: 1 target: https://example.com @@ -324,6 +423,267 @@ scenarios: assert.match(message, /advertise-host/); }); +test("runBench - remote failure needs advertised sink reachability", async () => { + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: remote-404 + type: failure + fault: remote-404 + sender: alice + load: { rate: 1/s } + duration: 1ms +`); + let code = -1; + let message = ""; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => Promise.reject(new Error("offline")), + }); + assert.strictEqual(code, 2); + assert.match(message, /advertise-host/); +}); + +test("runBench - default failure fault needs advertised sink reachability", async () => { + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: default-failure + type: failure + sender: alice + load: { rate: 1/s } + duration: 1ms +`); + let code = -1; + let message = ""; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => Promise.reject(new Error("offline")), + }); + assert.strictEqual(code, 2); + assert.match(message, /advertise-host/); +}); + +test("runBench - remote failure uses advertised sink reachability", async () => { + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: remote-404 + type: failure + fault: remote-404 + sender: alice + load: { concurrency: 1 } + duration: 25ms + queueDrainTimeout: 1s +`); + let code = -1; + let message = ""; + let triggerCalls = 0; + await runBench(command({ scenario: file, advertiseHost: "127.0.0.1" }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve( + new Response( + JSON.stringify({ + version: 1, + source: "server", + scopeMetrics: [{ + metrics: [ + sumMetric("fedify.queue.task.enqueued", triggerCalls), + sumMetric("fedify.queue.task.completed", triggerCalls), + sumMetric("fedify.queue.task.failed", 0), + sumMetric( + "activitypub.delivery.permanent_failure", + triggerCalls, + ), + ], + }], + }), + { headers: { "content-type": "application/json" } }, + ), + ); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve( + new Response(JSON.stringify({ version: 1 }), { + status: 202, + headers: { "content-type": "application/json" }, + }), + ); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.strictEqual(code, 0, message); + assert.ok(triggerCalls > 0); +}); + +test("runBench - missing-actor failure needs no advertise host", async () => { + const recipientTarget = await spawnBenchmarkTarget(); + try { + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: missing-actor + type: failure + fault: missing-actor + recipient: "${new URL("/users/alice", recipientTarget.url).href}" + load: { rate: 1/s } + duration: 1ms +`); + let code = -1; + let message = ""; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if ( + url.origin === recipientTarget.url.origin && + url.pathname === "/inbox" + ) { + return Promise.resolve( + new Response("actor not found", { status: 401 }), + ); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + assert.strictEqual(code, 0, message); + } finally { + await recipientTarget.close(); + } +}); + +test("runBench - missing-actor failure allows private inbox without advertise host", async () => { + const actorServer = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request): Response { + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return new Response( + JSON.stringify({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Person", + id: url.href, + inbox: "http://10.0.0.8/inbox", + }), + { headers: { "content-type": "application/activity+json" } }, + ); + } + return new Response("not found", { status: 404 }); + }, + }); + await actorServer.ready(); + try { + const actorUrl = new URL("/users/alice", new URL(actorServer.url!)); + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: missing-actor + type: failure + fault: missing-actor + recipient: "${actorUrl.href}" + load: { rate: 1/s } + duration: 1ms +`); + let code = -1; + let message = ""; + let inboxPosts = 0; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.href === "http://10.0.0.8/inbox") { + inboxPosts++; + return Promise.resolve( + new Response("actor not found", { status: 401 }), + ); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + assert.strictEqual(code, 0, message); + assert.ok(inboxPosts > 0); + } finally { + await actorServer.close(true); + } +}); + +test("runBench - unauthenticated actor read needs no advertise host", async () => { + const file = await writeSuite(`version: 1 +target: http://10.10.0.5:8000 +scenarios: + - name: actor-read + type: actor + recipient: http://10.10.0.6/users/alice + load: { rate: 1/s } + duration: 1ms +`); + let code = -1; + let output = ""; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: (c) => { + output = c; + return Promise.resolve(); + }, + log: () => {}, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + }); + assert.strictEqual(code, 0); + assert.strictEqual(JSON.parse(output).scenarios[0].requests.successRate, 1); +}); + test("runBench - refuses an inbox destination off the gated target (exit 2)", async () => { // A loopback target passes the gate, but an explicit public `inbox:` is the // actual load destination; it must be gated too, or production could be @@ -568,6 +928,131 @@ scenarios: assert.strictEqual(fetched, false); }); +test("runBench - invalid mixed child exits 2 before any probe", async () => { + const file = await writeSuite(`version: 1 +target: http://localhost:3000 +scenarios: + - name: mixed + type: mixed + mix: + - scenario: missing + weight: 1 +`); + let code = -1; + let message = ""; + let fetched = false; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => { + fetched = true; + return Promise.reject(new Error("no request should be sent")); + }, + }); + assert.strictEqual(code, 2); + assert.match(message, /unknown mixed child/); + assert.strictEqual(fetched, false); +}); + +test("runBench - mixed server expectation exits 2 before any probe", async () => { + const file = await writeSuite(`version: 1 +target: http://localhost:3000 +scenarios: + - name: lookup + type: webfinger + recipient: acct:alice@example.com + - name: mixed + type: mixed + mix: + - scenario: lookup + weight: 1 + expect: + queueDrain.p95: "< 10ms" +`); + let code = -1; + let message = ""; + let fetched = false; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => { + fetched = true; + return Promise.reject(new Error("no request should be sent")); + }, + }); + assert.strictEqual(code, 2); + assert.match(message, /queueDrain\.p95/); + assert.strictEqual(fetched, false); +}); + +test("runBench - invalid object source exits 2 before any probe", async () => { + const file = await writeSuite(`version: 1 +target: http://localhost:3000 +scenarios: + - name: object + type: object + source: objects/1 +`); + let code = -1; + let message = ""; + let fetched = false; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => { + fetched = true; + return Promise.reject(new Error("no request should be sent")); + }, + }); + assert.strictEqual(code, 2); + assert.match(message, /invalid object source URL/); + assert.strictEqual(fetched, false); +}); + +test("runBench - invalid actor recipient exits 2 before any probe", async () => { + const file = await writeSuite(`version: 1 +target: http://localhost:3000 +scenarios: + - name: actor + type: actor + recipient: alice +`); + let code = -1; + let message = ""; + let fetched = false; + await runBench(command({ scenario: file }), { + exit: (c) => { + code = c; + }, + writeOutput: () => Promise.resolve(), + log: (m) => { + message = m; + }, + fetch: () => { + fetched = true; + return Promise.reject(new Error("no request should be sent")); + }, + }); + assert.strictEqual(code, 2); + assert.match(message, /invalid actor recipient/); + assert.strictEqual(fetched, false); +}); + test("runBench - invalid suite exits 2", async () => { const file = await writeSuite(`target: http://localhost:3000 scenarios: @@ -589,3 +1074,11 @@ scenarios: assert.strictEqual(code, 2); assert.match(message, /Invalid/); }); + +function sumMetric(name: string, value: number): Record { + return { + name, + dataPointType: "sum", + dataPoints: [{ value }], + }; +} diff --git a/packages/cli/src/bench/action.ts b/packages/cli/src/bench/action.ts index 00798f2d5..afc1f6f01 100644 --- a/packages/cli/src/bench/action.ts +++ b/packages/cli/src/bench/action.ts @@ -10,6 +10,10 @@ import { discoverInbox, selectInbox, } from "./discovery/discover.ts"; +import { + actorUrlsFromRecipients, + objectUrlsFromSource, +} from "./scenarios/object-discovery.ts"; import { buildReport, buildScenarioResult, @@ -63,9 +67,6 @@ export interface RunBenchDeps { readonly resolveTargetAddresses?: ResolveTargetAddresses; } -/** The scenario types that need the synthetic actor/key server. */ -const SIGNED_TYPES = new Set(["inbox"]); - /** * Runs the `fedify bench` command: load and validate the suite, gate the * target, run each scenario, and render the report. The process exits 0 when @@ -112,7 +113,7 @@ export default async function runBench( try { runners = suite.scenarios.map((scenario) => { const runner = runnerFor(scenario.type); - runner.validate?.(scenario); + runner.validate?.(scenario, { scenarios: suite.scenarios }); validateExpectBlock(scenario.expect); return runner; }); @@ -194,6 +195,53 @@ export default async function runBench( defaults: validated.defaults, }); }; + const assertDestinationWithoutSyntheticServerAllowed = async ( + url: URL, + scenario: ResolvedScenario, + loadDescription: string, + ): Promise => { + const sameOrigin = url.origin === suite.target.origin; + const destinationTier = sameOrigin + ? tier + : await classifyResolvedTarget(url, deps.resolveTargetAddresses); + const inheritsTargetGate = sameOrigin && probe.benchmarkMode; + if ( + destinationTier === "public" && !inheritsTargetGate && + !command.allowUnsafeTarget + ) { + throw new UnsafeTargetError( + `Refusing to send ${loadDescription} to ${url.href}: it is public ` + + "and not part of the benchmarked target. Pass " + + "--allow-unsafe-target to override.", + ); + } + assertPublicDestinationOverrideAllowed(url, scenario, { + targetOrigin: suite.target.origin, + targetBenchmarkMode: probe.benchmarkMode, + allowUnsafe: command.allowUnsafeTarget, + explicitCliTarget: command.target != null, + destinationTier, + defaults: validated.defaults, + }); + }; + const assertReadDestinationAllowed = ( + url: URL, + scenario: ResolvedScenario, + ): Promise => + assertDestinationWithoutSyntheticServerAllowed( + url, + scenario, + "benchmark read load", + ); + const assertActorlessDestinationAllowed = ( + url: URL, + scenario: ResolvedScenario, + ): Promise => + assertDestinationWithoutSyntheticServerAllowed( + url, + scenario, + "benchmark load", + ); if (command.dryRun) { try { @@ -202,7 +250,9 @@ export default async function runBench( documentLoader, contextLoader, allowPrivateAddress, + fetch: fetchImpl, assertDestinationAllowed, + assertReadDestinationAllowed, }), command.output, ); @@ -220,14 +270,16 @@ export default async function runBench( // rather than let every signed delivery fail key lookup. if ( tier !== "loopback" && command.advertiseHost == null && - suite.scenarios.some((s) => SIGNED_TYPES.has(s.type)) + suite.scenarios.some((scenario) => + scenarioNeedsReachableLocalServer(scenario, suite.scenarios) + ) ) { log( - "Signed scenarios (inbox) need the benchmark's synthetic actor server to " + - "be reachable from the target. A loopback target reaches it " + - "automatically; for a non-loopback target, pass --advertise-host with " + - "an address the target can reach (the synthetic server then binds all " + - "interfaces), or use a read scenario such as webfinger.", + "Some scenarios need benchmark-owned local servers to be reachable from " + + "the target. A loopback target reaches them automatically; for a " + + "non-loopback target, pass --advertise-host with an address the target " + + "can reach, or use a scenario that does not need local benchmark " + + "servers such as webfinger.", ); return void exit(2); } @@ -235,7 +287,11 @@ export default async function runBench( let fleet: SyntheticServer | undefined; const startedAt = new Date().toISOString(); try { - if (suite.scenarios.some((s) => SIGNED_TYPES.has(s.type))) { + if ( + suite.scenarios.some((scenario) => + scenarioNeedsSyntheticServer(scenario, suite.scenarios) + ) + ) { fleet = await spawnSyntheticServer(await buildFleet(suite.actors), { advertiseHost: command.advertiseHost, }); @@ -246,14 +302,20 @@ export default async function runBench( log(`Running scenario "${scenario.name}" (${scenario.type})…`); const measurement = await runners[i].run({ scenario, + scenarios: suite.scenarios, target: suite.target, documentLoader, contextLoader, allowPrivateAddress, fleet: fleet ?? null, + advertiseHost: command.advertiseHost, fetch: fetchImpl, - assertDestinationAllowed: (url) => - assertDestinationAllowed(url, scenario), + assertDestinationAllowed: (url, gateScenario) => + assertDestinationAllowed(url, gateScenario ?? scenario), + assertReadDestinationAllowed: (url, gateScenario) => + assertReadDestinationAllowed(url, gateScenario ?? scenario), + assertActorlessDestinationAllowed: (url, gateScenario) => + assertActorlessDestinationAllowed(url, gateScenario ?? scenario), }); results.push(buildScenarioResult(scenario, measurement)); } @@ -345,10 +407,15 @@ interface DryRunPlanContext { readonly documentLoader: DocumentLoader; readonly contextLoader: DocumentLoader; readonly allowPrivateAddress: boolean; + readonly fetch: typeof fetch; readonly assertDestinationAllowed: ( url: URL, scenario: ResolvedScenario, ) => Promise; + readonly assertReadDestinationAllowed: ( + url: URL, + scenario: ResolvedScenario, + ) => Promise; } async function renderPlan( @@ -392,6 +459,12 @@ async function describeDiscoveryPlan( return await describeInboxDiscoveryPlan(scenario, context); case "webfinger": return describeWebFingerPlan(scenario, suite.target); + case "actor": + return await describeActorPlan(scenario, suite, context); + case "object": + return await describeObjectPlan(scenario, suite, context); + case "mixed": + return describeMixedPlan(scenario); default: return [" discovery: not available for this scenario type"]; } @@ -447,13 +520,83 @@ function describeWebFingerPlan( }); } +async function describeActorPlan( + scenario: ResolvedScenario, + suite: ResolvedSuite, + context: DryRunPlanContext, +): Promise { + try { + const urls = await actorUrlsFromRecipients(scenario.recipients, { + target: suite.target, + fetch: context.fetch, + }); + const lines: string[] = []; + for (const url of urls) { + lines.push(` actor: GET ${url.href}`); + lines.push( + ` destination safety: ${await describeDestinationSafety( + url, + scenario, + context, + )}`, + ); + } + return lines; + } catch (error) { + return [` actor discovery failed (${describeError(error)})`]; + } +} + +async function describeObjectPlan( + scenario: ResolvedScenario, + suite: ResolvedSuite, + context: DryRunPlanContext, +): Promise { + try { + const urls = await objectUrlsFromSource({ + source: scenario.source, + target: suite.target, + fetch: context.fetch, + assertReadDestinationAllowed: (url) => + context.assertReadDestinationAllowed(url, scenario), + }); + const lines = [` objects: ${urls.length} URL(s) resolved`]; + for (const url of urls.slice(0, 10)) { + lines.push(` object: GET ${url.href}`); + lines.push( + ` destination safety: ${await describeDestinationSafety( + url, + scenario, + context, + )}`, + ); + } + if (urls.length > 10) lines.push(` ... ${urls.length - 10} more`); + return lines; + } catch (error) { + return [` object discovery failed (${describeError(error)})`]; + } +} + +function describeMixedPlan(scenario: ResolvedScenario): string[] { + const entries = scenario.raw.mix ?? []; + if (entries.length < 1) return [" mix: no child scenarios"]; + return entries.map((entry) => + ` mix: ${entry.scenario} weight ${entry.weight}` + ); +} + async function describeDestinationSafety( - inbox: URL, + url: URL, scenario: ResolvedScenario, context: DryRunPlanContext, ): Promise { try { - await context.assertDestinationAllowed(inbox, scenario); + if (usesReadDestinationGate(scenario)) { + await context.assertReadDestinationAllowed(url, scenario); + } else { + await context.assertDestinationAllowed(url, scenario); + } return "allowed"; } catch (error) { if (error instanceof UnsafeTargetError) { @@ -463,6 +606,11 @@ async function describeDestinationSafety( } } +function usesReadDestinationGate(scenario: ResolvedScenario): boolean { + return (scenario.type === "actor" || scenario.type === "object") && + !scenario.authenticated; +} + interface PublicDestinationOverrideContext { readonly targetOrigin: string; readonly targetBenchmarkMode: boolean; @@ -522,3 +670,74 @@ function hasExplicitLoad(load: LoadConfig | undefined): boolean { (("rate" in load && load.rate != null) || ("concurrency" in load && load.concurrency != null)); } + +function scenarioNeedsSyntheticServer( + scenario: ResolvedScenario, + scenarios: readonly ResolvedScenario[], + seen: ReadonlySet = new Set(), +): boolean { + if (seen.has(scenario.name)) return false; + const nextSeen = new Set(seen).add(scenario.name); + switch (scenario.type) { + case "inbox": + return true; + case "actor": + case "object": + return scenario.authenticated; + case "failure": + return failureFaultsOf(scenario).some(isInboundFailureFault); + case "mixed": + return mixedChildrenOf(scenario, scenarios).some((child) => + scenarioNeedsSyntheticServer(child, scenarios, nextSeen) + ); + default: + return false; + } +} + +function scenarioNeedsReachableLocalServer( + scenario: ResolvedScenario, + scenarios: readonly ResolvedScenario[], + seen: ReadonlySet = new Set(), +): boolean { + if (scenario.type === "fanout") return scenario.raw.sinkBase == null; + if (scenario.type === "failure") { + const faults = failureFaultsOf(scenario); + return faults.includes("invalid-signature") || + (scenario.raw.sinkBase == null && + faults.some(isRemoteFailureFault)); + } + if (scenario.type === "mixed") { + if (seen.has(scenario.name)) return false; + const nextSeen = new Set(seen).add(scenario.name); + return mixedChildrenOf(scenario, scenarios).some((child) => + scenarioNeedsReachableLocalServer(child, scenarios, nextSeen) + ); + } + return scenarioNeedsSyntheticServer(scenario, scenarios, seen); +} + +function failureFaultsOf(scenario: ResolvedScenario): readonly string[] { + return scenario.faults.length < 1 ? ["remote-404"] : scenario.faults; +} + +function mixedChildrenOf( + scenario: ResolvedScenario, + scenarios: readonly ResolvedScenario[], +): readonly ResolvedScenario[] { + return (scenario.raw.mix ?? []).flatMap((entry) => { + const child = scenarios.find((candidate) => + candidate.name === entry.scenario + ); + return child == null ? [] : [child]; + }); +} + +function isInboundFailureFault(fault: string): boolean { + return fault === "invalid-signature" || fault === "missing-actor"; +} + +function isRemoteFailureFault(fault: string): boolean { + return fault === "remote-404" || fault === "remote-410" || + fault === "slow-inbox" || fault === "network-error"; +} diff --git a/packages/cli/src/bench/command.ts b/packages/cli/src/bench/command.ts index 553ecc39c..1104753b3 100644 --- a/packages/cli/src/bench/command.ts +++ b/packages/cli/src/bench/command.ts @@ -98,9 +98,10 @@ host in the actor and key URLs the target dereferences.`, description: message`Run an ActivityPub-specific load benchmark against a \ cooperative Fedify target running in benchmark mode. -The suite file declares the target, actors, and scenarios. Only the \`inbox\` \ -and \`webfinger\` scenario types are executed in this version; the format \ -itself can express every scenario type.`, +The suite file declares the target, actors, and scenarios. This version \ +executes the \`inbox\`, \`webfinger\`, \`actor\`, \`object\`, \`fanout\`, \ +\`failure\`, and \`mixed\` scenario types; \`collection\` remains reserved by \ +the suite format.`, }, ); diff --git a/packages/cli/src/bench/metrics/stats-client.test.ts b/packages/cli/src/bench/metrics/stats-client.test.ts index bf6958350..f337cede5 100644 --- a/packages/cli/src/bench/metrics/stats-client.test.ts +++ b/packages/cli/src/bench/metrics/stats-client.test.ts @@ -6,6 +6,7 @@ import { fetchServerSnapshot, parseServerMetrics, parseServerSnapshot, + queueTaskRemaining, type ServerSnapshot, snapshotToMetrics, } from "./stats-client.ts"; @@ -142,6 +143,43 @@ test("parseServerSnapshot - extracts raw histogram and queue depth", () => { assert.strictEqual(snap?.queueDepthMax, 7); }); +test("parseServerSnapshot - extracts permanent delivery failures", () => { + const snap = parseServerSnapshot({ + scopeMetrics: [{ + metrics: [{ + name: "activitypub.delivery.permanent_failure", + dataPointType: "sum", + dataPoints: [{ value: 3 }, { value: 2 }], + }], + }], + }); + assert.strictEqual(snap?.deliveryPermanentFailures, 5); +}); + +test("parseServerSnapshot - skips malformed sum data points", () => { + const snap = parseServerSnapshot({ + scopeMetrics: [{ + metrics: [ + { + name: "fedify.queue.task.enqueued", + dataPointType: "sum", + dataPoints: [null, { value: 5 }], + }, + { + name: "fedify.queue.task.completed", + dataPointType: "sum", + dataPoints: [{ value: 3 }], + }, + ], + }], + }); + assert.deepEqual(snap?.queueTasks, { + enqueued: 5, + completed: 3, + failed: 0, + }); +}); + test("parseServerSnapshot - empty (non-null) when no relevant instruments", () => { // A parseable-but-empty snapshot yields an empty snapshot, not null, so a // successful baseline fetch is distinguishable from an unavailable one. @@ -155,15 +193,18 @@ test("diffSnapshots - subtracts the baseline bucket counts", () => { const baseline: ServerSnapshot = { signature: { boundaries: [5, 10, 25], counts: [4, 6, 10, 0] }, queueDepthMax: 2, + deliveryPermanentFailures: 2, }; const end: ServerSnapshot = { signature: { boundaries: [5, 10, 25], counts: [10, 16, 30, 4] }, queueDepthMax: 9, + deliveryPermanentFailures: 5, }; const diff = diffSnapshots(baseline, end); assert.deepEqual(diff?.signature?.counts, [6, 10, 20, 4]); // The queue depth is a gauge, so the end value is kept (not subtracted). assert.strictEqual(diff?.queueDepthMax, 9); + assert.strictEqual(diff?.deliveryPermanentFailures, 3); }); test("diffSnapshots - an empty baseline keeps the full end histogram", () => { @@ -179,6 +220,28 @@ test("diffSnapshots - an empty baseline keeps the full end histogram", () => { assert.strictEqual(diff.queueDepthMax, 4); }); +test("queueTaskRemaining - accounts for baseline backlog", () => { + const diff: ServerSnapshot = { + signature: null, + queueDepthMax: null, + queueTasks: { + enqueued: 1, + completed: 1, + failed: 0, + }, + }; + + assert.strictEqual(queueTaskRemaining(diff), 0); + assert.strictEqual(queueTaskRemaining(diff, 1), 1); + assert.strictEqual( + queueTaskRemaining({ + ...diff, + queueTasks: { enqueued: 1, completed: 2, failed: 0 }, + }, 1), + 0, + ); +}); + test("diffSnapshots - incompatible bucketing drops the signature histogram", () => { // Same length but different boundary values is not comparable; refuse to // subtract rather than misattribute counts. diff --git a/packages/cli/src/bench/metrics/stats-client.ts b/packages/cli/src/bench/metrics/stats-client.ts index 2280aa202..e2590d028 100644 --- a/packages/cli/src/bench/metrics/stats-client.ts +++ b/packages/cli/src/bench/metrics/stats-client.ts @@ -57,6 +57,17 @@ export interface ServerSnapshot { readonly signature: ServerHistogram | null; /** The maximum observed queue depth, or `null` if absent. */ readonly queueDepthMax: number | null; + /** Queue task counters, or `null` if absent. */ + readonly queueTasks?: QueueTaskCounts | null; + /** Permanent outbound delivery failure count, or `null` if absent. */ + readonly deliveryPermanentFailures?: number | null; +} + +/** Queue task counts extracted from benchmark stats. */ +export interface QueueTaskCounts { + readonly enqueued: number; + readonly completed: number; + readonly failed: number; } /** @@ -86,7 +97,20 @@ export function parseServerSnapshot(snapshot: unknown): ServerSnapshot | null { if (values.length > 0) queueDepthMax = Math.max(...values); } - return { signature, queueDepthMax }; + const queueTasks = parseQueueTasks(metrics); + const deliveryPermanentFailures = sumMetric( + metrics, + "activitypub.delivery.permanent_failure", + ); + + return { + signature, + queueDepthMax, + ...(queueTasks == null ? {} : { queueTasks }), + ...(deliveryPermanentFailures == null + ? {} + : { deliveryPermanentFailures }), + }; } catch { return null; } @@ -107,9 +131,19 @@ export function diffSnapshots( baseline: ServerSnapshot, end: ServerSnapshot, ): ServerSnapshot { + const queueTasks = diffQueueTasks( + baseline.queueTasks ?? null, + end.queueTasks ?? null, + ); + const deliveryPermanentFailures = diffCounter( + baseline.deliveryPermanentFailures ?? null, + end.deliveryPermanentFailures ?? null, + ); return { signature: diffHistogram(baseline.signature, end.signature), queueDepthMax: end.queueDepthMax, + ...(queueTasks == null ? {} : { queueTasks }), + ...(deliveryPermanentFailures == null ? {} : { deliveryPermanentFailures }), }; } @@ -194,6 +228,25 @@ export async function fetchServerMetrics( return snapshotToMetrics(await fetchServerSnapshot(target, fetchImpl)); } +/** + * Returns the remaining queue task backlog represented by a diffed snapshot. + * @param snapshot The server snapshot to inspect, usually already diffed + * against a baseline. + * @param baselineRemaining Queue tasks that were already outstanding when the + * diff baseline was taken. These must drain before diffed completions can be + * attributed to newly enqueued tasks. + * @returns `Math.max(0, baselineRemaining + enqueued - completed - failed)`, + * or `null` when the snapshot has no queue task counters. + */ +export function queueTaskRemaining( + snapshot: ServerSnapshot | null, + baselineRemaining = 0, +): number | null { + if (snapshot?.queueTasks == null) return null; + const { enqueued, completed, failed } = snapshot.queueTasks; + return Math.max(0, baselineRemaining + enqueued - completed - failed); +} + function isFiniteNumber(value: unknown): value is number { return typeof value === "number" && Number.isFinite(value); } @@ -238,6 +291,37 @@ function mergeHistogram( return boundaries != null && counts != null ? { boundaries, counts } : null; } +function parseQueueTasks( + metrics: readonly SnapshotMetric[], +): QueueTaskCounts | null { + const enqueued = sumMetric(metrics, "fedify.queue.task.enqueued"); + const completed = sumMetric(metrics, "fedify.queue.task.completed"); + const failed = sumMetric(metrics, "fedify.queue.task.failed"); + return enqueued == null && completed == null && failed == null ? null : { + enqueued: enqueued ?? 0, + completed: completed ?? 0, + failed: failed ?? 0, + }; +} + +function sumMetric( + metrics: readonly SnapshotMetric[], + name: string, +): number | null { + let total = 0; + let found = false; + for (const metric of metrics) { + if (metric.name !== name || !Array.isArray(metric.dataPoints)) continue; + for (const point of metric.dataPoints) { + if (isRecord(point) && isFiniteNumber(point["value"])) { + total += point["value"]; + found = true; + } + } + } + return found ? total : null; +} + function diffHistogram( baseline: ServerHistogram | null, end: ServerHistogram | null, @@ -256,6 +340,28 @@ function diffHistogram( return { boundaries: end.boundaries, counts }; } +function diffQueueTasks( + baseline: QueueTaskCounts | null, + end: QueueTaskCounts | null, +): QueueTaskCounts | null { + if (end == null) return null; + if (baseline == null) return end; + return { + enqueued: Math.max(0, end.enqueued - baseline.enqueued), + completed: Math.max(0, end.completed - baseline.completed), + failed: Math.max(0, end.failed - baseline.failed), + }; +} + +function diffCounter( + baseline: number | null, + end: number | null, +): number | null { + if (end == null) return null; + if (baseline == null) return end; + return Math.max(0, end - baseline); +} + function histogramsCompatible( a: ServerHistogram, b: ServerHistogram, @@ -265,6 +371,10 @@ function histogramsCompatible( a.boundaries.every((boundary, i) => boundary === b.boundaries[i]); } +function isRecord(value: unknown): value is Record { + return value != null && typeof value === "object" && !Array.isArray(value); +} + function histogramPercentile(histogram: ServerHistogram, p: number): number { const { boundaries, counts } = histogram; const total = counts.reduce((sum, n) => sum + n, 0); diff --git a/packages/cli/src/bench/render/markdown.ts b/packages/cli/src/bench/render/markdown.ts index 511f34b26..b0e31ab2e 100644 --- a/packages/cli/src/bench/render/markdown.ts +++ b/packages/cli/src/bench/render/markdown.ts @@ -54,6 +54,13 @@ function renderScenario(scenario: ScenarioResult): string[] { lines.push(`| Requests | ${formatNumber(r.total)} |`); lines.push(`| Success rate | ${formatPercent(r.successRate)} |`); lines.push(`| Throughput | ${formatNumber(scenario.throughputPerSec)}/s |`); + if (scenario.deliveryThroughputPerSec != null) { + lines.push( + `| Delivery throughput | ${ + formatNumber(scenario.deliveryThroughputPerSec) + }/s |`, + ); + } const l = scenario.client.latencyMs; lines.push(`| Latency p50 | ${formatNumber(l.p50)}ms |`); lines.push(`| Latency p95 | ${formatNumber(l.p95)}ms |`); diff --git a/packages/cli/src/bench/render/render.test.ts b/packages/cli/src/bench/render/render.test.ts index 4c6688a61..d96b732bc 100644 --- a/packages/cli/src/bench/render/render.test.ts +++ b/packages/cli/src/bench/render/render.test.ts @@ -5,7 +5,7 @@ import { dirname, join } from "node:path"; import test from "node:test"; import { fileURLToPath } from "node:url"; import type { BenchReport } from "../result/model.ts"; -import { reportSchemaV1 } from "../result/schema.ts"; +import { reportSchemaV2 } from "../result/schema.ts"; import { renderReport } from "./index.ts"; // `import.meta.dirname` needs Node >= 20.11; derive it from the URL instead. @@ -21,7 +21,7 @@ test("renderReport json - valid JSON that validates against the schema", () => { const json = renderReport(report, "json"); const parsed = JSON.parse(json); const validator = new Validator( - reportSchemaV1 as unknown as Schema, + reportSchemaV2 as unknown as Schema, "2020-12", ); assert.ok(validator.validate(parsed).valid); @@ -74,6 +74,23 @@ test("renderReport - shows queue depth even without drain latency", () => { assert.match(md, /Queue depth max \(server\) \| 42/); }); +test("renderReport - shows delivery throughput when present", () => { + const base = report.scenarios[0]; + const r: BenchReport = { + ...report, + scenarios: [{ + ...base, + deliveryThroughputPerSec: 123, + }], + }; + const json = JSON.parse(renderReport(r, "json")); + assert.strictEqual(json.scenarios[0].deliveryThroughputPerSec, 123); + const text = renderReport(r, "text"); + assert.match(text, /Delivery throughput: 123 deliveries\/s/); + const md = renderReport(r, "markdown"); + assert.match(md, /Delivery throughput \| 123\/s/); +}); + test("renderReport - empty drain latency falls back to the depth line", () => { // An empty drainMs object carries no percentile, so neither form should print // a meaningless drain line; both still surface the depth (here zero). diff --git a/packages/cli/src/bench/render/text.ts b/packages/cli/src/bench/render/text.ts index da13f30c1..8b40f8233 100644 --- a/packages/cli/src/bench/render/text.ts +++ b/packages/cli/src/bench/render/text.ts @@ -64,6 +64,13 @@ function renderScenario(scenario: ScenarioResult): string[] { })`, ); lines.push(` Throughput: ${formatNumber(scenario.throughputPerSec)} req/s`); + if (scenario.deliveryThroughputPerSec != null) { + lines.push( + ` Delivery throughput: ${ + formatNumber(scenario.deliveryThroughputPerSec) + } deliveries/s`, + ); + } const l = scenario.client.latencyMs; lines.push( ` Client latency (ms): p50 ${formatNumber(l.p50)} p95 ${ diff --git a/packages/cli/src/bench/result/build.test.ts b/packages/cli/src/bench/result/build.test.ts index 19b14b9be..4a268dc57 100644 --- a/packages/cli/src/bench/result/build.test.ts +++ b/packages/cli/src/bench/result/build.test.ts @@ -9,7 +9,7 @@ import { detectEnvironment, type ScenarioMeasurement, } from "./build.ts"; -import { reportSchemaV1 } from "./schema.ts"; +import { reportSchemaV2 } from "./schema.ts"; function resolvedInbox() { return normalizeSuite({ @@ -62,6 +62,14 @@ test("buildScenarioResult - a run that measured nothing never passes", () => { assert.strictEqual(result.passed, false); }); +test("buildScenarioResult - preserves delivery throughput", () => { + const result = buildScenarioResult(resolvedInbox(), { + ...measurement(), + deliveryThroughputPerSec: 42, + }); + assert.strictEqual(result.deliveryThroughputPerSec, 42); +}); + test("buildReport - gate passes only when all scenarios pass", () => { const ok = buildScenarioResult(resolvedInbox(), measurement()); const bad = buildScenarioResult(resolvedInbox(), { @@ -93,7 +101,7 @@ test("buildReport - output validates against the report schema", () => { suite: { name: "suite", configHash: configHash({ a: 1 }) }, }); const validator = new Validator( - reportSchemaV1 as unknown as Schema, + reportSchemaV2 as unknown as Schema, "2020-12", ); const result = validator.validate(JSON.parse(JSON.stringify(report))); diff --git a/packages/cli/src/bench/result/build.ts b/packages/cli/src/bench/result/build.ts index 37f9d0eed..9e04d73b3 100644 --- a/packages/cli/src/bench/result/build.ts +++ b/packages/cli/src/bench/result/build.ts @@ -32,6 +32,7 @@ import type { export interface ScenarioMeasurement { readonly requests: RequestSummary; readonly throughputPerSec: number; + readonly deliveryThroughputPerSec?: number; readonly client: ClientMetrics; readonly server: ServerMetrics | null; readonly errors: ErrorBucket[]; @@ -60,6 +61,9 @@ export function buildScenarioResult( load: loadSummary(scenario), requests: measurement.requests, throughputPerSec: measurement.throughputPerSec, + ...(measurement.deliveryThroughputPerSec == null ? {} : { + deliveryThroughputPerSec: measurement.deliveryThroughputPerSec, + }), client: measurement.client, server: measurement.server, errors: measurement.errors, @@ -88,7 +92,7 @@ export interface ReportInput { export function buildReport(input: ReportInput): BenchReport { return { $schema: REPORT_SCHEMA_ID, - schemaVersion: 1, + schemaVersion: 2, tool: { name: "@fedify/cli", version: metadata.version }, environment: input.environment, target: input.target, diff --git a/packages/cli/src/bench/result/expect/evaluate.test.ts b/packages/cli/src/bench/result/expect/evaluate.test.ts index cdbfc398b..b28a8eabc 100644 --- a/packages/cli/src/bench/result/expect/evaluate.test.ts +++ b/packages/cli/src/bench/result/expect/evaluate.test.ts @@ -84,11 +84,22 @@ test("evaluateExpect - missing server metric fails (actual null)", () => { assert.strictEqual(passed, false); }); -test("evaluateExpect - unmeasured metric yields null actual and fails", () => { - const { results } = evaluateExpect( +test("evaluateExpect - reads delivery throughput", () => { + const { passed, results } = evaluateExpect( { deliveryThroughput: ">= 1/s" }, - metrics(), + metrics({ deliveryThroughputPerSec: 12 }), ); + assert.strictEqual(passed, true); + assert.strictEqual(results[0].actual, 12); + assert.strictEqual(results[0].pass, true); +}); + +test("evaluateExpect - does not alias delivery throughput to request throughput", () => { + const { passed, results } = evaluateExpect( + { deliveryThroughput: ">= 1/s" }, + metrics({ throughputPerSec: 12 }), + ); + assert.strictEqual(passed, false); assert.strictEqual(results[0].actual, null); assert.strictEqual(results[0].pass, false); }); diff --git a/packages/cli/src/bench/result/expect/evaluate.ts b/packages/cli/src/bench/result/expect/evaluate.ts index 092226468..174137859 100644 --- a/packages/cli/src/bench/result/expect/evaluate.ts +++ b/packages/cli/src/bench/result/expect/evaluate.ts @@ -50,10 +50,14 @@ export function validateExpectBlock(expect: ExpectBlock): void { } /** The subset of a scenario result that `expect` metrics are looked up from. */ -export type MetricView = Pick< - ScenarioResult, - "requests" | "throughputPerSec" | "client" | "server" | "errors" ->; +export interface MetricView { + readonly requests: ScenarioResult["requests"]; + readonly throughputPerSec: ScenarioResult["throughputPerSec"]; + readonly deliveryThroughputPerSec?: number; + readonly client: ScenarioResult["client"]; + readonly server: ScenarioResult["server"]; + readonly errors: ScenarioResult["errors"]; +} /** The outcome of evaluating an `expect` block. */ export interface ExpectEvaluation { @@ -133,8 +137,7 @@ function lookupValue(metrics: MetricView, metric: string): number | null { case "throughputPerSec": return metrics.throughputPerSec; case "deliveryThroughput": - // Recognized (fanout/mixed) but not measured by the runners yet. - return null; + return metrics.deliveryThroughputPerSec ?? null; case "errors.total": return sumErrors(metrics.errors); case "errors.4xx": diff --git a/packages/cli/src/bench/result/model.ts b/packages/cli/src/bench/result/model.ts index 44b885992..bbdf2d8bc 100644 --- a/packages/cli/src/bench/result/model.ts +++ b/packages/cli/src/bench/result/model.ts @@ -143,6 +143,7 @@ export interface ScenarioResult { readonly load: LoadSummary; readonly requests: RequestSummary; readonly throughputPerSec: number; + readonly deliveryThroughputPerSec?: number; readonly client: ClientMetrics; readonly server: ServerMetrics | null; readonly errors: ErrorBucket[]; @@ -156,7 +157,7 @@ export interface ScenarioResult { export interface BenchReport { /** The published report schema URL. */ readonly $schema?: string; - readonly schemaVersion: 1; + readonly schemaVersion: 2; readonly tool: { readonly name: string; readonly version: string }; readonly environment: Environment; readonly target: TargetInfo; diff --git a/packages/cli/src/bench/result/schema.ts b/packages/cli/src/bench/result/schema.ts index b465bf88f..787e4b63c 100644 --- a/packages/cli/src/bench/result/schema.ts +++ b/packages/cli/src/bench/result/schema.ts @@ -2,20 +2,24 @@ * The embedded JSON Schema (draft 2020-12) for benchmark report output. * * Like the scenario schema, this object is the runtime copy and is published, - * byte-for-byte, as *schema/bench/report-v1.json*; a drift guard keeps the two - * in sync. The matching TypeScript types live in {@link ./model.ts}. + * byte-for-byte, as files under *schema/bench/*; a drift guard keeps the two in + * sync. The matching TypeScript types live in {@link ./model.ts}. * @since 2.3.0 * @module */ /** The hosted URL that serves the report schema. */ export const REPORT_SCHEMA_ID = + "https://json-schema.fedify.dev/bench/report-v2.json"; + +/** The hosted URL for the original report schema. */ +export const REPORT_SCHEMA_V1_ID = "https://json-schema.fedify.dev/bench/report-v1.json"; /** The benchmark report JSON Schema (draft 2020-12). */ export const reportSchemaV1 = { $schema: "https://json-schema.org/draft/2020-12/schema", - $id: REPORT_SCHEMA_ID, + $id: REPORT_SCHEMA_V1_ID, title: "Fedify benchmark report", type: "object", additionalProperties: false, @@ -284,3 +288,23 @@ export const reportSchemaV1 = { }, }, } as const; + +/** The benchmark report JSON Schema (draft 2020-12). */ +export const reportSchemaV2 = { + ...reportSchemaV1, + $id: REPORT_SCHEMA_ID, + properties: { + ...reportSchemaV1.properties, + schemaVersion: { const: 2 }, + }, + $defs: { + ...reportSchemaV1.$defs, + scenarioResult: { + ...reportSchemaV1.$defs.scenarioResult, + properties: { + ...reportSchemaV1.$defs.scenarioResult.properties, + deliveryThroughputPerSec: { type: "number" }, + }, + }, + }, +} as const; diff --git a/packages/cli/src/bench/scenario/schema.ts b/packages/cli/src/bench/scenario/schema.ts index 83c4983b5..a6dc9387b 100644 --- a/packages/cli/src/bench/scenario/schema.ts +++ b/packages/cli/src/bench/scenario/schema.ts @@ -1,14 +1,14 @@ /** * The embedded JSON Schema (draft 2020-12) for benchmark scenario suite files. * - * This object is the runtime copy used by the validator; it is published, - * byte-for-byte, as *schema/bench/scenario-v1.json* and a drift guard keeps the - * two in sync. The matching TypeScript types live in {@link ./types.ts}. + * These objects are the runtime copies used by the validator; they are + * published, byte-for-byte, under *schema/bench/* and a drift guard keeps them + * in sync. The matching TypeScript types live in {@link ./types.ts}. * * The schema expresses every scenario type discussed for `fedify bench` * (`inbox`, `webfinger`, `actor`, `object`, `fanout`, `collection`, `failure`, - * `mixed`), even though only `inbox` and `webfinger` have runners in this - * version. Three cross-field rules are enforced here rather than in code: + * `mixed`). All but `collection` have runners in this version. Three + * cross-field rules are enforced here rather than in code: * * - exactly one HTTP request signature scheme per actor group * (`contains` + `minContains`/`maxContains`); @@ -19,8 +19,12 @@ * @module */ -/** The hosted URL that serves the scenario schema. */ +/** The hosted URL that serves the current scenario schema. */ export const SCENARIO_SCHEMA_ID = + "https://json-schema.fedify.dev/bench/scenario-v2.json"; + +/** The hosted URL that serves the version 1 scenario schema. */ +export const SCENARIO_SCHEMA_ID_V1 = "https://json-schema.fedify.dev/bench/scenario-v1.json"; const READ_METRICS = [ @@ -54,13 +58,17 @@ const FANOUT_METRICS = [ "queueDrain.p99", ]; -// A `mixed` scenario blends others, so it may assert any of their metrics. +// Schema v1 allowed mixed scenarios to assert child server metrics. Keep this +// list stable so the published v1 schema remains byte-for-byte immutable. const MIXED_METRICS = [...new Set([...INBOX_METRICS, ...FANOUT_METRICS])]; +// Schema v2 matches the runtime mixed runner, which merges only client-side +// request metrics and delivery throughput. +const MIXED_V2_METRICS = [...READ_METRICS, "deliveryThroughput"]; /** The benchmark scenario suite JSON Schema (draft 2020-12). */ export const scenarioSchemaV1 = { $schema: "https://json-schema.org/draft/2020-12/schema", - $id: SCENARIO_SCHEMA_ID, + $id: SCENARIO_SCHEMA_ID_V1, title: "Fedify benchmark scenario suite", type: "object", required: ["version", "scenarios"], @@ -377,3 +385,65 @@ export const scenarioSchemaV1 = { }, }, } as const; + +/** The current benchmark scenario suite JSON Schema (draft 2020-12). */ +export const scenarioSchemaV2 = { + ...scenarioSchemaV1, + $id: SCENARIO_SCHEMA_ID, + $defs: { + ...scenarioSchemaV1.$defs, + scenario: { + ...scenarioSchemaV1.$defs.scenario, + properties: { + ...scenarioSchemaV1.$defs.scenario.properties, + sinkBase: { type: "string" }, + }, + allOf: scenarioSchemaV1.$defs.scenario.allOf.map((condition) => + condition.if.properties.type.const === "failure" + ? { + if: condition.if, + then: { + properties: condition.then.properties, + }, + } + : condition.if.properties.type.const === "actor" || + condition.if.properties.type.const === "object" + ? { + if: condition.if, + then: { + required: condition.then.required, + allOf: [ + { + if: { + required: ["authenticated"], + properties: { authenticated: { const: true } }, + }, + then: { + properties: { + expect: { propertyNames: { enum: INBOX_METRICS } }, + }, + }, + else: { + properties: { + expect: { propertyNames: { enum: READ_METRICS } }, + }, + }, + }, + ], + }, + } + : condition.if.properties.type.const === "mixed" + ? { + if: condition.if, + then: { + required: condition.then.required, + properties: { + expect: { propertyNames: { enum: MIXED_V2_METRICS } }, + }, + }, + } + : condition + ), + }, + }, +} as const; diff --git a/packages/cli/src/bench/scenario/types.ts b/packages/cli/src/bench/scenario/types.ts index 18a295480..8bf1a4186 100644 --- a/packages/cli/src/bench/scenario/types.ts +++ b/packages/cli/src/bench/scenario/types.ts @@ -2,7 +2,7 @@ * Hand-written TypeScript types for the benchmark scenario suite format. * * These mirror the published JSON Schema in {@link ./schema.ts} and - * *schema/bench/scenario-v1.json*. Runtime validation is done with + * *schema/bench/scenario-v2.json*. Runtime validation is done with * `@cfworker/json-schema`; after a value validates, it is narrowed to * {@link Suite} with an `as unknown as` cast (see {@link ./validate.ts}). * @since 2.3.0 @@ -24,7 +24,7 @@ export const HTTP_SIGNATURE_STANDARDS: readonly SignatureStandard[] = [ "rfc9421", ]; -/** A scenario type. Only `inbox` and `webfinger` have runners so far. */ +/** A scenario type. `collection` is reserved but not executable so far. */ export type ScenarioType = | "inbox" | "webfinger" @@ -132,6 +132,7 @@ export interface Scenario { // fanout readonly sender?: string; readonly followers?: number; + readonly sinkBase?: string; readonly trigger?: Record; readonly sinkBehavior?: Record; readonly queueDrainTimeout?: string; diff --git a/packages/cli/src/bench/scenario/validate.test.ts b/packages/cli/src/bench/scenario/validate.test.ts index ebc938c92..b53ebe704 100644 --- a/packages/cli/src/bench/scenario/validate.test.ts +++ b/packages/cli/src/bench/scenario/validate.test.ts @@ -100,6 +100,63 @@ test("validateSuite - enforces per-type expect metric allowlist", () => { assert.throws(() => validateSuite(bad), SuiteValidationError); }); +test("validateSuite - rejects unsigned actor signature expectations", () => { + const bad = { + version: 1, + target: "http://localhost:3000", + scenarios: [{ + name: "actor", + type: "actor", + recipient: "http://localhost:3000/users/alice", + expect: { "signatureVerification.p95": "< 10ms" }, + }], + }; + assert.throws(() => validateSuite(bad), SuiteValidationError); +}); + +test("validateSuite - accepts authenticated actor signature expectations", () => { + const suite = { + version: 1, + target: "http://localhost:3000", + scenarios: [{ + name: "actor", + type: "actor", + recipient: "http://localhost:3000/users/alice", + authenticated: true, + expect: { "signatureVerification.p95": "< 10ms" }, + }], + }; + assert.doesNotThrow(() => validateSuite(suite)); +}); + +test("validateSuite - rejects mixed server-side expect metrics", () => { + const bad = { + version: 1, + target: "http://localhost:3000", + scenarios: [{ + name: "blend", + type: "mixed", + mix: [{ scenario: "fanout", weight: 1 }], + expect: { "queueDrain.p95": "< 10ms" }, + }], + }; + assert.throws(() => validateSuite(bad), SuiteValidationError); +}); + +test("validateSuite - accepts mixed delivery throughput expectations", () => { + const suite = { + version: 1, + target: "http://localhost:3000", + scenarios: [{ + name: "blend", + type: "mixed", + mix: [{ scenario: "fanout", weight: 1 }], + expect: { deliveryThroughput: ">= 1/s" }, + }], + }; + assert.doesNotThrow(() => validateSuite(suite)); +}); + test("validateSuite - error message names the failing location", () => { try { validateSuite({ target: "http://localhost:3000", scenarios: [] }); diff --git a/packages/cli/src/bench/scenario/validate.ts b/packages/cli/src/bench/scenario/validate.ts index d160bec81..02dbd6f93 100644 --- a/packages/cli/src/bench/scenario/validate.ts +++ b/packages/cli/src/bench/scenario/validate.ts @@ -5,14 +5,14 @@ */ import { type Schema, Validator } from "@cfworker/json-schema"; -import { scenarioSchemaV1 } from "./schema.ts"; +import { scenarioSchemaV2 } from "./schema.ts"; import { type RawValidationError, SuiteValidationError } from "./errors.ts"; import type { Suite } from "./types.ts"; let validator: Validator | undefined; function getValidator(): Validator { - validator ??= new Validator(scenarioSchemaV1 as unknown as Schema, "2020-12"); + validator ??= new Validator(scenarioSchemaV2 as unknown as Schema, "2020-12"); return validator; } diff --git a/packages/cli/src/bench/scenarios/actor.test.ts b/packages/cli/src/bench/scenarios/actor.test.ts new file mode 100644 index 000000000..bcdf8f15b --- /dev/null +++ b/packages/cli/src/bench/scenarios/actor.test.ts @@ -0,0 +1,156 @@ +import { + createFederation, + generateCryptoKeyPair, + MemoryKvStore, +} from "@fedify/fedify"; +import { Create, Endpoints, Person } from "@fedify/vocab"; +import assert from "node:assert/strict"; +import test from "node:test"; +import { serve } from "srvx"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import { buildFleet } from "../actor/fleet.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import type { Suite } from "../scenario/types.ts"; +import { spawnSyntheticServer } from "../server/synthetic.ts"; +import { actorRunner } from "./actor.ts"; + +async function spawnActorTarget() { + const federation = createFederation({ + kv: new MemoryKvStore(), + benchmarkMode: true, + }); + const keyPairs = Promise.all([ + generateCryptoKeyPair("RSASSA-PKCS1-v1_5"), + generateCryptoKeyPair("Ed25519"), + ]); + federation + .setActorDispatcher("/users/{identifier}", async (ctx, identifier) => { + if (identifier !== "alice") return null; + const pairs = await ctx.getActorKeyPairs(identifier); + return new Person({ + id: ctx.getActorUri(identifier), + preferredUsername: identifier, + inbox: ctx.getInboxUri(identifier), + endpoints: new Endpoints({ sharedInbox: ctx.getInboxUri() }), + publicKey: pairs[0]?.cryptographicKey, + assertionMethods: pairs.map((p) => p.multikey), + }); + }) + .mapHandle((_ctx, username) => (username === "alice" ? "alice" : null)) + .setKeyPairsDispatcher(async () => await keyPairs); + federation.setInboxListeners("/users/{identifier}/inbox", "/inbox").on( + Create, + () => {}, + ); + + let actorGets = 0; + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request) { + if (new URL(request.url).pathname === "/users/alice") actorGets++; + return federation.fetch(request, { contextData: undefined }); + }, + }); + await server.ready(); + return { + url: new URL(server.url!), + actorGets: () => actorGets, + close: () => server.close(true), + }; +} + +test("actorRunner - fetches actor documents", async () => { + const target = await spawnActorTarget(); + try { + const suite: Suite = { + version: 1, + target: target.url.href, + scenarios: [{ + name: "actor", + type: "actor", + recipient: new URL("/users/alice", target.url).href, + load: { concurrency: 2 }, + duration: "80ms", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await actorRunner.run({ + scenario, + target: target.url, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(target.actorGets() > 0); + } finally { + await target.close(); + } +}); + +test("actorRunner - signs authenticated actor fetches", async () => { + const target = await spawnActorTarget(); + let fleet: Awaited> | undefined; + try { + fleet = await spawnSyntheticServer( + await buildFleet([{ + count: 1, + signatureStandards: ["draft-cavage-http-signatures-12"], + }]), + ); + const suite: Suite = { + version: 1, + target: target.url.href, + scenarios: [{ + name: "actor-auth", + type: "actor", + recipient: new URL("/users/alice", target.url).href, + authenticated: true, + load: { concurrency: 1 }, + duration: "80ms", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await actorRunner.run({ + scenario, + target: target.url, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + } finally { + try { + await fleet?.close(); + } finally { + await target.close(); + } + } +}); + +test("actorRunner.validate - rejects non-http actor recipient URLs", () => { + for (const recipient of ["ftp://target.test/users/alice", "mailto:alice"]) { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "actor", + type: "actor", + recipient, + }], + }).scenarios[0]; + + assert.throws( + () => actorRunner.validate?.(scenario), + /actor recipient must be an acct: handle or a bare http\(s\) URL/, + ); + } +}); diff --git a/packages/cli/src/bench/scenarios/actor.ts b/packages/cli/src/bench/scenarios/actor.ts new file mode 100644 index 000000000..f1e6ed4a5 --- /dev/null +++ b/packages/cli/src/bench/scenarios/actor.ts @@ -0,0 +1,53 @@ +/** + * The `actor` scenario runner. + * @since 2.3.0 + * @module + */ + +import { convertUrlIfHandle } from "../../webfinger/lib.ts"; +import { actorUrlsFromRecipients } from "./object-discovery.ts"; +import { runReadLoad } from "./read.ts"; +import { + isBareHttpUrl, + type RunContext, + type ScenarioRunner, +} from "./runner.ts"; + +/** The `actor` scenario runner. */ +export const actorRunner: ScenarioRunner = { + validate(scenario): void { + if (scenario.recipients.length < 1) { + throw new Error("The actor scenario requires a recipient."); + } + for (const recipient of scenario.recipients) { + let url: URL; + try { + url = convertUrlIfHandle(recipient); + } catch { + throw new Error( + `Scenario "${scenario.name}": invalid actor recipient ` + + `${JSON.stringify(recipient)}.`, + ); + } + if (url.protocol !== "acct:" && !isBareHttpUrl(url)) { + throw new Error( + `Scenario "${scenario.name}": actor recipient must be an acct: ` + + `handle or a bare http(s) URL with a host and no credentials; ` + + `got ${JSON.stringify(url.href)}.`, + ); + } + } + }, + + async run(context: RunContext) { + this.validate?.(context.scenario); + const urls = await actorUrlsFromRecipients(context.scenario.recipients, { + target: context.target, + fetch: context.fetch, + }); + return await runReadLoad(context, { + urls, + authenticated: context.scenario.authenticated, + }); + }, +}; diff --git a/packages/cli/src/bench/scenarios/failure.test.ts b/packages/cli/src/bench/scenarios/failure.test.ts new file mode 100644 index 000000000..a41984c5f --- /dev/null +++ b/packages/cli/src/bench/scenarios/failure.test.ts @@ -0,0 +1,700 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { serve } from "srvx"; +import { buildFleet } from "../actor/fleet.ts"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import type { Clock } from "../load/clock.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import type { Suite } from "../scenario/types.ts"; +import { spawnSyntheticServer } from "../server/synthetic.ts"; +import { failureRunner } from "./failure.ts"; + +for ( + const [fault, outcome] of [ + ["remote-404", "permanent"], + ["remote-410", "permanent"], + ["slow-inbox", "completed"], + ["network-error", "failed"], + ] as const +) { + test(`failureRunner - drives ${fault} through the target`, async () => { + const target = new URL("http://target.test/"); + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault, + sender: "alice", + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "1s", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + let triggerCalls = 0; + let triggerRecipientCount = 0; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(statsJson(statsSnapshot({ + enqueued: triggerCalls, + completed: outcome === "failed" ? 0 : triggerCalls, + failed: outcome === "failed" ? triggerCalls : 0, + permanentFailures: outcome === "permanent" ? triggerCalls : 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + const body = JSON.parse(String(init?.body)); + triggerRecipientCount = body.recipients.length; + assert.deepStrictEqual(body.sender, { identifier: "alice" }); + return Promise.resolve(statsJson({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.failed, 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(triggerCalls > 0); + assert.strictEqual(triggerRecipientCount, 1); + }); +} + +test("failureRunner - uses configured sink base for remote faults", async () => { + const target = new URL("http://target.test/"); + const sinkBase = `http://127.0.0.1:${await reservePort()}/`; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault: "remote-404", + sender: "alice", + sinkBase, + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + let triggerCalls = 0; + let recipientInbox = ""; + + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(statsJson(statsSnapshot({ + enqueued: triggerCalls, + completed: triggerCalls, + failed: 0, + permanentFailures: triggerCalls, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + const body = JSON.parse(String(init?.body)); + recipientInbox = body.recipients[0].inbox; + return Promise.resolve(statsJson({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(recipientInbox, new URL("/inbox/0", sinkBase).href); +}); + +test("failureRunner - gates remote fault sinks before triggering", async () => { + const target = new URL("http://target.test/"); + const sinkBase = `http://127.0.0.1:${await reservePort()}/`; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault: "remote-404", + sender: "alice", + sinkBase, + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + let gateCalls = 0; + let triggerCalls = 0; + + await assert.rejects( + async () => + failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertActorlessDestinationAllowed: (url) => { + gateCalls++; + throw new Error(`refused ${url.href}`); + }, + assertDestinationAllowed: () => {}, + }), + /refused http:\/\/127\.0\.0\.1:/, + ); + + assert.strictEqual(gateCalls, 1); + assert.strictEqual(triggerCalls, 0); +}); + +test("failureRunner - shares sink base across remote fault mix", async () => { + const target = new URL("http://target.test/"); + const sinkBase = `http://127.0.0.1:${await reservePort()}/`; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault: ["remote-404", "remote-410"], + sender: "alice", + sinkBase, + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + let triggerCalls = 0; + const recipientInboxes: string[] = []; + + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(statsJson(statsSnapshot({ + enqueued: triggerCalls, + completed: triggerCalls, + failed: 0, + permanentFailures: triggerCalls, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + const body = JSON.parse(String(init?.body)); + recipientInboxes.push(body.recipients[0].inbox); + return Promise.resolve(statsJson({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 1); + assert.strictEqual(measurement.requests.failed, 0); + assert.ok(recipientInboxes.includes(new URL("/inbox/0", sinkBase).href)); + assert.ok(recipientInboxes.includes(new URL("/inbox/1", sinkBase).href)); +}); + +test("failureRunner.validate - rejects sinkBase for mixed network faults", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "failure", + type: "failure", + fault: ["network-error", "remote-404"], + sender: "alice", + sinkBase: "http://127.0.0.1:29999/", + }], + }).scenarios[0]; + + assert.throws( + () => failureRunner.validate?.(scenario), + /cannot combine network-error with other remote failure faults/, + ); +}); + +test("failureRunner - detects network-error retries", async () => { + const target = new URL("http://target.test/"); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault: "network-error", + sender: "alice", + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "50ms", + }], + }).scenarios[0]; + let triggerCalls = 0; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(statsJson(statsSnapshot({ + enqueued: triggerCalls * 2, + completed: triggerCalls, + failed: 0, + permanentFailures: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve(statsJson({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.failed, 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(triggerCalls > 0); +}); + +test("failureRunner - tolerates transient remote fault stats failures", async () => { + const target = new URL("http://target.test/"); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "failure", + type: "failure", + fault: "remote-404", + sender: "alice", + load: { concurrency: 1 }, + duration: "25ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + let statsCalls = 0; + let triggerCalls = 0; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + statsCalls++; + if (statsCalls === 2) { + return Promise.resolve( + new Response("temporarily unavailable", { + status: 503, + }), + ); + } + return Promise.resolve(statsJson(statsSnapshot({ + enqueued: triggerCalls, + completed: triggerCalls, + failed: 0, + permanentFailures: statsCalls > 1 ? triggerCalls : 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve(statsJson({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.failed, 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(statsCalls >= 3); +}); + +test("failureRunner.validate - requires sender for remote faults", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "failure", + type: "failure", + fault: "remote-404", + }], + }).scenarios[0]; + + assert.throws(() => failureRunner.validate?.(scenario), /sender/); +}); + +test("failureRunner.validate - rejects unsupported faults", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "failure", + type: "failure", + fault: "unknown-fault", + }], + }).scenarios[0]; + + assert.throws(() => failureRunner.validate?.(scenario), /unsupported/); +}); + +test("failureRunner.validate - rejects invalid explicit inbound inboxes", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "failure", + type: "failure", + fault: "invalid-signature", + recipient: "http://target.test/users/alice", + inbox: "shraed", + }], + }).scenarios[0]; + + assert.throws( + () => failureRunner.validate?.(scenario), + /inbox must be "shared", "personal", or an http\(s\) URL/, + ); +}); + +test("failureRunner - discovers inbound failure inboxes once", async () => { + let actorGets = 0; + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request): Response { + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + actorGets++; + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Person", + id: url.href, + inbox: new URL("/users/alice/inbox", url).href, + endpoints: { + sharedInbox: new URL("/inbox", url).href, + }, + }); + } + return new Response("not found", { status: 404 }); + }, + }); + await server.ready(); + let fleet: Awaited> | undefined; + try { + const target = new URL(server.url!); + fleet = await spawnSyntheticServer( + await buildFleet([{ + count: 1, + signatureStandards: ["draft-cavage-http-signatures-12"], + }]), + ); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "bad-signature", + type: "failure", + fault: "invalid-signature", + recipient: new URL("/users/alice", target).href, + load: { rate: 100 }, + duration: "30ms", + }], + }).scenarios[0]; + let now = 0; + const clock: Clock = { + now: () => now, + sleepUntil: (timeMs) => { + now = Math.max(now, timeMs); + return Promise.resolve(); + }, + }; + let malformedSignatureRequests = 0; + let signedDateRequests = 0; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet, + fetch: async (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/inbox") { + await request.clone().json(); + const signature = request.headers.get("signature"); + const authorization = request.headers.get("authorization"); + if ( + signature?.endsWith("0") === true || + authorization?.endsWith("0") === true + ) { + malformedSignatureRequests++; + } + if (!Number.isNaN(Date.parse(request.headers.get("date") ?? ""))) { + signedDateRequests++; + } + return new Response("bad signature", { + status: 401, + }); + } + return new Response("not found", { status: 404 }); + }, + assertDestinationAllowed: () => {}, + clock, + }); + + assert.strictEqual(measurement.requests.total, 3); + assert.strictEqual(measurement.requests.successRate, 1); + assert.strictEqual(malformedSignatureRequests, 0); + assert.strictEqual(signedDateRequests, 3); + assert.strictEqual(actorGets, 1); + } finally { + try { + await fleet?.close(); + } finally { + await server.close(true); + } + } +}); + +test("failureRunner - treats inbound 5xx as target failures", async () => { + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request): Response { + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Person", + id: url.href, + inbox: new URL("/inbox", url).href, + }); + } + return new Response("not found", { status: 404 }); + }, + }); + await server.ready(); + let fleet: Awaited> | undefined; + try { + const target = new URL(server.url!); + fleet = await spawnSyntheticServer( + await buildFleet([{ + count: 1, + signatureStandards: ["draft-cavage-http-signatures-12"], + }]), + ); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "bad-signature", + type: "failure", + fault: "invalid-signature", + recipient: new URL("/users/alice", target).href, + load: { concurrency: 1 }, + duration: "50ms", + }], + }).scenarios[0]; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/inbox") { + return Promise.resolve( + new Response("internal error", { status: 500 }), + ); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 0); + assert.ok(measurement.errors.some((e) => e.status === 500)); + } finally { + try { + await fleet?.close(); + } finally { + await server.close(true); + } + } +}); + +test("failureRunner - treats unexpected inbound 4xx as target failures", async () => { + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request): Response { + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Person", + id: url.href, + inbox: new URL("/missing-inbox", url).href, + }); + } + return new Response("not found", { status: 404 }); + }, + }); + await server.ready(); + let fleet: Awaited> | undefined; + try { + const target = new URL(server.url!); + fleet = await spawnSyntheticServer( + await buildFleet([{ + count: 1, + signatureStandards: ["draft-cavage-http-signatures-12"], + }]), + ); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "bad-signature", + type: "failure", + fault: "invalid-signature", + recipient: new URL("/users/alice", target).href, + load: { concurrency: 1 }, + duration: "50ms", + }], + }).scenarios[0]; + const measurement = await failureRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/missing-inbox") { + return Promise.resolve( + new Response("not found", { status: 404 }), + ); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + assertDestinationAllowed: () => {}, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 0); + assert.ok(measurement.errors.some((e) => e.status === 404)); + } finally { + try { + await fleet?.close(); + } finally { + await server.close(true); + } + } +}); + +function json(body: unknown): Response { + return new Response(JSON.stringify(body), { + headers: { "content-type": "application/activity+json" }, + }); +} + +function statsJson(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); +} + +async function reservePort(): Promise { + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch: () => new Response("reserved"), + }); + await server.ready(); + const port = Number(new URL(server.url!).port); + await server.close(true); + return port; +} + +function statsSnapshot(counts: { + readonly enqueued: number; + readonly completed: number; + readonly failed: number; + readonly permanentFailures: number; +}): Record { + return { + version: 1, + source: "server", + scopeMetrics: [{ + metrics: [ + sum("fedify.queue.task.enqueued", counts.enqueued), + sum("fedify.queue.task.completed", counts.completed), + sum("fedify.queue.task.failed", counts.failed), + sum( + "activitypub.delivery.permanent_failure", + counts.permanentFailures, + ), + ], + }], + }; +} + +function sum(name: string, value: number): Record { + return { + name, + dataPointType: "sum", + dataPoints: [{ value }], + }; +} diff --git a/packages/cli/src/bench/scenarios/failure.ts b/packages/cli/src/bench/scenarios/failure.ts new file mode 100644 index 000000000..323d96e5e --- /dev/null +++ b/packages/cli/src/bench/scenarios/failure.ts @@ -0,0 +1,584 @@ +/** + * The `failure` scenario runner. + * @since 2.3.0 + * @module + */ + +import { Create, Note } from "@fedify/vocab"; +import { discoverInbox, selectInbox } from "../discovery/discover.ts"; +import { runLoad, type SendOutcome } from "../load/generator.ts"; +import { aggregateSamples } from "../metrics/aggregate.ts"; +import { + diffSnapshots, + fetchServerSnapshot, + queueTaskRemaining, +} from "../metrics/stats-client.ts"; +import type { SyntheticActor } from "../server/synthetic.ts"; +import { createActivityIdMinter } from "../signing/activity-id.ts"; +import { signInboxDelivery } from "../signing/signer.ts"; +import { + assertSinkRecipientsAllowed, + resolveSinkBase, + spawnSinkServer, +} from "./fanout.ts"; +import { + loadPlanOf, + measuredWindowMs, + type RunContext, + type ScenarioRunner, + sendRequest, + validateInboxSelector, +} from "./runner.ts"; + +const SUPPORTED_FAULTS = [ + "invalid-signature", + "missing-actor", + "remote-404", + "remote-410", + "slow-inbox", + "network-error", +] as const; + +type SupportedFault = typeof SUPPORTED_FAULTS[number]; +type RemoteFailureFault = Exclude< + SupportedFault, + "invalid-signature" | "missing-actor" +>; + +interface FailureDeliveryTarget { + readonly inbox: URL; + readonly actorUri: URL; +} + +interface RemoteFailureTarget { + readonly recipient: Record; + readonly close: () => Promise; +} + +const DEFAULT_DRAIN_TIMEOUT_MS = 60_000; +const DRAIN_POLL_MS = 25; + +/** The `failure` scenario runner. */ +export const failureRunner: ScenarioRunner = { + validate(scenario): void { + const faults = scenario.faults.length < 1 + ? ["remote-404"] + : scenario.faults; + const remoteFaults = [...new Set(faults.filter(isRemoteFault))]; + for (const fault of faults) { + if (!isSupportedFault(fault)) { + throw new Error( + `Scenario "${scenario.name}": unsupported failure fault ` + + `${JSON.stringify(fault)}; supported faults: ${ + SUPPORTED_FAULTS.join(", ") + }.`, + ); + } + } + if (faults.some(isInboundFault)) { + validateInboxSelector(scenario.name, scenario.inbox); + } + if (faults.some(isInboundFault) && scenario.recipients.length < 1) { + throw new Error( + `Scenario "${scenario.name}": invalid-signature and missing-actor ` + + "faults require a recipient.", + ); + } + if (faults.some(isRemoteFault) && scenario.sender == null) { + throw new Error( + `Scenario "${scenario.name}": remote failure faults require a ` + + "sender.", + ); + } + if (faults.some(isRemoteFault)) { + resolveSinkBase(scenario.name, scenario.raw.sinkBase); + } + if ( + scenario.raw.sinkBase != null && + remoteFaults.includes("network-error") && + remoteFaults.some((fault) => fault !== "network-error") + ) { + throw new Error( + `Scenario "${scenario.name}": sinkBase cannot combine ` + + "network-error with other remote failure faults because the same " + + "port cannot be both open and unreachable.", + ); + } + }, + + async run(context: RunContext) { + this.validate?.(context.scenario); + const faults = faultsOf(context); + const deliveryTarget = await resolveFailureDeliveryTarget(context, faults); + const remoteTargets = await resolveRemoteFailureTargets(context, faults); + const remoteActivityIds = createActivityIdMinter(context.target); + try { + await assertSinkRecipientsAllowed( + [...remoteTargets.values()].map((target) => target.recipient), + context, + ); + let index = 0; + const sendOne = () => + sendForFault( + context, + faults[index++ % faults.length], + deliveryTarget, + remoteTargets, + remoteActivityIds, + ); + let send = sendOne; + if (faults.some(isRemoteFault)) { + let previous = Promise.resolve(); + send = () => { + const current = previous.then(sendOne); + previous = current.then( + () => {}, + () => {}, + ); + return current; + }; + } + const result = await runLoad( + loadPlanOf(context.scenario, context.rng), + send, + context.clock, + ); + return aggregateSamples(result.samples, { + measuredWindowMs: measuredWindowMs(context.scenario), + includeHistogram: true, + }); + } finally { + await Promise.all( + [...remoteTargets.values()].map((target) => target.close()), + ); + } + }, +}; + +function faultsOf(context: RunContext): SupportedFault[] { + const faults = context.scenario.faults.length < 1 + ? ["remote-404"] + : context.scenario.faults; + return faults.map((fault) => { + if (isSupportedFault(fault)) return fault; + throw new Error( + `Scenario "${context.scenario.name}": unsupported failure fault ` + + `${JSON.stringify(fault)}.`, + ); + }); +} + +function isSupportedFault(fault: string): fault is SupportedFault { + return SUPPORTED_FAULTS.includes(fault as SupportedFault); +} + +async function resolveFailureDeliveryTarget( + context: RunContext, + faults: readonly SupportedFault[], +): Promise { + if (!faults.some(isInboundFault)) return null; + const { scenario } = context; + const discovered = await discoverInbox(scenario.recipients[0], { + documentLoader: context.documentLoader, + contextLoader: context.contextLoader, + allowPrivateAddress: context.allowPrivateAddress, + }); + const inbox = selectInbox(discovered, scenario.inbox); + if (faults.every((fault) => fault === "missing-actor")) { + await context.assertActorlessDestinationAllowed?.(inbox); + } else { + await context.assertDestinationAllowed?.(inbox); + } + return { inbox, actorUri: discovered.actorUri }; +} + +function isInboundFault( + fault: string, +): fault is Extract { + return fault === "invalid-signature" || fault === "missing-actor"; +} + +function isRemoteFault(fault: string): fault is RemoteFailureFault { + return fault === "remote-404" || fault === "remote-410" || + fault === "slow-inbox" || fault === "network-error"; +} + +async function resolveRemoteFailureTargets( + context: RunContext, + faults: readonly SupportedFault[], +): Promise> { + const targets = new Map(); + try { + const remoteFaults = [...new Set(faults.filter(isRemoteFault))]; + const liveFaults = remoteFaults.filter((fault) => + fault !== "network-error" + ); + if (liveFaults.length > 0) { + const sink = await spawnSinkServer({ + followers: liveFaults.length, + rawBehavior: null, + rawBehaviors: liveFaults.map(remoteSinkBehavior), + advertiseHost: context.advertiseHost, + sinkBase: context.scenario.raw.sinkBase, + }); + const close = once(sink.close); + for (const [index, fault] of liveFaults.entries()) { + targets.set(fault, { recipient: sink.recipients[index], close }); + } + } + if (remoteFaults.includes("network-error")) { + const sink = await spawnSinkServer({ + followers: 1, + rawBehavior: remoteSinkBehavior("network-error"), + advertiseHost: context.advertiseHost, + sinkBase: context.scenario.raw.sinkBase, + }); + const recipient = sink.recipients[0]; + try { + await sink.close(); + targets.set("network-error", { + recipient, + close: () => Promise.resolve(), + }); + } catch (error) { + await sink.close().catch(() => {}); + throw error; + } + } + return targets; + } catch (error) { + await Promise.all([...targets.values()].map((target) => target.close())); + throw error; + } +} + +function once(close: () => Promise): () => Promise { + let closed: Promise | null = null; + return () => { + closed ??= close(); + return closed; + }; +} + +function remoteSinkBehavior( + fault: RemoteFailureFault, +): Record { + switch (fault) { + case "remote-404": + return { status: 404 }; + case "remote-410": + return { status: 410 }; + case "slow-inbox": + return { status: 202, latency: "25ms" }; + case "network-error": + return { status: 202 }; + } +} + +async function sendForFault( + context: RunContext, + fault: SupportedFault, + deliveryTarget: FailureDeliveryTarget | null, + remoteTargets: ReadonlyMap, + remoteActivityIds: ReturnType, +): Promise { + switch (fault) { + case "invalid-signature": + return await sendInvalidSignature( + context, + requiredTarget(deliveryTarget), + ); + case "missing-actor": + return await sendMissingActor(context, requiredTarget(deliveryTarget)); + case "remote-404": + case "remote-410": + case "slow-inbox": + case "network-error": + return await sendRemoteFailure( + context, + fault, + requiredRemoteTarget(fault, remoteTargets), + remoteActivityIds, + ); + } +} + +async function sendRemoteFailure( + context: RunContext, + fault: RemoteFailureFault, + target: RemoteFailureTarget, + remoteActivityIds: ReturnType, +): Promise { + const fetchImpl = context.fetch ?? fetch; + const baseline = await fetchServerSnapshot(context.target, fetchImpl); + const response = await fetchImpl( + new URL("/.well-known/fedify/bench/trigger", context.target), + { + method: "POST", + headers: { "content-type": "application/json" }, + redirect: "manual", + body: JSON.stringify({ + sender: { identifier: requiredSender(context) }, + recipients: [target.recipient], + activity: buildRemoteFailureActivity(context, remoteActivityIds.next()), + }), + }, + ); + await response.arrayBuffer().catch(() => {}); + if (!response.ok) { + return { + ok: false, + status: response.status, + reason: `status_${response.status}`, + }; + } + const observation = await waitForRemoteFault({ + target: context.target, + fetch: fetchImpl, + baseline, + fault, + timeoutMs: context.scenario.queueDrainTimeoutMs ?? + DEFAULT_DRAIN_TIMEOUT_MS, + }); + if (observation == null) { + return { + ok: false, + errorKind: "server", + reason: "stats_unavailable", + }; + } + if (observation.timedOut) { + return { + ok: false, + errorKind: "server", + reason: "expected_remote_failure_not_observed", + }; + } + return expectedRemoteFailure(fault); +} + +function buildRemoteFailureActivity( + context: RunContext, + id: URL, +): Record { + const objectId = new URL(`/objects/${crypto.randomUUID()}`, context.target); + return { + "@context": "https://www.w3.org/ns/activitystreams", + type: "Create", + id: id.href, + actor: new URL(`/users/${requiredSender(context)}`, context.target).href, + object: { + type: "Note", + id: objectId.href, + content: "Benchmark failure activity.", + }, + }; +} + +interface RemoteFaultObservation { + readonly timedOut: boolean; +} + +async function waitForRemoteFault(options: { + readonly target: URL; + readonly fetch: typeof fetch; + readonly baseline: Awaited>; + readonly fault: RemoteFailureFault; + readonly timeoutMs: number; +}): Promise { + if (options.baseline == null) return null; + const baselineRemaining = queueTaskRemaining(options.baseline) ?? 0; + const deadline = Date.now() + options.timeoutMs; + do { + const snapshot = await fetchServerSnapshot(options.target, options.fetch); + if (snapshot != null) { + const diff = diffSnapshots(options.baseline, snapshot); + const queueTasks = diff.queueTasks; + if (options.fault === "remote-404" || options.fault === "remote-410") { + if ((diff.deliveryPermanentFailures ?? 0) > 0) { + return { timedOut: false }; + } + } else if (queueTasks != null) { + const remaining = queueTaskRemaining(diff, baselineRemaining); + if (remaining != null) { + if (options.fault === "slow-inbox") { + if (queueTasks.completed > 0 && remaining === 0) { + return { timedOut: false }; + } + } else if (options.fault === "network-error") { + if ( + queueTasks.failed > 0 || + (queueTasks.completed > 0 && remaining > 0) + ) { + return { timedOut: false }; + } + } + } + } + } + await new Promise((resolve) => setTimeout(resolve, DRAIN_POLL_MS)); + } while (Date.now() < deadline); + return { timedOut: true }; +} + +function expectedRemoteFailure(fault: RemoteFailureFault): SendOutcome { + switch (fault) { + case "remote-404": + return { ok: true, status: 404 }; + case "remote-410": + return { ok: true, status: 410 }; + case "slow-inbox": + return { ok: true, status: 202 }; + case "network-error": + return { + ok: true, + errorKind: "network", + reason: "expected_network_error", + }; + } +} + +async function sendInvalidSignature( + context: RunContext, + deliveryTarget: FailureDeliveryTarget, +): Promise { + const request = await signedFailureRequest( + context, + "invalid-signature", + deliveryTarget, + ); + const body = await request.arrayBuffer(); + const headers = new Headers(request.headers); + invalidateSignedDate(headers); + return expectedFailure( + await sendRequest( + new Request(request.url, { + method: request.method, + headers, + body, + redirect: "manual", + }), + context.fetch ?? fetch, + ), + 401, + ); +} + +function invalidateSignedDate(headers: Headers): void { + const timestamp = Date.parse(headers.get("date") ?? ""); + const shifted = Number.isNaN(timestamp) + ? new Date() + : new Date(timestamp + 1000); + headers.set("date", shifted.toUTCString()); +} + +async function sendMissingActor( + context: RunContext, + deliveryTarget: FailureDeliveryTarget, +): Promise { + const request = await signedFailureRequest( + context, + "missing-actor", + deliveryTarget, + ); + return expectedFailure( + await sendRequest(request, context.fetch ?? fetch), + 401, + ); +} + +async function signedFailureRequest( + context: RunContext, + fault: "invalid-signature" | "missing-actor", + deliveryTarget: FailureDeliveryTarget, +): Promise { + const { fleet, scenario } = context; + if (fleet == null || fleet.actors.length < 1) { + throw new Error( + "The failure scenario requires the synthetic actor server.", + ); + } + if (scenario.recipients.length < 1) { + throw new Error( + "The invalid-signature and missing-actor faults require a recipient.", + ); + } + const actor = fault === "missing-actor" + ? missingActor(fleet.actors[0], context.target) + : fleet.actors[0]; + const id = createActivityIdMinter(fleet.url).next(); + const note = new Note({ + id: new URL(`/objects/${crypto.randomUUID()}`, fleet.url), + attribution: actor.id, + content: "Benchmark failure activity.", + to: deliveryTarget.actorUri, + }); + const activity = new Create({ + id, + actor: actor.id, + object: note, + to: deliveryTarget.actorUri, + }); + return await signInboxDelivery({ + actor, + inbox: deliveryTarget.inbox, + activity, + contextLoader: context.contextLoader, + }); +} + +function requiredTarget( + target: FailureDeliveryTarget | null, +): FailureDeliveryTarget { + if (target == null) { + throw new Error( + "The invalid-signature and missing-actor faults require discovery.", + ); + } + return target; +} + +function requiredRemoteTarget( + fault: RemoteFailureFault, + targets: ReadonlyMap, +): RemoteFailureTarget { + const target = targets.get(fault); + if (target == null) { + throw new Error(`The ${fault} fault requires a benchmark sink.`); + } + return target; +} + +function requiredSender(context: RunContext): string { + const sender = context.scenario.sender; + if (sender == null) { + throw new Error("Remote failure faults require a sender."); + } + return sender; +} + +function missingActor(actor: SyntheticActor, target: URL): SyntheticActor { + const id = new URL(`/__fedify_bench/missing/${crypto.randomUUID()}`, target); + return { + ...actor, + id, + rsaKeyId: actor.rsaKeyId == null ? undefined : new URL("#main-key", id), + ed25519KeyId: actor.ed25519KeyId == null + ? undefined + : new URL("#ed25519-key", id), + }; +} + +function expectedFailure( + outcome: SendOutcome, + expectedStatus: number, +): SendOutcome { + if (outcome.status === expectedStatus) { + return { ok: true, status: outcome.status }; + } + return { + ...outcome, + ok: false, + reason: outcome.reason ?? "expected_failure_not_observed", + }; +} diff --git a/packages/cli/src/bench/scenarios/fanout.test.ts b/packages/cli/src/bench/scenarios/fanout.test.ts new file mode 100644 index 000000000..a81ed7ccb --- /dev/null +++ b/packages/cli/src/bench/scenarios/fanout.test.ts @@ -0,0 +1,583 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { serve } from "srvx"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import type { Suite } from "../scenario/types.ts"; +import { fanoutRunner, spawnSinkServer } from "./fanout.ts"; + +test("fanoutRunner - triggers benchmark hook and reports drain", async () => { + const target = new URL("http://target.test/"); + let triggerCalls = 0; + let triggerRecipients = 0; + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "50ms", + queueDrainTimeout: "1s", + expect: { deliveryThroughput: ">= 1/s" }, + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(json(statsSnapshot({ + enqueued: triggerCalls * 6, + completed: triggerCalls * 6, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + const triggerBody = JSON.parse(String(init?.body)); + triggerRecipients = triggerBody.recipients.length; + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(measurement.server?.queue?.drainMs?.p95 != null); + assert.ok(measurement.throughputPerSec > 0); + assert.ok(measurement.deliveryThroughputPerSec != null); + assert.ok( + Math.abs( + measurement.deliveryThroughputPerSec - + measurement.throughputPerSec * triggerRecipients, + ) < 1e-9, + ); + assert.strictEqual(triggerRecipients, 5); +}); + +test("fanoutRunner.validate - accepts schema-minimum follower count", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 1, + }], + }).scenarios[0]; + assert.doesNotThrow(() => fanoutRunner.validate?.(scenario)); +}); + +test("fanoutRunner.validate - rejects invalid sinkBase URLs", () => { + for ( + const sinkBase of [ + "http://target.test/", + "https://target.test:8443/", + "http://user:pass@target.test:9090/", + "http://target.test:9090/sinks/", + ] + ) { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + sinkBase, + }], + }).scenarios[0]; + + assert.throws( + () => fanoutRunner.validate?.(scenario), + /sinkBase must be an http URL/, + ); + } +}); + +test("fanoutRunner - serializes overlapping trigger drains", async () => { + const target = new URL("http://target.test/"); + let statsCalls = 0; + let activeTriggers = 0; + let maxActiveTriggers = 0; + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 3 }, + duration: "40ms", + queueDrainTimeout: "1s", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: async (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + statsCalls++; + const drained = statsCalls % 2 === 0; + return json(statsSnapshot({ + enqueued: drained ? 6 : 0, + completed: drained ? 6 : 0, + failed: 0, + })); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + activeTriggers++; + maxActiveTriggers = Math.max(maxActiveTriggers, activeTriggers); + await new Promise((resolve) => setTimeout(resolve, 5)); + activeTriggers--; + return json({ version: 1 }, 202); + } + return new Response("unexpected", { status: 500 }); + }, + }); + + assert.strictEqual(maxActiveTriggers, 1); +}); + +test("fanoutRunner - counts failed queue tasks as delivery failures", async () => { + const target = new URL("http://target.test/"); + let statsCalls = 0; + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "40ms", + queueDrainTimeout: "1s", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + statsCalls++; + const failedTasks = Math.floor(statsCalls / 2) * 6; + return Promise.resolve(json(statsSnapshot({ + enqueued: failedTasks, + completed: 0, + failed: failedTasks, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 0); + assert.ok(measurement.throughputPerSec > 0); + assert.strictEqual(measurement.deliveryThroughputPerSec, 0); +}); + +test("fanoutRunner - waits for observed queue work before drain", async () => { + const target = new URL("http://target.test/"); + let statsCalls = 0; + let triggerCalls = 0; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "40ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + statsCalls++; + const observed = statsCalls > 2; + const queueTasks = observed ? triggerCalls * 6 : 0; + return Promise.resolve(json(statsSnapshot({ + enqueued: queueTasks, + completed: queueTasks, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(statsCalls > 2); +}); + +test("fanoutRunner - ignores baseline backlog completions while draining", async () => { + const target = new URL("http://target.test/"); + let triggerCalls = 0; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "1ms", + queueDrainTimeout: "30ms", + }], + }).scenarios[0]; + + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(json(statsSnapshot({ + enqueued: 6 + triggerCalls * 6, + completed: triggerCalls > 0 ? 6 : 0, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 0); + assert.ok(triggerCalls > 0); +}); + +test("fanoutRunner - tolerates transient drain stats failures", async () => { + const target = new URL("http://target.test/"); + let statsCalls = 0; + let triggerCalls = 0; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "40ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + statsCalls++; + if (statsCalls === 2) { + return Promise.resolve( + new Response("temporarily unavailable", { + status: 503, + }), + ); + } + return Promise.resolve(json(statsSnapshot({ + enqueued: triggerCalls * 6, + completed: triggerCalls * 6, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.failed, 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(statsCalls >= 3); +}); + +test("fanoutRunner - uses configured sink base for recipients", async () => { + const target = new URL("http://target.test/"); + const sinkBase = `http://127.0.0.1:${await reservePort()}/`; + let triggerCalls = 0; + let recipientInboxes: string[] = []; + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + sinkBase, + load: { concurrency: 1 }, + duration: "40ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input, init) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(json(statsSnapshot({ + enqueued: triggerCalls * 6, + completed: triggerCalls * 6, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + const body = JSON.parse(String(init?.body)); + recipientInboxes = body.recipients.map(( + recipient: Record, + ) => recipient.inbox); + return Promise.resolve(json({ version: 1 }, 202)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.deepStrictEqual( + recipientInboxes, + Array.from({ length: 5 }, (_, i) => new URL(`/inbox/${i}`, sinkBase).href), + ); +}); + +test("fanoutRunner - gates sink recipients before triggering", async () => { + const target = new URL("http://target.test/"); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + sinkBase: `http://127.0.0.1:${await reservePort()}/`, + load: { concurrency: 1 }, + duration: "30ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + let gateCalls = 0; + let triggerCalls = 0; + + await assert.rejects( + async () => + fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/trigger") { + triggerCalls++; + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + assertActorlessDestinationAllowed: (url) => { + gateCalls++; + throw new Error(`refused ${url.href}`); + }, + }), + /refused http:\/\/127\.0\.0\.1:/, + ); + + assert.strictEqual(gateCalls, 1); + assert.strictEqual(triggerCalls, 0); +}); + +test("spawnSinkServer - ignores invalid sink latency", async () => { + const sink = await spawnSinkServer({ + followers: 1, + rawBehavior: { latency: "not-a-duration", status: 202 }, + }); + try { + const response = await fetch(String(sink.recipients[0].inbox), { + method: "POST", + body: "{}", + }); + assert.strictEqual(response.status, 202); + } finally { + await sink.close(); + } +}); + +test("spawnSinkServer - ignores out-of-range sink status", async () => { + const sink = await spawnSinkServer({ + followers: 1, + rawBehavior: { status: 999 }, + }); + try { + const response = await fetch(String(sink.recipients[0].inbox), { + method: "POST", + body: "{}", + }); + assert.strictEqual(response.status, 202); + } finally { + await sink.close(); + } +}); + +test("fanoutRunner - omits queue drain metrics without drain samples", async () => { + const target = new URL("http://target.test/"); + const scenario = normalizeSuite({ + version: 1, + target: target.href, + scenarios: [{ + name: "fanout", + type: "fanout", + sender: "alice", + followers: 5, + load: { concurrency: 1 }, + duration: "30ms", + queueDrainTimeout: "1s", + }], + }).scenarios[0]; + + const measurement = await fanoutRunner.run({ + scenario, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(json(statsSnapshot({ + enqueued: 0, + completed: 0, + failed: 0, + }))); + } + if (url.pathname === "/.well-known/fedify/bench/trigger") { + return Promise.resolve(json({ error: "unavailable" }, 503)); + } + return Promise.resolve(new Response("unexpected", { status: 500 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 0); + assert.strictEqual(measurement.server?.queue?.drainMs, undefined); +}); + +function json(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); +} + +async function reservePort(): Promise { + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch: () => new Response("reserved"), + }); + await server.ready(); + const port = Number(new URL(server.url!).port); + await server.close(true); + return port; +} + +function statsSnapshot(counts: { + readonly enqueued: number; + readonly completed: number; + readonly failed: number; +}): Record { + return { + version: 1, + source: "server", + scopeMetrics: [{ + metrics: [ + sum("fedify.queue.task.enqueued", counts.enqueued), + sum("fedify.queue.task.completed", counts.completed), + sum("fedify.queue.task.failed", counts.failed), + ], + }], + }; +} + +function sum(name: string, value: number): Record { + return { + name, + dataPointType: "sum", + dataPoints: [{ value }], + }; +} diff --git a/packages/cli/src/bench/scenarios/fanout.ts b/packages/cli/src/bench/scenarios/fanout.ts new file mode 100644 index 000000000..045fbff96 --- /dev/null +++ b/packages/cli/src/bench/scenarios/fanout.ts @@ -0,0 +1,374 @@ +/** + * The `fanout` scenario runner. + * @since 2.3.0 + * @module + */ + +import { serve } from "srvx"; +import { runLoad, type SendOutcome } from "../load/generator.ts"; +import { aggregateSamples } from "../metrics/aggregate.ts"; +import { LogLinearHistogram } from "../metrics/histogram.ts"; +import { + diffSnapshots, + fetchServerSnapshot, + queueTaskRemaining, +} from "../metrics/stats-client.ts"; +import type { PartialLatencyMs, ServerMetrics } from "../result/model.ts"; +import { parseDuration } from "../scenario/units.ts"; +import { resolveAdvertiseHost } from "../server/synthetic.ts"; +import { createActivityIdMinter } from "../signing/activity-id.ts"; +import { + loadPlanOf, + measuredWindowMs, + type RunContext, + type ScenarioRunner, +} from "./runner.ts"; + +const DEFAULT_FOLLOWERS = 5; +const DEFAULT_DRAIN_TIMEOUT_MS = 60_000; +const DRAIN_POLL_MS = 25; + +/** The `fanout` scenario runner. */ +export const fanoutRunner: ScenarioRunner = { + validate(scenario): void { + const kind = triggerKind(scenario.raw.trigger); + if (kind !== "benchmark-hook") { + throw new Error( + `Scenario "${scenario.name}": fanout currently supports only ` + + `trigger.kind: "benchmark-hook".`, + ); + } + resolveSinkBase(scenario.name, scenario.raw.sinkBase); + }, + + async run(context: RunContext) { + if (context.scenario.sender == null) { + throw new Error("The fanout scenario requires a sender."); + } + this.validate?.(context.scenario); + const fetchImpl = context.fetch ?? fetch; + const followers = context.scenario.followers ?? DEFAULT_FOLLOWERS; + const sink = await spawnSinkServer({ + followers, + rawBehavior: context.scenario.raw.sinkBehavior, + advertiseHost: context.advertiseHost, + sinkBase: context.scenario.raw.sinkBase, + }); + const minter = createActivityIdMinter(context.target); + const drainHistogram = new LogLinearHistogram(); + let delivered = 0; + try { + await assertSinkRecipientsAllowed(sink.recipients, context); + const sendOne = async (scheduledAtMs: number): Promise => { + const baseline = await fetchServerSnapshot(context.target, fetchImpl); + const started = Date.now(); + const response = await fetchImpl( + new URL( + "/.well-known/fedify/bench/trigger", + context.target, + ), + { + method: "POST", + headers: { "content-type": "application/json" }, + redirect: "manual", + body: JSON.stringify({ + sender: { identifier: context.scenario.sender }, + recipients: sink.recipients, + activity: buildActivity(context, minter.next()), + }), + }, + ); + await response.arrayBuffer().catch(() => {}); + if (!response.ok) { + return { + ok: false, + status: response.status, + reason: `status_${response.status}`, + }; + } + const drain = await waitForDrain({ + target: context.target, + fetch: fetchImpl, + baseline, + timeoutMs: context.scenario.queueDrainTimeoutMs ?? + DEFAULT_DRAIN_TIMEOUT_MS, + }); + if (drain == null) { + return { + ok: false, + errorKind: "server", + reason: "stats_unavailable", + }; + } + if (drain.timedOut) { + return { + ok: false, + errorKind: "server", + reason: "queue_drain_timeout", + }; + } + if (drain.failed > 0) { + return { + ok: false, + errorKind: "server", + reason: "queue_delivery_failed", + }; + } + if (scheduledAtMs >= context.scenario.warmupMs) { + drainHistogram.record(Date.now() - started); + delivered += sink.recipients.length; + } + return { ok: true, status: response.status }; + }; + let previous = Promise.resolve(); + const send = (scheduledAtMs: number): Promise => { + const current = previous.then(() => sendOne(scheduledAtMs)); + previous = current.then( + () => {}, + () => {}, + ); + return current; + }; + const result = await runLoad( + loadPlanOf(context.scenario, context.rng), + send, + context.clock, + ); + const measurement = aggregateSamples(result.samples, { + measuredWindowMs: measuredWindowMs(context.scenario), + includeHistogram: true, + }); + const server = addQueueDrain(measurement.server, drainHistogram); + const deliveryThroughputPerSec = delivered / + (Math.max(measuredWindowMs(context.scenario), 1) / 1000); + return { + ...measurement, + deliveryThroughputPerSec, + server, + }; + } finally { + await sink.close(); + } + }, +}; + +function triggerKind(trigger: unknown): string { + if (trigger == null) return "benchmark-hook"; + if (typeof trigger !== "object" || Array.isArray(trigger)) return ""; + const kind = (trigger as Record).kind; + return typeof kind === "string" ? kind : "benchmark-hook"; +} + +function buildActivity(context: RunContext, id: URL): Record { + const objectId = new URL(`/objects/${crypto.randomUUID()}`, context.target); + return { + "@context": "https://www.w3.org/ns/activitystreams", + type: "Create", + id: id.href, + actor: new URL(`/users/${context.scenario.sender}`, context.target).href, + object: { + type: "Note", + id: objectId.href, + content: "Benchmark fanout activity.", + }, + }; +} + +export async function assertSinkRecipientsAllowed( + recipients: readonly Record[], + context: RunContext, +): Promise { + for (const recipient of recipients) { + if (typeof recipient.inbox !== "string") continue; + await context.assertActorlessDestinationAllowed?.( + new URL(recipient.inbox), + context.scenario, + ); + } +} + +export async function spawnSinkServer(options: { + readonly followers: number; + readonly rawBehavior: unknown; + readonly rawBehaviors?: readonly unknown[]; + readonly advertiseHost?: string; + readonly sinkBase?: string; +}): Promise<{ + readonly recipients: readonly Record[]; + readonly close: () => Promise; +}> { + const sinkBase = resolveSinkBase("benchmark sink", options.sinkBase); + const advertised = options.advertiseHost == null + ? null + : resolveAdvertiseHost(options.advertiseHost); + const behaviors = Array.from( + { length: options.followers }, + (_, i) => + parseSinkBehavior(options.rawBehaviors?.[i] ?? options.rawBehavior), + ); + const server = serve({ + port: sinkBase?.port ?? 0, + hostname: sinkBase?.bindHost ?? advertised?.bindHost ?? "127.0.0.1", + silent: true, + async fetch(request: Request): Promise { + const match = /^\/inbox\/(\d+)(?:\/|$)/.exec( + new URL(request.url).pathname, + ); + if (match != null) { + const behavior = behaviors[Number(match[1])] ?? + parseSinkBehavior(options.rawBehavior); + await request.arrayBuffer().catch(() => {}); + if (behavior.latencyMs > 0) { + await new Promise((resolve) => + setTimeout(resolve, behavior.latencyMs) + ); + } + return new Response("accepted", { status: behavior.status }); + } + return new Response("Not found", { status: 404 }); + }, + }); + await server.ready(); + const bound = new URL(server.url!); + const base = sinkBase?.base ?? + (advertised == null + ? bound + : new URL(`http://${advertised.urlHost}:${bound.port}/`)); + const recipients = Array.from({ length: options.followers }, (_, i) => ({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Service", + id: new URL(`/actors/${i}`, base).href, + inbox: new URL(`/inbox/${i}`, base).href, + })); + return { + recipients, + close: () => server.close(true), + }; +} + +interface SinkBase { + readonly base: URL; + readonly bindHost: string; + readonly port: number; +} + +export function resolveSinkBase( + scenarioName: string, + value: string | undefined, +): SinkBase | null { + if (value == null) return null; + let url: URL; + try { + url = new URL(value); + } catch { + throw new Error( + `Scenario "${scenarioName}": sinkBase must be an http URL with an ` + + `explicit port; got ${JSON.stringify(value)}.`, + ); + } + const port = Number(url.port); + if ( + url.protocol !== "http:" || + url.hostname === "" || + url.username !== "" || + url.password !== "" || + url.port === "" || + !Number.isInteger(port) || + port < 1 || + url.pathname !== "/" || + url.search !== "" || + url.hash !== "" + ) { + throw new Error( + `Scenario "${scenarioName}": sinkBase must be an http URL with a ` + + "host, an explicit non-zero port, no credentials, and no path, " + + `query, or fragment; got ${JSON.stringify(value)}.`, + ); + } + const advertised = resolveAdvertiseHost(url.hostname); + return { + base: new URL(`http://${advertised.urlHost}:${url.port}/`), + bindHost: advertised.bindHost, + port, + }; +} + +function parseSinkBehavior( + raw: unknown, +): { latencyMs: number; status: number } { + if (raw == null || typeof raw !== "object" || Array.isArray(raw)) { + return { latencyMs: 0, status: 202 }; + } + const record = raw as Record; + const latency = record.latency; + const status = record.status; + let latencyMs = 0; + if (typeof latency === "string") { + try { + latencyMs = parseDuration(latency); + } catch { + latencyMs = 0; + } + } + return { + latencyMs, + status: typeof status === "number" && Number.isInteger(status) && + status >= 100 && status <= 599 + ? status + : 202, + }; +} + +interface DrainResult { + readonly timedOut: boolean; + readonly failed: number; +} + +async function waitForDrain(options: { + readonly target: URL; + readonly fetch: typeof fetch; + readonly baseline: Awaited>; + readonly timeoutMs: number; +}): Promise { + if (options.baseline == null) return null; + const baselineRemaining = queueTaskRemaining(options.baseline) ?? 0; + const deadline = Date.now() + options.timeoutMs; + do { + const snapshot = await fetchServerSnapshot(options.target, options.fetch); + if (snapshot != null) { + const diff = diffSnapshots(options.baseline, snapshot); + const queueTasks = diff.queueTasks; + const remaining = queueTaskRemaining(diff, baselineRemaining); + if ( + queueTasks != null && + queueTasks.enqueued > 0 && + remaining != null && + remaining === 0 + ) { + return { timedOut: false, failed: queueTasks.failed }; + } + } + await new Promise((resolve) => setTimeout(resolve, DRAIN_POLL_MS)); + } while (Date.now() < deadline); + return { timedOut: true, failed: 0 }; +} + +function addQueueDrain( + server: ServerMetrics | null, + histogram: LogLinearHistogram, +): ServerMetrics | null { + if (histogram.count < 1) return server; + const queue = { + ...(server?.queue ?? {}), + drainMs: partialFromHistogram(histogram), + }; + return { ...(server ?? {}), queue }; +} + +function partialFromHistogram(histogram: LogLinearHistogram): PartialLatencyMs { + return { + p50: histogram.percentile(50), + p95: histogram.percentile(95), + p99: histogram.percentile(99), + }; +} diff --git a/packages/cli/src/bench/scenarios/inbox.ts b/packages/cli/src/bench/scenarios/inbox.ts index e23d88b78..4bf8379d0 100644 --- a/packages/cli/src/bench/scenarios/inbox.ts +++ b/packages/cli/src/bench/scenarios/inbox.ts @@ -38,6 +38,7 @@ import { type RunContext, type ScenarioRunner, sendRequest, + validateInboxSelector, withMeasuredWindowStart, } from "./runner.ts"; @@ -163,26 +164,7 @@ export const inboxRunner: ScenarioRunner = { * mid-run, and a non-http URL would slip through to the send path. */ function validateInbox(scenario: ResolvedScenario): void { - const mode = scenario.inbox; - if (mode == null || mode === "shared" || mode === "personal") return; - let url: URL; - try { - url = new URL(mode); - } catch { - throw new Error( - `Scenario "${scenario.name}": inbox must be "shared", "personal", or an ` + - `http(s) URL; got ${JSON.stringify(mode)}.`, - ); - } - if ( - (url.protocol !== "http:" && url.protocol !== "https:") || - url.hostname === "" || url.username !== "" || url.password !== "" - ) { - throw new Error( - `Scenario "${scenario.name}": inbox URL must be a bare http(s) URL with ` + - `a host and no credentials; got ${JSON.stringify(mode)}.`, - ); - } + validateInboxSelector(scenario.name, scenario.inbox); } /** diff --git a/packages/cli/src/bench/scenarios/mixed.test.ts b/packages/cli/src/bench/scenarios/mixed.test.ts new file mode 100644 index 000000000..3b3d3b838 --- /dev/null +++ b/packages/cli/src/bench/scenarios/mixed.test.ts @@ -0,0 +1,550 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import { LogLinearHistogram } from "../metrics/histogram.ts"; +import type { ScenarioMeasurement } from "../result/build.ts"; +import { evaluateExpect } from "../result/expect/evaluate.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import type { Suite } from "../scenario/types.ts"; +import { createLimiter, mergeMeasurements, mixedRunner } from "./mixed.ts"; + +test("mixedRunner - runs weighted child scenarios together", async () => { + const target = new URL("http://target.test/"); + let webfingerCalls = 0; + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [ + { + name: "lookup-a", + type: "webfinger", + recipient: "acct:alice@target.test", + }, + { + name: "lookup-b", + type: "webfinger", + recipient: "acct:bob@target.test", + }, + { + name: "mixed", + type: "mixed", + load: { rate: 20 }, + duration: "50ms", + mix: [ + { scenario: "lookup-a", weight: 3 }, + { scenario: "lookup-b", weight: 1 }, + ], + }, + ], + }; + const scenarios = normalizeSuite(suite).scenarios; + const scenario = scenarios[2]; + const measurement = await mixedRunner.run({ + scenario, + scenarios, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/webfinger") { + webfingerCalls++; + return Promise.resolve(json({ + subject: url.searchParams.get("resource"), + links: [], + })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(webfingerCalls > 0); + assert.strictEqual(measurement.requests.failed, 0); + assert.strictEqual(measurement.requests.successRate, 1); +}); + +test("mixedRunner - gates child destinations with child scenarios", async () => { + const target = new URL("http://target.test/"); + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [ + { + name: "child-read", + type: "actor", + recipient: "http://remote.test/users/alice", + load: { concurrency: 1 }, + duration: "25ms", + }, + { + name: "mixed", + type: "mixed", + load: { concurrency: 1 }, + duration: "25ms", + mix: [{ scenario: "child-read", weight: 1 }], + }, + ], + }; + const scenarios = normalizeSuite(suite).scenarios; + const gatedScenarioNames: string[] = []; + + await mixedRunner.run({ + scenario: scenarios[1], + scenarios, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertReadDestinationAllowed: (_url, scenario) => { + gatedScenarioNames.push(scenario?.name ?? ""); + }, + }); + + assert.deepStrictEqual(gatedScenarioNames, ["mixed/child-read"]); +}); + +test("mixedRunner - enforces parent maxInFlight across children", async () => { + const target = new URL("http://target.test/"); + let activeLookups = 0; + let maxActiveLookups = 0; + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [ + { + name: "lookup-a", + type: "webfinger", + recipient: "acct:alice@target.test", + }, + { + name: "lookup-b", + type: "webfinger", + recipient: "acct:bob@target.test", + }, + { + name: "mixed", + type: "mixed", + load: { rate: 1000, maxInFlight: 2 }, + duration: "60ms", + mix: [ + { scenario: "lookup-a", weight: 1 }, + { scenario: "lookup-b", weight: 1 }, + ], + }, + ], + }; + const scenarios = normalizeSuite(suite).scenarios; + await mixedRunner.run({ + scenario: scenarios[2], + scenarios, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: async (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/webfinger") { + activeLookups++; + maxActiveLookups = Math.max(maxActiveLookups, activeLookups); + await new Promise((resolve) => setTimeout(resolve, 20)); + activeLookups--; + return json({ + subject: url.searchParams.get("resource"), + links: [], + }); + } + return new Response("not found", { status: 404 }); + }, + }); + + assert.ok( + maxActiveLookups <= 2, + `expected at most 2 in-flight lookups, got ${maxActiveLookups}`, + ); +}); + +test("createLimiter - ignores duplicate releases", async () => { + const limiter = createLimiter(1); + const releaseFirst = await limiter.acquire(); + let secondAcquired = false; + const second = limiter.acquire().then((release) => { + secondAcquired = true; + return release; + }); + + releaseFirst(); + releaseFirst(); + const releaseSecond = await second; + assert.strictEqual(secondAcquired, true); + + let thirdAcquired = false; + const third = limiter.acquire().then((release) => { + thirdAcquired = true; + return release; + }); + await Promise.resolve(); + assert.strictEqual(thirdAcquired, false); + + releaseSecond(); + const releaseThird = await third; + assert.strictEqual(thirdAcquired, true); + releaseSecond(); + releaseThird(); + releaseThird(); +}); + +test("mixedRunner - waits for siblings before propagating child errors", async () => { + const target = new URL("http://target.test/"); + const suite: Suite = { + version: 1, + target: target.href, + scenarios: [ + { + name: "lookup", + type: "webfinger", + recipient: "acct:alice@target.test", + }, + { + name: "object", + type: "object", + source: { + seed: "http://target.test/users/missing", + collection: "outbox", + limit: 1, + }, + }, + { + name: "mixed", + type: "mixed", + load: { concurrency: 2 }, + duration: "40ms", + mix: [ + { scenario: "lookup", weight: 1 }, + { scenario: "object", weight: 1 }, + ], + }, + ], + }; + const scenarios = normalizeSuite(suite).scenarios; + let webfingerStarted = false; + let webfingerSettled = false; + + await assert.rejects( + async () => + await mixedRunner.run({ + scenario: scenarios[2], + scenarios, + target, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: async (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return new Response("not found", { status: 404 }); + } + if (url.pathname === "/.well-known/webfinger") { + webfingerStarted = true; + await new Promise((resolve) => setTimeout(resolve, 30)); + webfingerSettled = true; + return json({ + subject: url.searchParams.get("resource"), + links: [], + }); + } + if (url.pathname === "/users/missing") { + const deadline = Date.now() + 100; + while (!webfingerStarted && Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, 1)); + } + await new Promise((resolve) => setTimeout(resolve, 5)); + return new Response("not found", { status: 404 }); + } + return new Response("unexpected", { status: 500 }); + }, + }), + /Failed to fetch http:\/\/target\.test\/users\/missing: HTTP 404/, + ); + + assert.strictEqual(webfingerSettled, true); +}); + +test("mixedRunner - rejects unknown children", async () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "mixed", + type: "mixed", + mix: [{ scenario: "missing", weight: 1 }], + }], + }).scenarios; + + await assert.rejects( + async () => + await mixedRunner.run({ + scenario: scenarios[0], + scenarios, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + }), + /unknown mixed child/, + ); +}); + +test("mixedRunner.validate - rejects unknown children with suite context", () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "mixed", + type: "mixed", + mix: [{ scenario: "missing", weight: 1 }], + }], + }).scenarios; + + assert.throws( + () => mixedRunner.validate?.(scenarios[0], { scenarios }), + /unknown mixed child/, + ); +}); + +test("mixedRunner.validate - rejects ambiguous child names", () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [ + { + name: "mixed", + type: "mixed", + mix: [{ scenario: "lookup", weight: 1 }], + }, + { + name: "lookup", + type: "webfinger", + }, + { + name: "lookup", + type: "actor", + recipient: "http://target.test/users/alice", + }, + ], + }).scenarios; + + assert.throws( + () => mixedRunner.validate?.(scenarios[0], { scenarios }), + /ambiguous mixed child/, + ); +}); + +test("mixedRunner.validate - rejects nested mixed children with suite context", () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [ + { + name: "outer", + type: "mixed", + mix: [{ scenario: "inner", weight: 1 }], + }, + { + name: "inner", + type: "mixed", + mix: [{ scenario: "lookup", weight: 1 }], + }, + { + name: "lookup", + type: "webfinger", + }, + ], + }).scenarios; + + assert.throws( + () => mixedRunner.validate?.(scenarios[0], { scenarios }), + /nested mixed/, + ); +}); + +test("mixedRunner.validate - rejects too-small closed load", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "mixed", + type: "mixed", + load: { concurrency: 1 }, + mix: [ + { scenario: "one", weight: 1 }, + { scenario: "two", weight: 1 }, + ], + }], + }).scenarios[0]; + + assert.throws(() => mixedRunner.validate?.(scenario), /concurrency/); +}); + +test("mixedRunner.validate - rejects server metric expectations", () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "mixed", + type: "mixed", + mix: [{ scenario: "fanout", weight: 1 }], + expect: { "queueDrain.p95": "< 1s" }, + }], + }).scenarios[0]; + + assert.throws( + () => mixedRunner.validate?.(scenario), + /server-side expectations/, + ); +}); + +test("mixedRunner.validate - rejects ambiguous target queue observation", () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [ + { + name: "delivery", + type: "inbox", + recipient: "http://target.test/users/alice", + }, + { + name: "fanout", + type: "fanout", + sender: "alice", + }, + { + name: "mixed", + type: "mixed", + mix: [ + { scenario: "delivery", weight: 1 }, + { scenario: "fanout", weight: 1 }, + ], + }, + ], + }).scenarios; + + assert.throws( + () => mixedRunner.validate?.(scenarios[2], { scenarios }), + /target queue counters/, + ); +}); + +test("mixedRunner.validate - rejects remote failure with queue producers", () => { + const scenarios = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [ + { + name: "delivery", + type: "inbox", + recipient: "http://target.test/users/alice", + }, + { + name: "remote-failure", + type: "failure", + sender: "alice", + fault: "remote-404", + }, + { + name: "mixed", + type: "mixed", + mix: [ + { scenario: "delivery", weight: 1 }, + { scenario: "remote-failure", weight: 1 }, + ], + }, + ], + }).scenarios; + + assert.throws( + () => mixedRunner.validate?.(scenarios[2], { scenarios }), + /target queue counters/, + ); +}); + +test("mergeMeasurements - merges latency histograms", () => { + const measurement = mergeMeasurements([ + fakeMeasurement(Array.from({ length: 99 }, () => 1)), + fakeMeasurement([1000]), + ]); + + assert.ok(measurement.client.latencyMs.p50 < 10); + assert.strictEqual(measurement.client.latencyMs.max, 1000); + assert.strictEqual(measurement.histogram?.count, 100); +}); + +test("mergeMeasurements - keeps delivery throughput separate from reads", () => { + const measurement = mergeMeasurements([ + fakeMeasurement([1], { throughputPerSec: 100 }), + fakeMeasurement([1], { + throughputPerSec: 0, + deliveryThroughputPerSec: 0, + }), + ]); + + assert.strictEqual(measurement.throughputPerSec, 100); + assert.strictEqual(measurement.deliveryThroughputPerSec, 0); + const { passed, results } = evaluateExpect( + { deliveryThroughput: ">= 1/s" }, + measurement, + ); + assert.strictEqual(passed, false); + assert.strictEqual(results[0].actual, 0); +}); + +function fakeMeasurement( + samples: readonly number[], + overrides: Partial = {}, +): ScenarioMeasurement { + const histogram = new LogLinearHistogram(); + for (const sample of samples) histogram.record(sample); + return { + requests: { + total: samples.length, + ok: samples.length, + failed: 0, + successRate: 1, + }, + throughputPerSec: samples.length, + client: { + latencyMs: { + p50: histogram.percentile(50), + p95: histogram.percentile(95), + p99: histogram.percentile(99), + mean: histogram.mean, + max: histogram.max, + }, + }, + server: null, + errors: [], + histogram: histogram.toJSON(), + ...overrides, + }; +} + +function json(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/jrd+json" }, + }); +} diff --git a/packages/cli/src/bench/scenarios/mixed.ts b/packages/cli/src/bench/scenarios/mixed.ts new file mode 100644 index 000000000..e0c00b288 --- /dev/null +++ b/packages/cli/src/bench/scenarios/mixed.ts @@ -0,0 +1,425 @@ +/** + * The `mixed` scenario runner. + * @since 2.3.0 + * @module + */ + +import { actorRunner } from "./actor.ts"; +import { failureRunner } from "./failure.ts"; +import { fanoutRunner } from "./fanout.ts"; +import { inboxRunner } from "./inbox.ts"; +import { objectRunner } from "./object.ts"; +import { webfingerRunner } from "./webfinger.ts"; +import { LogLinearHistogram } from "../metrics/histogram.ts"; +import type { LoadModel, ResolvedScenario } from "../scenario/normalize.ts"; +import type { ErrorBucket, LatencyMs } from "../result/model.ts"; +import type { ScenarioMeasurement } from "../result/build.ts"; +import type { ScenarioType } from "../scenario/types.ts"; +import type { RunContext, ScenarioRunner, ValidateContext } from "./runner.ts"; + +/** The `mixed` scenario runner. */ +export const mixedRunner: ScenarioRunner = { + validate(scenario, context?: ValidateContext): void { + if (scenario.raw.mix == null || scenario.raw.mix.length < 1) { + throw new Error( + `Scenario "${scenario.name}": mixed requires at least one mix entry.`, + ); + } + for (const entry of scenario.raw.mix) { + if (entry.weight <= 0) { + throw new Error( + `Scenario "${scenario.name}": mix entry ${entry.scenario} has a ` + + `non-positive weight.`, + ); + } + } + if ( + scenario.load.kind === "closed" && + scenario.load.concurrency < scenario.raw.mix.length + ) { + throw new Error( + `Scenario "${scenario.name}": closed-loop mixed load needs at least ` + + "one concurrency slot per mix entry.", + ); + } + const serverExpectation = Object.keys(scenario.expect).find( + isServerExpectation, + ); + if (serverExpectation != null) { + throw new Error( + `Scenario "${scenario.name}": mixed server-side expectations are ` + + `not supported (${JSON.stringify(serverExpectation)}).`, + ); + } + if (context?.scenarios != null) { + const children = childScenarios(scenario, context.scenarios); + for (const child of children) { + runnerForChild(child.type).validate?.(child, context); + } + validateTargetQueueObservation(scenario, children); + } + }, + + async run(context: RunContext) { + this.validate?.(context.scenario, { scenarios: context.scenarios }); + if (context.scenarios == null) { + throw new Error( + "The mixed scenario requires the resolved scenario list.", + ); + } + const children = childScenarios(context.scenario, context.scenarios); + const fetchImpl = limitedFetch( + context.fetch ?? fetch, + context.scenario.load.maxInFlight, + ); + const results = await Promise.allSettled( + children.map((child) => + runnerForChild(child.type).run({ + ...context, + scenario: child, + fetch: fetchImpl, + assertDestinationAllowed: (url, scenario) => + context.assertDestinationAllowed?.(url, scenario ?? child), + assertReadDestinationAllowed: (url, scenario) => + context.assertReadDestinationAllowed?.(url, scenario ?? child), + assertActorlessDestinationAllowed: (url, scenario) => + context.assertActorlessDestinationAllowed?.(url, scenario ?? child), + }) + ), + ); + const rejected = results.find((result) => result.status === "rejected"); + if (rejected != null) throw rejected.reason; + const measurements = results + .filter((result): result is PromiseFulfilledResult => + result.status === "fulfilled" + ) + .map((result) => result.value); + return mergeMeasurements(measurements); + }, +}; + +function isServerExpectation(metric: string): boolean { + return metric.startsWith("signatureVerification.") || + metric.startsWith("queueDrain."); +} + +function validateTargetQueueObservation( + scenario: ResolvedScenario, + children: readonly ResolvedScenario[], +): void { + const observers = children.filter(observesTargetQueue); + if (observers.length < 1) return; + const producers = children.filter(producesTargetQueue); + if (producers.length <= 1) return; + throw new Error( + `Scenario "${scenario.name}": mixed scenarios cannot run queue-observing ` + + `children (${observers.map((child) => child.name).join(", ")}) ` + + `concurrently with other target queue producers because target queue ` + + `counters are not scoped per child.`, + ); +} + +function observesTargetQueue(scenario: ResolvedScenario): boolean { + return scenario.type === "fanout" || + (scenario.type === "failure" && faultsOf(scenario).some(isRemoteFault)); +} + +function producesTargetQueue(scenario: ResolvedScenario): boolean { + return scenario.type === "inbox" || scenario.type === "fanout" || + (scenario.type === "failure" && faultsOf(scenario).some(isRemoteFault)); +} + +function faultsOf(scenario: ResolvedScenario): readonly string[] { + return scenario.faults.length < 1 ? ["remote-404"] : scenario.faults; +} + +function isRemoteFault(fault: string): boolean { + return fault === "remote-404" || fault === "remote-410" || + fault === "slow-inbox" || fault === "network-error"; +} + +function childScenarios( + scenario: ResolvedScenario, + scenarios: readonly ResolvedScenario[], +): ResolvedScenario[] { + const entries = scenario.raw.mix ?? []; + const totalWeight = entries.reduce((sum, entry) => sum + entry.weight, 0); + const closedLoads = scenario.load.kind === "closed" + ? scaledClosedConcurrencies( + scenario.load.concurrency, + entries.map((entry) => entry.weight), + ) + : undefined; + return entries.map((entry, index) => { + const children = scenarios.filter((candidate) => + candidate.name === entry.scenario + ); + const child = children[0]; + if (child == null) { + throw new Error( + `Scenario "${scenario.name}": unknown mixed child ` + + `${JSON.stringify(entry.scenario)}.`, + ); + } + if (children.length > 1) { + throw new Error( + `Scenario "${scenario.name}": ambiguous mixed child ` + + `${JSON.stringify(entry.scenario)} matches ${children.length} ` + + "scenarios.", + ); + } + if (child.type === "mixed") { + throw new Error( + `Scenario "${scenario.name}": nested mixed scenarios are not ` + + "supported.", + ); + } + const load = scaledLoad( + scenario.load, + entry.weight, + totalWeight, + closedLoads?.[index], + ); + return { + ...child, + name: `${scenario.name}/${child.name}`, + load, + durationMs: scenario.durationMs, + warmupMs: scenario.warmupMs, + signing: scenario.signing, + signatureTimeWindow: scenario.signatureTimeWindow, + expect: {}, + raw: { + ...child.raw, + name: `${scenario.name}/${child.name}`, + load: rawLoad(load), + duration: `${scenario.durationMs}ms`, + warmup: `${scenario.warmupMs}ms`, + signing: scenario.signing, + signatureTimeWindow: scenario.signatureTimeWindow, + expect: {}, + }, + }; + }); +} + +function scaledLoad( + load: LoadModel, + weight: number, + totalWeight: number, + closedConcurrency?: number, +): LoadModel { + // `maxInFlight` is a parent-wide safety cap for mixed scenarios; run() + // enforces it with a shared limiter instead of copying it to each child. + if (load.kind === "open") { + return { + kind: "open", + ratePerSec: load.ratePerSec * (weight / totalWeight), + arrival: load.arrival, + }; + } + return { + kind: "closed", + concurrency: closedConcurrency ?? + Math.max(1, Math.round(load.concurrency * weight / totalWeight)), + }; +} + +function limitedFetch(fetchImpl: typeof fetch, maxInFlight?: number) { + if (maxInFlight == null) return fetchImpl; + const limiter = createLimiter(maxInFlight); + const limited = async ( + input: RequestInfo | URL, + init?: RequestInit, + ): Promise => { + const release = await limiter.acquire(); + try { + return await fetchImpl(input, init); + } finally { + release(); + } + }; + return limited as typeof fetch; +} + +export function createLimiter(maxInFlight: number): { + acquire(): Promise<() => void>; +} { + if (!Number.isInteger(maxInFlight) || maxInFlight < 1) { + throw new RangeError( + `maxInFlight must be a positive integer; got ${maxInFlight}.`, + ); + } + let active = 0; + const waiters: Array<() => void> = []; + return { + async acquire(): Promise<() => void> { + if (active < maxInFlight) { + active++; + } else { + await new Promise((resolve) => waiters.push(resolve)); + } + let released = false; + return () => { + if (released) return; + released = true; + const next = waiters.shift(); + if (next == null) active--; + else next(); + }; + }, + }; +} + +function scaledClosedConcurrencies( + concurrency: number, + weights: readonly number[], +): number[] { + const totalWeight = weights.reduce((sum, weight) => sum + weight, 0); + const ideal = weights.map((weight) => concurrency * weight / totalWeight); + const allocations = weights.map(() => 1); + for ( + let remaining = concurrency - weights.length; + remaining > 0; + remaining-- + ) { + let best = 0; + for (let i = 1; i < allocations.length; i++) { + if (ideal[i] - allocations[i] > ideal[best] - allocations[best]) { + best = i; + } + } + allocations[best]++; + } + return allocations; +} + +function rawLoad( + load: LoadModel, +): NonNullable { + if (load.kind === "open") { + return { + rate: load.ratePerSec, + arrival: load.arrival, + maxInFlight: load.maxInFlight, + }; + } + return { concurrency: load.concurrency, maxInFlight: load.maxInFlight }; +} + +function runnerForChild(type: ScenarioType): ScenarioRunner { + switch (type) { + case "inbox": + return inboxRunner; + case "webfinger": + return webfingerRunner; + case "actor": + return actorRunner; + case "object": + return objectRunner; + case "fanout": + return fanoutRunner; + case "failure": + return failureRunner; + default: + throw new Error( + `The "${type}" scenario type cannot be used inside a mixed scenario.`, + ); + } +} + +export function mergeMeasurements( + measurements: readonly ScenarioMeasurement[], +): ScenarioMeasurement { + const total = measurements.reduce((sum, m) => sum + m.requests.total, 0); + const ok = measurements.reduce((sum, m) => sum + m.requests.ok, 0); + const histogram = mergeHistograms(measurements); + const deliveryThroughputs = measurements + .map((m) => m.deliveryThroughputPerSec) + .filter((value): value is number => value != null); + return { + requests: { + total, + ok, + failed: total - ok, + successRate: total === 0 ? 1 : ok / total, + }, + throughputPerSec: measurements.reduce( + (sum, m) => sum + m.throughputPerSec, + 0, + ), + ...(deliveryThroughputs.length < 1 ? {} : { + deliveryThroughputPerSec: deliveryThroughputs.reduce( + (sum, value) => sum + value, + 0, + ), + }), + client: { + latencyMs: histogram == null + ? mergeLatencyFallback(measurements) + : latencyFromHistogram(histogram), + }, + server: null, + errors: mergeErrors(measurements), + ...(histogram == null ? {} : { histogram: histogram.toJSON() }), + }; +} + +function mergeHistograms( + measurements: readonly ScenarioMeasurement[], +): LogLinearHistogram | null { + let merged: LogLinearHistogram | null = null; + for (const measurement of measurements) { + if (measurement.histogram == null) return null; + const histogram = LogLinearHistogram.fromJSON(measurement.histogram); + if (merged == null) merged = histogram; + else merged.merge(histogram); + } + return merged; +} + +function latencyFromHistogram(histogram: LogLinearHistogram): LatencyMs { + return { + p50: histogram.percentile(50), + p95: histogram.percentile(95), + p99: histogram.percentile(99), + mean: histogram.mean, + max: histogram.max, + }; +} + +function mergeLatencyFallback( + measurements: readonly ScenarioMeasurement[], +): LatencyMs { + const total = measurements.reduce((sum, m) => sum + m.requests.total, 0); + if (measurements.length < 1) { + return { p50: 0, p95: 0, p99: 0, mean: 0, max: 0 }; + } + return { + p50: Math.max(...measurements.map((m) => m.client.latencyMs.p50)), + p95: Math.max(...measurements.map((m) => m.client.latencyMs.p95)), + p99: Math.max(...measurements.map((m) => m.client.latencyMs.p99)), + mean: total === 0 ? 0 : measurements.reduce( + (sum, m) => sum + m.client.latencyMs.mean * m.requests.total, + 0, + ) / total, + max: Math.max(...measurements.map((m) => m.client.latencyMs.max)), + }; +} + +function mergeErrors( + measurements: readonly ScenarioMeasurement[], +): ErrorBucket[] { + const buckets = new Map(); + for (const measurement of measurements) { + for (const error of measurement.errors) { + const key = `${error.kind}|${error.status ?? ""}|${error.reason}`; + const existing = buckets.get(key); + buckets.set(key, { + ...error, + count: (existing?.count ?? 0) + error.count, + }); + } + } + return [...buckets.values()].sort((a, b) => b.count - a.count); +} diff --git a/packages/cli/src/bench/scenarios/object-discovery.ts b/packages/cli/src/bench/scenarios/object-discovery.ts new file mode 100644 index 000000000..3551c9424 --- /dev/null +++ b/packages/cli/src/bench/scenarios/object-discovery.ts @@ -0,0 +1,328 @@ +/** + * Discovery helpers for actor and object benchmark scenarios. + * @since 2.3.0 + * @module + */ + +import { convertUrlIfHandle } from "../../webfinger/lib.ts"; +import { asList } from "../scenario/coerce.ts"; +import type { ObjectSource } from "../scenario/types.ts"; + +const ACTIVITY_JSON_ACCEPT = "application/activity+json, application/ld+json"; +const WEBFINGER_ACCEPT = "application/jrd+json, application/json"; +const MAX_COLLECTION_CRAWL_PAGES = 100; +const MAX_ACTIVITY_UNWRAP_DEPTH = 10; +const ACTIVITY_WRAPPER_TYPES = new Set([ + "Accept", + "Add", + "Announce", + "Create", + "Delete", + "Dislike", + "Flag", + "Ignore", + "Invite", + "Join", + "Leave", + "Like", + "Listen", + "Move", + "Offer", + "Question", + "Read", + "Reject", + "Remove", + "TentativeAccept", + "TentativeReject", + "Travel", + "Undo", + "Update", + "View", +]); + +/** + * Options for resolving actor URLs from recipients. + * @property target The benchmark target base URL. + * @property fetch Fetch implementation used for WebFinger discovery. + * @property assertReadDestinationAllowed Optional gate for discovered read URLs. + */ +export interface ActorUrlOptions { + readonly target: URL; + readonly fetch?: typeof fetch; + readonly assertReadDestinationAllowed?: (url: URL) => void | Promise; +} + +/** + * Options for resolving object URLs from source definitions. + * @property source The explicit object URL list or crawl source to resolve. + */ +export interface ObjectUrlOptions extends ActorUrlOptions { + readonly source: ObjectSource | undefined; +} + +/** Resolves scenario recipients into actor document URLs. */ +export async function actorUrlsFromRecipients( + recipients: readonly string[], + options: ActorUrlOptions, +): Promise { + const urls: URL[] = []; + for (const recipient of recipients) { + urls.push(await actorUrlFromRecipient(recipient, options)); + } + return urls; +} + +/** Resolves object scenario sources into object URLs. */ +export async function objectUrlsFromSource( + options: ObjectUrlOptions, +): Promise { + const { source } = options; + if (source == null) return []; + if (typeof source === "string" || Array.isArray(source)) { + return asList(source).map((url) => new URL(url)); + } + const limit = source.limit ?? 100; + const types = new Set(asList(source.type)); + const urls: URL[] = []; + for (const seed of asList(source.seed)) { + const actorUrl = await actorUrlFromRecipient(seed, options); + await options.assertReadDestinationAllowed?.(actorUrl); + const actor = await fetchJson(actorUrl, options.fetch); + for (const collectionName of asList(source.collection ?? "outbox")) { + const collectionUrl = propertyUrl(actor, collectionName, actorUrl); + if (collectionUrl == null) continue; + for await ( + const objectUrl of crawlCollection(collectionUrl, { + fetch: options.fetch, + assertReadDestinationAllowed: options.assertReadDestinationAllowed, + types, + limit: limit - urls.length, + }) + ) { + urls.push(objectUrl); + if (urls.length >= limit) return urls; + } + } + } + return urls; +} + +async function actorUrlFromRecipient( + recipient: string, + options: ActorUrlOptions, +): Promise { + const identifier = convertUrlIfHandle(recipient); + if (identifier.protocol !== "acct:") return identifier; + const url = new URL("/.well-known/webfinger", options.target); + url.searchParams.set("resource", identifier.href); + const jrd = await fetchJson(url, options.fetch, WEBFINGER_ACCEPT); + const links = Array.isArray(jrd.links) ? jrd.links : []; + const self = links.find((link) => + isRecord(link) && link.rel === "self" && typeof link.href === "string" + ); + if (!isRecord(self) || typeof self.href !== "string") { + throw new Error(`WebFinger response for ${recipient} has no self link.`); + } + return new URL(self.href); +} + +async function* crawlCollection( + start: URL, + options: { + readonly fetch?: typeof fetch; + readonly assertReadDestinationAllowed?: (url: URL) => void | Promise; + readonly types: ReadonlySet; + readonly limit: number; + }, +): AsyncGenerator { + let next: URL | null = start; + let remaining = options.limit; + let pages = 0; + const visited = new Set(); + while (next != null && remaining > 0 && pages < MAX_COLLECTION_CRAWL_PAGES) { + if (visited.has(next.href)) return; + visited.add(next.href); + await options.assertReadDestinationAllowed?.(next); + const page = await fetchJson(next, options.fetch); + pages++; + const items = arrayProperty(page, "orderedItems") ?? + arrayProperty(page, "items") ?? []; + for (const item of items) { + const url = await objectUrl(item, { + base: next, + fetch: options.fetch, + assertReadDestinationAllowed: options.assertReadDestinationAllowed, + types: options.types, + }); + if (url == null) continue; + yield url; + remaining--; + if (remaining <= 0) return; + } + const first = propertyUrl(page, "first", next); + const following = propertyUrl(page, "next", next); + next = following ?? (next.href === start.href ? first : null); + } +} + +async function fetchJson( + url: URL, + fetchImpl: typeof fetch = fetch, + accept = ACTIVITY_JSON_ACCEPT, +): Promise> { + const response = await fetchImpl( + new Request(url, { + headers: { accept }, + redirect: "manual", + }), + ); + if (!response.ok) { + await response.arrayBuffer().catch(() => {}); + throw new Error(`Failed to fetch ${url.href}: HTTP ${response.status}.`); + } + let json: unknown; + try { + json = await response.json(); + } catch (error) { + throw new Error(`Failed to parse JSON from ${url.href}: ${error}`); + } + if (!isRecord(json)) { + throw new Error(`Expected ${url.href} to return a JSON object.`); + } + return json; +} + +async function objectUrl( + item: unknown, + options: { + readonly base: URL; + readonly fetch?: typeof fetch; + readonly assertReadDestinationAllowed?: (url: URL) => void | Promise; + readonly types: ReadonlySet; + }, +): Promise { + for (const candidate of objectCandidates(item)) { + if (typeof candidate === "string") { + const url = safeUrl(candidate, options.base); + if (url == null) continue; + if (options.types.size < 1) return url; + const typedUrl = await typedReferencedObjectUrl(url, options); + if (typedUrl != null) return typedUrl; + continue; + } + if (!isRecord(candidate)) continue; + if ( + options.types.size > 0 && + !matchesType(candidate.type, options.types) + ) { + continue; + } + const url = propertyUrl(candidate, "id", options.base); + if (url != null) return url; + } + return null; +} + +async function typedReferencedObjectUrl( + url: URL, + options: { + readonly fetch?: typeof fetch; + readonly assertReadDestinationAllowed?: (url: URL) => void | Promise; + readonly types: ReadonlySet; + }, + seen: Set = new Set(), + depth = 0, +): Promise { + if (depth > MAX_ACTIVITY_UNWRAP_DEPTH) return null; + if (seen.has(url.href)) return null; + seen.add(url.href); + await options.assertReadDestinationAllowed?.(url); + let object: Record; + try { + object = await fetchJson(url, options.fetch); + } catch { + return null; + } + for (const candidate of objectCandidates(object)) { + if (typeof candidate === "string") { + const candidateUrl = safeUrl(candidate, url); + if (candidateUrl == null) continue; + const typedUrl = await typedReferencedObjectUrl( + candidateUrl, + options, + seen, + depth + 1, + ); + if (typedUrl != null) return typedUrl; + continue; + } + if (!isRecord(candidate)) continue; + if (!matchesType(candidate.type, options.types)) continue; + return propertyUrl(candidate, "id", url) ?? url; + } + return null; +} + +function objectCandidates( + item: unknown, + depth = 0, + seen: WeakSet = new WeakSet(), +): unknown[] { + if (depth > MAX_ACTIVITY_UNWRAP_DEPTH) return []; + if (!isRecord(item) || !matchesType(item.type, ACTIVITY_WRAPPER_TYPES)) { + return [item]; + } + if (seen.has(item)) return []; + seen.add(item); + const object = item.object; + if (object == null) return []; + if (Array.isArray(object)) { + return object.flatMap((entry) => + entry == null ? [] : objectCandidates(entry, depth + 1, seen) + ); + } + return objectCandidates(object, depth + 1, seen); +} + +function matchesType( + type: unknown, + expected: ReadonlySet, +): boolean { + if (typeof type === "string") return expected.has(type); + return Array.isArray(type) && + type.some((item) => typeof item === "string" && expected.has(item)); +} + +function propertyUrl( + object: Record, + key: string, + base?: URL, +): URL | null { + const value = object[key]; + if (typeof value === "string") return safeUrl(value, base); + if (isRecord(value)) { + if (typeof value.href === "string") return safeUrl(value.href, base); + if (typeof value.id === "string") return safeUrl(value.id, base); + } + return null; +} + +function arrayProperty( + object: Record, + key: string, +): unknown[] | null { + const value = object[key]; + return Array.isArray(value) ? value : null; +} + +function isRecord(value: unknown): value is Record { + return value != null && typeof value === "object" && !Array.isArray(value); +} + +function safeUrl(value: string, base?: URL): URL | null { + try { + return new URL(value, base); + } catch { + return null; + } +} diff --git a/packages/cli/src/bench/scenarios/object.test.ts b/packages/cli/src/bench/scenarios/object.test.ts new file mode 100644 index 000000000..c12a77649 --- /dev/null +++ b/packages/cli/src/bench/scenarios/object.test.ts @@ -0,0 +1,1343 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { serve } from "srvx"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import type { Suite } from "../scenario/types.ts"; +import { objectRunner } from "./object.ts"; + +async function spawnObjectTarget() { + let objectGets = 0; + const server = serve({ + port: 0, + hostname: "127.0.0.1", + silent: true, + fetch(request: Request): Response { + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "Person", + id: new URL("/users/alice", url).href, + outbox: new URL("/users/alice/outbox", url).href, + }); + } + if (url.pathname === "/users/alice/outbox") { + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: "OrderedCollection", + id: url.href, + orderedItems: [ + { + type: "Note", + id: new URL("/objects/1", url).href, + content: "one", + }, + { + type: "Article", + id: new URL("/objects/2", url).href, + content: "two", + }, + ], + }); + } + if (url.pathname.startsWith("/objects/")) { + objectGets++; + return json({ + "@context": "https://www.w3.org/ns/activitystreams", + type: url.pathname.endsWith("/1") ? "Note" : "Article", + id: url.href, + content: "object", + }); + } + return new Response("Not found", { status: 404 }); + }, + }); + await server.ready(); + return { + url: new URL(server.url!), + objectGets: () => objectGets, + close: () => server.close(true), + }; +} + +test("objectRunner - fetches explicit object URLs", async () => { + const target = await spawnObjectTarget(); + try { + const suite: Suite = { + version: 1, + target: target.url.href, + scenarios: [{ + name: "object", + type: "object", + source: [ + new URL("/objects/1", target.url).href, + new URL("/objects/2", target.url).href, + ], + load: { concurrency: 2 }, + duration: "80ms", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await objectRunner.run({ + scenario, + target: target.url, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(target.objectGets() > 0); + } finally { + await target.close(); + } +}); + +test("objectRunner - crawls actor collections before fetching objects", async () => { + const target = await spawnObjectTarget(); + try { + const suite: Suite = { + version: 1, + target: target.url.href, + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: new URL("/users/alice", target.url).href, + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "80ms", + }], + }; + const scenario = normalizeSuite(suite).scenarios[0]; + const measurement = await objectRunner.run({ + scenario, + target: target.url, + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(target.objectGets() > 0); + } finally { + await target.close(); + } +}); + +test("objectRunner - unwraps activities while crawling object sources", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + id: "http://target.test/activities/create-1", + object: { + type: "Note", + id: "http://target.test/objects/1", + }, + }], + })); + } + if (url.pathname === "/objects/1") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/1"); +}); + +test("objectRunner - recursively unwraps nested activities", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Announce", + id: "http://target.test/activities/announce-1", + object: { + type: "Create", + id: "http://target.test/activities/create-1", + object: { + type: "Note", + id: "http://target.test/objects/1", + }, + }, + }], + })); + } + if (url.pathname === "/objects/1") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/1"); +}); + +test("objectRunner - selects matching objects from activity arrays", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + id: "http://target.test/activities/create-1", + object: [ + { type: "Article", id: "http://target.test/objects/article" }, + "http://target.test/objects/url-only", + { type: "Note", id: "http://target.test/objects/note" }, + ], + }], + })); + } + if (url.pathname === "/objects/note") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/note"); +}); + +test("objectRunner - unwraps fetched activity references before filtering", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: ["http://target.test/activities/create-1"], + })); + } + if (url.pathname === "/activities/create-1") { + return Promise.resolve(json({ + id: url.href, + type: "Create", + object: { + type: "Note", + id: "http://target.test/objects/1", + }, + })); + } + if (url.pathname === "/objects/1") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/1"); +}); + +test("objectRunner - skips cyclic fetched activity references", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let activityFetches = 0; + + await assert.rejects( + async () => + await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ + allowPrivateAddress: true, + }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: ["http://target.test/activities/create-1"], + })); + } + if (url.pathname === "/activities/create-1") { + activityFetches++; + return Promise.resolve(json({ + id: url.href, + type: "Create", + object: "http://target.test/activities/create-1", + })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }), + /did not resolve any URLs/, + ); + + assert.strictEqual(activityFetches, 1); +}); + +test("objectRunner - limits deep fetched activity reference chains", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const fetchedActivities: number[] = []; + + await assert.rejects( + async () => + await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ + allowPrivateAddress: true, + }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: ["http://target.test/activities/0"], + })); + } + const match = /^\/activities\/(\d+)$/.exec(url.pathname); + if (match != null) { + const index = Number(match[1]); + fetchedActivities.push(index); + return Promise.resolve(json({ + id: url.href, + type: "Create", + object: index < 12 + ? `http://target.test/activities/${index + 1}` + : { + type: "Note", + id: "http://target.test/objects/deep", + }, + })); + } + if (url.pathname === "/objects/deep") { + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }), + /did not resolve any URLs/, + ); + + assert.deepStrictEqual( + fetchedActivities, + Array.from({ length: 11 }, (_, i) => i), + ); +}); + +test("objectRunner - prefers unwrapped object URLs without type filters", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedActivity = false; + let fetchedObject = false; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + id: "http://target.test/activities/create-1", + object: "http://target.test/objects/1", + }], + })); + } + if (url.pathname === "/activities/create-1") { + fetchedActivity = true; + return Promise.resolve(json({ id: url.href, type: "Create" })); + } + if (url.pathname === "/objects/1") { + fetchedObject = true; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObject, true); + assert.strictEqual(fetchedActivity, false); +}); + +test("objectRunner - skips wrapper activities without objects", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedActivity = false; + + await assert.rejects( + async () => + objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + id: "http://target.test/activities/create-1", + }], + })); + } + if (url.pathname === "/activities/create-1") { + fetchedActivity = true; + return Promise.resolve(json({ id: url.href, type: "Create" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }), + /did not resolve any URLs/, + ); + assert.strictEqual(fetchedActivity, false); +}); + +test("objectRunner - sends ActivityPub Accept headers during object discovery", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const discoveryAccepts: string[] = []; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + discoveryAccepts.push(request.headers.get("accept") ?? ""); + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + discoveryAccepts.push(request.headers.get("accept") ?? ""); + return Promise.resolve(json({ + id: url.href, + orderedItems: ["http://target.test/objects/1"], + })); + } + if (url.pathname === "/objects/1") { + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.deepStrictEqual(discoveryAccepts, [ + "application/activity+json, application/ld+json", + "application/activity+json, application/ld+json", + ]); +}); + +test("objectRunner - dereferences URL-only items for type filters", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const fetched: string[] = []; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + fetched.push(url.href); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [ + "http://target.test/objects/article", + { + type: "Create", + object: "http://target.test/objects/note", + }, + ], + })); + } + if (url.pathname === "/objects/article") { + return Promise.resolve(json({ + id: url.href, + type: "Article", + })); + } + if (url.pathname === "/objects/note") { + return Promise.resolve(json({ + id: url.href, + type: "Note", + })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.ok(fetched.includes("http://target.test/objects/article")); + assert.ok(fetched.includes("http://target.test/objects/note")); +}); + +test("objectRunner - gates discovery URLs before fetching them", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://public.example/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + + await assert.rejects( + async () => + objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => { + throw new Error("discovery fetch should be gated first"); + }, + assertReadDestinationAllowed: (url) => { + throw new Error(`refused ${url.href}`); + }, + }), + /refused http:\/\/public\.example\/users\/alice/, + ); +}); + +test("objectRunner - gates collection URLs before crawling them", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + + await assert.rejects( + async () => + objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://public.example/outbox", + })); + } + throw new Error("collection fetch should be gated first"); + }, + assertReadDestinationAllowed: (url) => { + if (url.hostname === "public.example") { + throw new Error(`refused ${url.href}`); + } + }, + }), + /refused http:\/\/public\.example\/outbox/, + ); +}); + +test("objectRunner - drains failed discovery responses", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const failed = new Response("forbidden", { status: 403 }); + + await assert.rejects( + async () => + await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => Promise.resolve(failed), + }), + /Failed to fetch http:\/\/target\.test\/users\/alice: HTTP 403/, + ); + assert.strictEqual(failed.bodyUsed, true); +}); + +test("objectRunner - reports invalid discovery JSON with URL context", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + + await assert.rejects( + async () => + await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => Promise.resolve(new Response("not json", { status: 200 })), + }), + /Failed to parse JSON from http:\/\/target\.test\/users\/alice:/, + ); +}); + +test("objectRunner - resolves relative URLs while crawling object sources", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice/", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const fetched: string[] = []; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + fetched.push(url.href); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice/") { + return Promise.resolve(json({ + id: url.href, + outbox: "outbox", + })); + } + if (url.pathname === "/users/alice/outbox" && url.search === "?page=1") { + return Promise.resolve(json({ + id: url.href, + next: "?page=2", + orderedItems: [], + })); + } + if (url.pathname === "/users/alice/outbox" && url.search === "?page=2") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + object: { + type: "Note", + id: "./objects/relative", + }, + }], + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + first: "?page=1", + })); + } + if (url.pathname === "/users/alice/objects/relative") { + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.ok(fetched.includes("http://target.test/users/alice/outbox")); + assert.ok(fetched.includes("http://target.test/users/alice/outbox?page=1")); + assert.ok(fetched.includes("http://target.test/users/alice/outbox?page=2")); + assert.ok( + fetched.includes("http://target.test/users/alice/objects/relative"), + ); +}); + +test("objectRunner - resolves Link object href values while crawling", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice/", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const fetched: string[] = []; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + fetched.push(url.href); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice/") { + return Promise.resolve(json({ + id: url.href, + outbox: { + type: "Link", + href: "outbox", + }, + })); + } + if (url.pathname === "/users/alice/outbox" && url.search === "?page=1") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + object: { + type: "Note", + id: { + type: "Link", + href: "../objects/note", + }, + }, + }], + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + first: { + type: "Link", + href: "?page=1", + }, + })); + } + if (url.pathname === "/users/objects/note") { + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.ok(fetched.includes("http://target.test/users/alice/outbox")); + assert.ok(fetched.includes("http://target.test/users/alice/outbox?page=1")); + assert.ok(fetched.includes("http://target.test/users/objects/note")); +}); + +test("objectRunner - skips malformed URLs while crawling object sources", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [ + "http://[malformed", + { + type: "Note", + id: { + type: "Link", + href: "http://[malformed", + }, + }, + { + type: "Note", + id: "http://target.test/objects/valid", + }, + ], + })); + } + if (url.pathname === "/objects/valid") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/valid"); +}); + +test("objectRunner - continues after malformed activity object URLs", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchedObjectUrl = ""; + + const measurement = await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/users/alice/outbox", + })); + } + if (url.pathname === "/users/alice/outbox") { + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Create", + object: [ + "http://[malformed", + { + type: "Note", + id: "http://target.test/objects/valid", + }, + ], + }], + })); + } + if (url.pathname === "/objects/valid") { + fetchedObjectUrl = url.href; + return Promise.resolve(json({ id: url.href, type: "Note" })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(fetchedObjectUrl, "http://target.test/objects/valid"); +}); + +test("objectRunner - caps object source crawl pages", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: "http://target.test/users/alice", + collection: "outbox", + limit: 1, + type: "Note", + }, + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let pageFetches = 0; + + await assert.rejects( + async () => + await objectRunner.run({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/users/alice") { + return Promise.resolve(json({ + id: url.href, + outbox: "http://target.test/outbox?page=0", + })); + } + if (url.pathname === "/outbox") { + pageFetches++; + const page = Number(url.searchParams.get("page") ?? 0); + return Promise.resolve(json({ + id: url.href, + orderedItems: [{ + type: "Article", + id: `http://target.test/articles/${page}`, + }], + next: `?page=${page + 1}`, + })); + } + return Promise.resolve(new Response("not found", { status: 404 })); + }, + }), + /did not resolve any URLs/, + ); + + assert.strictEqual(pageFetches, 100); +}); + +test("objectRunner.validate - rejects malformed object source URLs", () => { + const explicit = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object", + type: "object", + source: "objects/1", + }], + }).scenarios[0]; + assert.throws( + () => objectRunner.validate?.(explicit), + /invalid object source URL/, + ); + + const crawl = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object-crawl", + type: "object", + source: { + seed: ["http://target.test/users/alice", "users/bob"], + }, + }], + }).scenarios[0]; + assert.throws( + () => objectRunner.validate?.(crawl), + /invalid object source seed URL/, + ); +}); + +test("objectRunner.validate - rejects non-fetchable direct object source URLs", () => { + for ( + const source of [ + "ftp://target.test/objects/1", + "http://user:pass@target.test/objects/1", + ] + ) { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "object", + type: "object", + source, + }], + }).scenarios[0]; + + assert.throws( + () => objectRunner.validate?.(scenario), + /object source URL must be a bare http\(s\) URL/, + ); + } +}); + +function json(body: unknown): Response { + return new Response(JSON.stringify(body), { + headers: { "content-type": "application/activity+json" }, + }); +} diff --git a/packages/cli/src/bench/scenarios/object.ts b/packages/cli/src/bench/scenarios/object.ts new file mode 100644 index 000000000..5cad0dc0e --- /dev/null +++ b/packages/cli/src/bench/scenarios/object.ts @@ -0,0 +1,71 @@ +/** + * The `object` scenario runner. + * @since 2.3.0 + * @module + */ + +import { convertUrlIfHandle } from "../../webfinger/lib.ts"; +import { asList } from "../scenario/coerce.ts"; +import { objectUrlsFromSource } from "./object-discovery.ts"; +import { runReadLoad } from "./read.ts"; +import { + assertBareHttpUrl, + isBareHttpUrl, + type RunContext, + type ScenarioRunner, +} from "./runner.ts"; + +/** The `object` scenario runner. */ +export const objectRunner: ScenarioRunner = { + validate(scenario): void { + const { source } = scenario; + if (source == null) return; + if (typeof source === "string" || Array.isArray(source)) { + for (const url of asList(source)) { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + throw new Error( + `Scenario "${scenario.name}": invalid object source URL ` + + `${JSON.stringify(url)}.`, + ); + } + assertBareHttpUrl(scenario.name, "object source URL", parsed); + } + return; + } + for (const seed of asList(source.seed)) { + let url: URL; + try { + url = convertUrlIfHandle(seed); + } catch { + throw new Error( + `Scenario "${scenario.name}": invalid object source seed URL ` + + `${JSON.stringify(seed)}.`, + ); + } + if (url.protocol !== "acct:" && !isBareHttpUrl(url)) { + throw new Error( + `Scenario "${scenario.name}": object source seed must be an acct: ` + + `handle or a bare http(s) URL with a host and no credentials; ` + + `got ${JSON.stringify(url.href)}.`, + ); + } + } + }, + + async run(context: RunContext) { + this.validate?.(context.scenario); + const urls = await objectUrlsFromSource({ + source: context.scenario.source, + target: context.target, + fetch: context.fetch, + assertReadDestinationAllowed: context.assertReadDestinationAllowed, + }); + return await runReadLoad(context, { + urls, + authenticated: context.scenario.authenticated, + }); + }, +}; diff --git a/packages/cli/src/bench/scenarios/read.test.ts b/packages/cli/src/bench/scenarios/read.test.ts new file mode 100644 index 000000000..565ddd527 --- /dev/null +++ b/packages/cli/src/bench/scenarios/read.test.ts @@ -0,0 +1,381 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { buildFleet } from "../actor/fleet.ts"; +import { getContextLoader, getDocumentLoader } from "../../docloader.ts"; +import { normalizeSuite } from "../scenario/normalize.ts"; +import { spawnSyntheticServer } from "../server/synthetic.ts"; +import { runReadLoad } from "./read.ts"; + +test("runReadLoad - unauthenticated reads use the read destination gate", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let readGateCalls = 0; + + const measurement = await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertDestinationAllowed: () => { + throw new Error("signed destination gate should not run"); + }, + assertReadDestinationAllowed: () => { + readGateCalls++; + }, + }, { + urls: [new URL("http://remote.test/users/alice")], + authenticated: false, + }); + + assert.strictEqual(readGateCalls, 1); + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); +}); + +test("runReadLoad - gates resolved read URLs concurrently", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let activeGates = 0; + let maxActiveGates = 0; + + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertReadDestinationAllowed: async () => { + activeGates++; + maxActiveGates = Math.max(maxActiveGates, activeGates); + await new Promise((resolve) => setTimeout(resolve, 5)); + activeGates--; + }, + }, { + urls: [ + new URL("http://remote.test/users/alice"), + new URL("http://remote.test/users/bob"), + ], + authenticated: false, + }); + + assert.strictEqual(maxActiveGates, 2); +}); + +test("runReadLoad - limits resolved read URL gate concurrency", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let activeGates = 0; + let maxActiveGates = 0; + + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: (input) => { + const url = new URL(input instanceof Request ? input.url : input); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertReadDestinationAllowed: async () => { + activeGates++; + maxActiveGates = Math.max(maxActiveGates, activeGates); + await new Promise((resolve) => setTimeout(resolve, 5)); + activeGates--; + }, + }, { + urls: Array.from( + { length: 32 }, + (_, i) => new URL(`http://remote.test/users/${i}`), + ), + authenticated: false, + }); + + assert.strictEqual(maxActiveGates, 16); +}); + +test("runReadLoad - stops read URL gates after a gate failure", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const urls = Array.from( + { length: 64 }, + (_, i) => new URL(`http://remote.test/users/${i}`), + ); + let gateCalls = 0; + + await assert.rejects( + async () => + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => Promise.resolve(new Response("{}", { status: 200 })), + assertReadDestinationAllowed: async () => { + gateCalls++; + if (gateCalls === 1) throw new Error("gate failed"); + await new Promise((resolve) => setTimeout(resolve, 1)); + }, + }, { + urls, + authenticated: false, + }), + /gate failed/, + ); + await new Promise((resolve) => setTimeout(resolve, 30)); + + assert.ok(gateCalls < urls.length); +}); + +test("runReadLoad - waits for in-flight read URL gates after a failure", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + const urls = Array.from( + { length: 32 }, + (_, i) => new URL(`http://remote.test/users/${i}`), + ); + let gateCalls = 0; + let delayedGatesSettled = 0; + + await assert.rejects( + async () => + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => Promise.resolve(new Response("{}", { status: 200 })), + assertReadDestinationAllowed: async () => { + gateCalls++; + if (gateCalls === 1) throw new Error("first gate failed"); + await new Promise((resolve) => setTimeout(resolve, 20)); + delayedGatesSettled++; + throw new Error("later gate failed"); + }, + }, { + urls, + authenticated: false, + }), + /first gate failed/, + ); + + assert.ok(delayedGatesSettled > 0); + assert.ok(gateCalls < urls.length); +}); + +test("runReadLoad - rejects invalid read URL schemes before load", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "object", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + let fetchCalls = 0; + let gateCalls = 0; + + for ( + const url of [ + new URL("ftp://remote.test/object"), + new URL("http://user:pass@remote.test/object"), + ] + ) { + await assert.rejects( + async () => + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ + allowPrivateAddress: true, + }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: null, + fetch: () => { + fetchCalls++; + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertReadDestinationAllowed: () => { + gateCalls++; + }, + }, { + urls: [url], + authenticated: false, + }), + /read URL must be a bare http\(s\) URL/, + ); + } + + assert.strictEqual(fetchCalls, 0); + assert.strictEqual(gateCalls, 0); +}); + +test("runReadLoad - authenticated reads support presign mode", async () => { + let fleet: Awaited> | undefined; + try { + fleet = await spawnSyntheticServer( + await buildFleet([{ + count: 1, + signatureStandards: ["draft-cavage-http-signatures-12"], + }]), + ); + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + authenticated: true, + signing: "presign", + load: { rate: "20/s" }, + duration: "50ms", + }], + }).scenarios[0]; + let signedGets = 0; + + const measurement = await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet, + fetch: (input) => { + const request = input instanceof Request ? input : new Request(input); + const url = new URL(request.url); + if (url.pathname === "/.well-known/fedify/bench/stats") { + return Promise.resolve(new Response("not found", { status: 404 })); + } + if ( + request.headers.has("authorization") || + request.headers.has("signature") + ) { + signedGets++; + } + return Promise.resolve(new Response("{}", { status: 200 })); + }, + assertDestinationAllowed: () => {}, + assertReadDestinationAllowed: () => { + throw new Error("authenticated reads should use the signed gate"); + }, + }, { + urls: [new URL("http://remote.test/users/alice")], + authenticated: true, + }); + + assert.ok(measurement.requests.total > 0); + assert.strictEqual(measurement.requests.successRate, 1); + assert.ok(signedGets > 0); + } finally { + await fleet?.close(); + } +}); + +test("runReadLoad - reports missing authenticated read keys clearly", async () => { + const scenario = normalizeSuite({ + version: 1, + target: "http://target.test/", + scenarios: [{ + name: "read", + type: "actor", + authenticated: true, + signing: "pipeline", + load: { concurrency: 1 }, + duration: "25ms", + }], + }).scenarios[0]; + + await assert.rejects( + async () => + await runReadLoad({ + scenario, + target: new URL("http://target.test/"), + documentLoader: await getDocumentLoader({ allowPrivateAddress: true }), + contextLoader: await getContextLoader({ allowPrivateAddress: true }), + allowPrivateAddress: true, + fleet: { + actors: [{ + id: new URL("http://synthetic.test/actors/0"), + index: 0, + keys: undefined, + }], + url: new URL("http://synthetic.test/"), + close: async () => {}, + } as unknown as Awaited>, + fetch: () => Promise.resolve(new Response("{}", { status: 200 })), + assertDestinationAllowed: () => {}, + }, { + urls: [new URL("http://remote.test/users/alice")], + authenticated: true, + }), + /Actor is missing the RSA key required for authenticated fetch signing/, + ); +}); diff --git a/packages/cli/src/bench/scenarios/read.ts b/packages/cli/src/bench/scenarios/read.ts new file mode 100644 index 000000000..0c7f24b55 --- /dev/null +++ b/packages/cli/src/bench/scenarios/read.ts @@ -0,0 +1,184 @@ +/** + * Shared helpers for read-only benchmark scenarios. + * @since 2.3.0 + * @module + */ + +import { signRequest } from "@fedify/fedify"; +import { runLoad } from "../load/generator.ts"; +import { aggregateSamples } from "../metrics/aggregate.ts"; +import { + diffSnapshots, + fetchServerSnapshot, + type ServerSnapshot, + snapshotToMetrics, +} from "../metrics/stats-client.ts"; +import type { SyntheticActor } from "../server/synthetic.ts"; +import { + createSigningPipeline, + type SigningPipeline, +} from "../signing/pipeline.ts"; +import { + assertBareHttpUrl, + estimateTotal, + loadPlanOf, + measuredWindowMs, + type RunContext, + sendRequest, + withMeasuredWindowStart, +} from "./runner.ts"; + +const READ_GATE_CONCURRENCY = 16; + +/** Options for {@link runReadLoad}. */ +export interface ReadLoadOptions { + /** URLs to GET during the measured load. */ + readonly urls: readonly URL[]; + /** Whether GETs should be authenticated with HTTP Signatures. */ + readonly authenticated?: boolean; +} + +/** + * Runs a read-only GET workload and aggregates client/server measurements. + * @param context The scenario run context. + * @param options Read workload options. + * @returns The scenario measurement. + */ +export async function runReadLoad( + context: RunContext, + options: ReadLoadOptions, +) { + if (options.urls.length < 1) { + throw new Error( + `Scenario "${context.scenario.name}" did not resolve any URLs to fetch.`, + ); + } + const fetchImpl = context.fetch ?? fetch; + const actors = context.fleet?.actors ?? []; + if (options.authenticated && actors.length < 1) { + throw new Error( + `Scenario "${context.scenario.name}" requires the synthetic actor server ` + + "for authenticated fetches.", + ); + } + for (const url of options.urls) { + assertBareHttpUrl(context.scenario.name, "read URL", url); + } + await mapWithConcurrency(options.urls, READ_GATE_CONCURRENCY, async (url) => { + if (options.authenticated) { + await context.assertDestinationAllowed?.(url); + } else { + await context.assertReadDestinationAllowed?.(url); + } + }); + + function unsignedRequest(index: number): Request { + const url = options.urls[index % options.urls.length]; + return new Request(url, { + headers: { accept: "application/activity+json, application/ld+json" }, + redirect: "manual", + }); + } + + let pipeline: SigningPipeline | null = null; + if (options.authenticated) { + let signIndex = 0; + pipeline = createSigningPipeline(context.scenario.signing, async () => { + const i = signIndex++; + return await signGetRequest( + unsignedRequest(i), + actors[i % actors.length], + ); + }, { total: estimateTotal(context.scenario) }); + } + + let index = 0; + const rawSend = async () => { + let request: Request; + if (pipeline != null) { + try { + request = await pipeline.next(); + } catch (error) { + return { ok: false, errorKind: "client", reason: String(error) }; + } + } else { + request = unsignedRequest(index++); + } + return await sendRequest(request, fetchImpl); + }; + + let baseline: ServerSnapshot | null = null; + let baselineTaken = false; + const send = withMeasuredWindowStart( + context.scenario.warmupMs, + async () => { + baseline = await fetchServerSnapshot(context.target, fetchImpl); + baselineTaken = true; + }, + rawSend, + ); + try { + await pipeline?.prime(); + const result = await runLoad( + loadPlanOf(context.scenario, context.rng), + send, + context.clock, + ); + const measurement = aggregateSamples(result.samples, { + measuredWindowMs: measuredWindowMs(context.scenario), + includeHistogram: true, + }); + const end = await fetchServerSnapshot(context.target, fetchImpl); + const server = baselineTaken && baseline != null && end != null + ? snapshotToMetrics(diffSnapshots(baseline, end)) + : null; + return { ...measurement, server }; + } finally { + await pipeline?.close(); + } +} + +async function mapWithConcurrency( + items: readonly T[], + concurrency: number, + callback: (item: T) => Promise, +): Promise { + let next = 0; + let firstError: unknown; + let hasError = false; + const workers = Array.from( + { length: Math.min(concurrency, items.length) }, + async () => { + while (next < items.length && !hasError) { + const item = items[next++]; + try { + await callback(item); + } catch (error) { + if (!hasError) { + hasError = true; + firstError = error; + } + } + } + }, + ); + await Promise.all(workers); + if (hasError) throw firstError; +} + +async function signGetRequest( + request: Request, + actor: SyntheticActor, +): Promise { + if (actor.keys?.rsa == null || actor.rsaKeyId == null) { + throw new TypeError( + "Actor is missing the RSA key required for authenticated fetch signing.", + ); + } + return await signRequest( + request, + actor.keys.rsa.privateKey, + actor.rsaKeyId, + { spec: actor.httpStandard }, + ); +} diff --git a/packages/cli/src/bench/scenarios/registry.test.ts b/packages/cli/src/bench/scenarios/registry.test.ts index c42f57e96..628649fd2 100644 --- a/packages/cli/src/bench/scenarios/registry.test.ts +++ b/packages/cli/src/bench/scenarios/registry.test.ts @@ -3,19 +3,19 @@ import test from "node:test"; import type { ScenarioType } from "../scenario/types.ts"; import { runnerFor } from "./registry.ts"; -test("runnerFor - returns the inbox and webfinger runners", () => { +test("runnerFor - returns implemented scenario runners", () => { assert.strictEqual(typeof runnerFor("inbox").run, "function"); assert.strictEqual(typeof runnerFor("webfinger").run, "function"); + assert.strictEqual(typeof runnerFor("actor").run, "function"); + assert.strictEqual(typeof runnerFor("object").run, "function"); + assert.strictEqual(typeof runnerFor("fanout").run, "function"); + assert.strictEqual(typeof runnerFor("failure").run, "function"); + assert.strictEqual(typeof runnerFor("mixed").run, "function"); }); test("runnerFor - throws for scenario types without a runner", () => { const unimplemented: ScenarioType[] = [ - "actor", - "object", - "fanout", "collection", - "failure", - "mixed", ]; for (const type of unimplemented) { assert.throws(() => runnerFor(type), /not implemented/); diff --git a/packages/cli/src/bench/scenarios/registry.ts b/packages/cli/src/bench/scenarios/registry.ts index 2cc09d0fd..53fad1694 100644 --- a/packages/cli/src/bench/scenarios/registry.ts +++ b/packages/cli/src/bench/scenarios/registry.ts @@ -1,15 +1,18 @@ /** * The scenario-runner registry. * - * Only `inbox` and `webfinger` have runners in this version; the other scenario - * types are expressible in the format but not yet executable, so requesting one - * fails with a clear message. + * Only `collection` is still reserved but not executable in this version. * @since 2.3.0 * @module */ import type { ScenarioType } from "../scenario/types.ts"; +import { actorRunner } from "./actor.ts"; +import { failureRunner } from "./failure.ts"; +import { fanoutRunner } from "./fanout.ts"; import { inboxRunner } from "./inbox.ts"; +import { mixedRunner } from "./mixed.ts"; +import { objectRunner } from "./object.ts"; import type { ScenarioRunner } from "./runner.ts"; import { webfingerRunner } from "./webfinger.ts"; @@ -17,6 +20,11 @@ import { webfingerRunner } from "./webfinger.ts"; export const IMPLEMENTED_SCENARIO_TYPES: readonly ScenarioType[] = [ "inbox", "webfinger", + "actor", + "object", + "fanout", + "failure", + "mixed", ]; /** @@ -31,6 +39,16 @@ export function runnerFor(type: ScenarioType): ScenarioRunner { return inboxRunner; case "webfinger": return webfingerRunner; + case "actor": + return actorRunner; + case "object": + return objectRunner; + case "fanout": + return fanoutRunner; + case "failure": + return failureRunner; + case "mixed": + return mixedRunner; default: throw new Error( `The "${type}" scenario type is not implemented in this version of ` + diff --git a/packages/cli/src/bench/scenarios/runner.ts b/packages/cli/src/bench/scenarios/runner.ts index 9770de794..eb1ce970d 100644 --- a/packages/cli/src/bench/scenarios/runner.ts +++ b/packages/cli/src/bench/scenarios/runner.ts @@ -17,6 +17,8 @@ import type { SyntheticServer } from "../server/synthetic.ts"; /** The context a scenario runner needs to execute. */ export interface RunContext { readonly scenario: ResolvedScenario; + /** Every scenario in the resolved suite, for composite runners. */ + readonly scenarios?: readonly ResolvedScenario[]; readonly target: URL; readonly documentLoader: DocumentLoader; readonly contextLoader: DocumentLoader; @@ -29,13 +31,41 @@ export interface RunContext { readonly rng?: Rng; /** Fetch implementation (overridable for tests). */ readonly fetch?: typeof fetch; + /** Host advertised for local benchmark-owned servers. */ + readonly advertiseHost?: string; /** * Gates a resolved load destination (a discovered or explicit inbox URL) * before any load is sent to it, throwing or rejecting if it is not allowed. * The suite `target` is gated by the orchestrator; this covers destinations * that differ from it. Optional so direct runner tests need not supply it. */ - readonly assertDestinationAllowed?: (url: URL) => void | Promise; + readonly assertDestinationAllowed?: ( + url: URL, + scenario?: ResolvedScenario, + ) => void | Promise; + /** + * Gates a resolved read-only destination before unauthenticated GET load is + * sent. Unlike signed inbox delivery, these reads do not require a + * reachable synthetic actor server. + */ + readonly assertReadDestinationAllowed?: ( + url: URL, + scenario?: ResolvedScenario, + ) => void | Promise; + /** + * Gates a destination for benchmark load that does not require remote + * dereferencing of benchmark-owned synthetic actors. + */ + readonly assertActorlessDestinationAllowed?: ( + url: URL, + scenario?: ResolvedScenario, + ) => void | Promise; +} + +/** Context available during runner preflight validation. */ +export interface ValidateContext { + /** Every scenario in the resolved suite, for composite runners. */ + readonly scenarios?: readonly ResolvedScenario[]; } /** A runner for one scenario type. */ @@ -46,7 +76,7 @@ export interface ScenarioRunner { * probe or load. Called during preflight; throwing here surfaces as a * configuration error (exit 2) with the thrown message. */ - validate?(scenario: ResolvedScenario): void; + validate?(scenario: ResolvedScenario, context?: ValidateContext): void; } /** Performs one HTTP send and classifies the result as a send outcome. */ @@ -107,6 +137,43 @@ export function estimateTotal(scenario: ResolvedScenario): number | undefined { return Math.ceil(scenario.load.ratePerSec * (scenario.durationMs / 1000)); } +/** Returns whether a URL is fetchable by benchmark runners without surprises. */ +export function isBareHttpUrl(url: URL): boolean { + return (url.protocol === "http:" || url.protocol === "https:") && + url.hostname !== "" && url.username === "" && url.password === ""; +} + +/** Rejects URLs that are not bare http(s) URLs with a host and no credentials. */ +export function assertBareHttpUrl( + scenarioName: string, + label: string, + url: URL, +): void { + if (isBareHttpUrl(url)) return; + throw new Error( + `Scenario "${scenarioName}": ${label} must be a bare http(s) URL with ` + + `a host and no credentials; got ${JSON.stringify(url.href)}.`, + ); +} + +/** Validates an inbox selector or explicit inbox URL. */ +export function validateInboxSelector( + scenarioName: string, + inbox: string | undefined, +): void { + if (inbox == null || inbox === "shared" || inbox === "personal") return; + let url: URL; + try { + url = new URL(inbox); + } catch { + throw new Error( + `Scenario "${scenarioName}": inbox must be "shared", "personal", or an ` + + `http(s) URL; got ${JSON.stringify(inbox)}.`, + ); + } + assertBareHttpUrl(scenarioName, "inbox URL", url); +} + /** * Wraps a send function so that `onMeasuredWindowStart` runs exactly once, at * the warm-up boundary, and *every* measured request waits for it to settle diff --git a/packages/cli/src/bench/schema.test.ts b/packages/cli/src/bench/schema.test.ts index 647a1c559..e3b33834f 100644 --- a/packages/cli/src/bench/schema.test.ts +++ b/packages/cli/src/bench/schema.test.ts @@ -143,7 +143,11 @@ for (const { name, fileName } of PUBLISHED_SCHEMAS) { published = execFileSync( "git", ["show", `${baseCommit}:schema/bench/${fileName}`], - { cwd: REPO_ROOT, encoding: "utf-8" }, + { + cwd: REPO_ROOT, + encoding: "utf-8", + stdio: ["ignore", "pipe", "ignore"], + }, ); } catch { // Not published at the base (a brand-new version file): nothing to guard. diff --git a/packages/cli/src/bench/schemas.ts b/packages/cli/src/bench/schemas.ts index 3812d6655..82f9e279b 100644 --- a/packages/cli/src/bench/schemas.ts +++ b/packages/cli/src/bench/schemas.ts @@ -9,8 +9,8 @@ * @module */ -import { reportSchemaV1 } from "./result/schema.ts"; -import { scenarioSchemaV1 } from "./scenario/schema.ts"; +import { reportSchemaV1, reportSchemaV2 } from "./result/schema.ts"; +import { scenarioSchemaV1, scenarioSchemaV2 } from "./scenario/schema.ts"; /** A published JSON Schema and where it is hosted. */ export interface PublishedSchema { @@ -26,11 +26,21 @@ export interface PublishedSchema { export const PUBLISHED_SCHEMAS: readonly PublishedSchema[] = [ { name: "scenario", + fileName: "scenario-v2.json", + schema: scenarioSchemaV2 as unknown as Record, + }, + { + name: "scenario-v1", fileName: "scenario-v1.json", schema: scenarioSchemaV1 as unknown as Record, }, { name: "report", + fileName: "report-v2.json", + schema: reportSchemaV2 as unknown as Record, + }, + { + name: "report-v1", fileName: "report-v1.json", schema: reportSchemaV1 as unknown as Record, }, diff --git a/schema/README.md b/schema/README.md index 0ae5a8f86..7bd588534 100644 --- a/schema/README.md +++ b/schema/README.md @@ -6,13 +6,19 @@ Fedify JSON schemas This directory holds the published JSON Schemas (draft 2020-12) for Fedify file formats. It is deployed to by Netlify on every push to the *main* branch; the directory layout maps onto the URL, so -*schema/bench/scenario-v1.json* is served at -. +*schema/bench/scenario-v2.json* is served at +. Current schemas: - - *bench/scenario-v1.json* — the `fedify bench` scenario suite format (input). - - *bench/report-v1.json* — the `fedify bench` report format (output). + - *bench/scenario-v2.json* — the current `fedify bench` scenario suite + format (input). + - *bench/scenario-v1.json* — the version 1 `fedify bench` scenario suite + format (input). + - *bench/report-v2.json* — the current `fedify bench` report format + (output). + - *bench/report-v1.json* — the version 1 `fedify bench` report format + (output). Versioning: append-only and immutable @@ -82,7 +88,7 @@ Editor support Add a schema reference to a scenario file for autocomplete and validation: ~~~~ yaml -# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v1.json +# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v2.json version: 1 target: http://localhost:3000 ~~~~ diff --git a/schema/bench/report-v2.json b/schema/bench/report-v2.json new file mode 100644 index 000000000..3797332e4 --- /dev/null +++ b/schema/bench/report-v2.json @@ -0,0 +1,523 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.fedify.dev/bench/report-v2.json", + "title": "Fedify benchmark report", + "type": "object", + "additionalProperties": false, + "required": [ + "schemaVersion", + "tool", + "environment", + "target", + "startedAt", + "finishedAt", + "suite", + "passed", + "scenarios" + ], + "properties": { + "$schema": { + "type": "string" + }, + "schemaVersion": { + "const": 2 + }, + "tool": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "version" + ], + "properties": { + "name": { + "type": "string" + }, + "version": { + "type": "string" + } + } + }, + "environment": { + "type": "object", + "additionalProperties": false, + "required": [ + "runtime", + "runtimeVersion", + "os", + "cpuCount" + ], + "properties": { + "runtime": { + "type": "string" + }, + "runtimeVersion": { + "type": "string" + }, + "os": { + "type": "string" + }, + "cpuCount": { + "type": "integer", + "minimum": 0 + } + } + }, + "target": { + "type": "object", + "additionalProperties": false, + "required": [ + "url", + "statsAvailable" + ], + "properties": { + "url": { + "type": "string" + }, + "fedifyVersion": { + "type": [ + "string", + "null" + ] + }, + "statsAvailable": { + "type": "boolean" + } + } + }, + "startedAt": { + "type": "string" + }, + "finishedAt": { + "type": "string" + }, + "suite": { + "type": "object", + "additionalProperties": false, + "required": [ + "configHash" + ], + "properties": { + "name": { + "type": "string" + }, + "configHash": { + "type": "string" + } + } + }, + "passed": { + "type": "boolean" + }, + "scenarios": { + "type": "array", + "items": { + "$ref": "#/$defs/scenarioResult" + } + } + }, + "$defs": { + "latencyMs": { + "type": "object", + "additionalProperties": false, + "required": [ + "p50", + "p95", + "p99", + "mean", + "max" + ], + "properties": { + "p50": { + "type": "number" + }, + "p95": { + "type": "number" + }, + "p99": { + "type": "number" + }, + "mean": { + "type": "number" + }, + "max": { + "type": "number" + } + } + }, + "partialLatencyMs": { + "type": "object", + "additionalProperties": false, + "properties": { + "p50": { + "type": "number" + }, + "p95": { + "type": "number" + }, + "p99": { + "type": "number" + } + } + }, + "loadSummary": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "durationMs", + "warmupMs" + ], + "properties": { + "model": { + "enum": [ + "open", + "closed" + ] + }, + "ratePerSec": { + "type": "number" + }, + "arrival": { + "type": "string" + }, + "concurrency": { + "type": "integer" + }, + "durationMs": { + "type": "number" + }, + "warmupMs": { + "type": "number" + }, + "maxInFlight": { + "type": "integer" + } + }, + "oneOf": [ + { + "properties": { + "model": { + "const": "open" + } + }, + "required": [ + "ratePerSec", + "arrival" + ], + "not": { + "required": [ + "concurrency" + ] + } + }, + { + "properties": { + "model": { + "const": "closed" + } + }, + "required": [ + "concurrency" + ], + "not": { + "anyOf": [ + { + "required": [ + "ratePerSec" + ] + }, + { + "required": [ + "arrival" + ] + } + ] + } + } + ] + }, + "requestSummary": { + "type": "object", + "additionalProperties": false, + "required": [ + "total", + "ok", + "failed", + "successRate" + ], + "properties": { + "total": { + "type": "integer", + "minimum": 0 + }, + "ok": { + "type": "integer", + "minimum": 0 + }, + "failed": { + "type": "integer", + "minimum": 0 + }, + "successRate": { + "type": "number", + "minimum": 0, + "maximum": 1 + } + } + }, + "clientMetrics": { + "type": "object", + "additionalProperties": false, + "required": [ + "latencyMs" + ], + "properties": { + "latencyMs": { + "$ref": "#/$defs/latencyMs" + } + } + }, + "serverMetrics": { + "type": "object", + "additionalProperties": false, + "properties": { + "signatureVerificationMs": { + "type": "object", + "additionalProperties": false, + "required": [ + "overall" + ], + "properties": { + "overall": { + "$ref": "#/$defs/partialLatencyMs" + }, + "byStandard": { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/partialLatencyMs" + } + } + } + }, + "queue": { + "type": "object", + "additionalProperties": false, + "properties": { + "drainMs": { + "$ref": "#/$defs/partialLatencyMs" + }, + "depthMax": { + "type": "number" + } + } + } + } + }, + "errorBucket": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "reason", + "count" + ], + "properties": { + "kind": { + "type": "string" + }, + "status": { + "type": "integer" + }, + "reason": { + "type": "string" + }, + "count": { + "type": "integer", + "minimum": 0 + } + } + }, + "expectResult": { + "type": "object", + "additionalProperties": false, + "required": [ + "metric", + "op", + "threshold", + "unit", + "actual", + "severity", + "pass" + ], + "properties": { + "metric": { + "type": "string" + }, + "op": { + "enum": [ + "lt", + "lte", + "gt", + "gte", + "eq" + ] + }, + "threshold": { + "type": "number" + }, + "unit": { + "type": [ + "string", + "null" + ] + }, + "actual": { + "type": [ + "number", + "null" + ] + }, + "severity": { + "enum": [ + "warn", + "fail" + ] + }, + "pass": { + "type": "boolean" + } + } + }, + "scenarioResult": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type", + "load", + "requests", + "throughputPerSec", + "client", + "server", + "errors", + "expectations", + "passed" + ], + "properties": { + "name": { + "type": "string" + }, + "type": { + "enum": [ + "inbox", + "webfinger", + "actor", + "object", + "fanout", + "collection", + "failure", + "mixed" + ] + }, + "load": { + "$ref": "#/$defs/loadSummary" + }, + "requests": { + "$ref": "#/$defs/requestSummary" + }, + "throughputPerSec": { + "type": "number" + }, + "client": { + "$ref": "#/$defs/clientMetrics" + }, + "server": { + "anyOf": [ + { + "$ref": "#/$defs/serverMetrics" + }, + { + "type": "null" + } + ] + }, + "errors": { + "type": "array", + "items": { + "$ref": "#/$defs/errorBucket" + } + }, + "expectations": { + "type": "array", + "items": { + "$ref": "#/$defs/expectResult" + } + }, + "passed": { + "type": "boolean" + }, + "histogram": { + "$ref": "#/$defs/serializedHistogram" + }, + "deliveryThroughputPerSec": { + "type": "number" + } + } + }, + "serializedHistogram": { + "type": "object", + "additionalProperties": false, + "required": [ + "version", + "subBucketCount", + "count", + "zeroCount", + "min", + "max", + "sum", + "indices", + "counts" + ], + "properties": { + "version": { + "const": 1 + }, + "subBucketCount": { + "type": "integer", + "minimum": 1 + }, + "count": { + "type": "integer", + "minimum": 0 + }, + "zeroCount": { + "type": "integer", + "minimum": 0 + }, + "min": { + "type": "number" + }, + "max": { + "type": "number" + }, + "sum": { + "type": "number" + }, + "indices": { + "type": "array", + "items": { + "type": "integer" + } + }, + "counts": { + "type": "array", + "items": { + "type": "integer", + "minimum": 0 + } + } + } + } + } +} diff --git a/schema/bench/scenario-v2.json b/schema/bench/scenario-v2.json new file mode 100644 index 000000000..01a15b02f --- /dev/null +++ b/schema/bench/scenario-v2.json @@ -0,0 +1,753 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.fedify.dev/bench/scenario-v2.json", + "title": "Fedify benchmark scenario suite", + "type": "object", + "required": [ + "version", + "scenarios" + ], + "additionalProperties": false, + "properties": { + "$schema": { + "type": "string", + "description": "An optional editor hint pointing at this schema." + }, + "version": { + "const": 1 + }, + "target": { + "type": "string", + "format": "uri", + "description": "The target base URL; may be overridden by --target." + }, + "defaults": { + "$ref": "#/$defs/defaults" + }, + "actors": { + "type": "array", + "items": { + "$ref": "#/$defs/actorGroup" + } + }, + "scenarios": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/scenario" + } + } + }, + "$defs": { + "duration": { + "type": "string", + "pattern": "^\\d+(\\.\\d+)?(ms|s|m|h)$", + "description": "A duration such as 500ms, 30s, 2m, or 1h." + }, + "rate": { + "description": "An open-loop arrival rate such as 200/s, or a number.", + "oneOf": [ + { + "type": "number", + "exclusiveMinimum": 0 + }, + { + "type": "string", + "pattern": "^\\d+(\\.\\d+)?\\s*/\\s*(s|m|h)$" + } + ] + }, + "size": { + "description": "A byte size such as 2KB or a plain number of bytes.", + "oneOf": [ + { + "type": "number", + "minimum": 0 + }, + { + "type": "string", + "pattern": "^\\s*\\d+(\\.\\d+)?\\s*([Bb]|[Kk][Bb]|[Kk][Ii][Bb]|[Mm][Bb]|[Mm][Ii][Bb]|[Gg][Bb]|[Gg][Ii][Bb])?\\s*$" + } + ] + }, + "signatureStandard": { + "enum": [ + "draft-cavage-http-signatures-12", + "rfc9421", + "ld-signatures", + "fep8b32" + ] + }, + "signingMode": { + "enum": [ + "jit", + "pipeline", + "presign" + ] + }, + "arrival": { + "enum": [ + "constant", + "poisson" + ] + }, + "scalarOrListString": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + } + ] + }, + "load": { + "type": "object", + "additionalProperties": false, + "properties": { + "rate": { + "$ref": "#/$defs/rate" + }, + "concurrency": { + "type": "integer", + "minimum": 1 + }, + "arrival": { + "$ref": "#/$defs/arrival" + }, + "maxInFlight": { + "type": "integer", + "minimum": 1 + } + }, + "not": { + "required": [ + "rate", + "concurrency" + ] + } + }, + "defaults": { + "type": "object", + "additionalProperties": false, + "properties": { + "duration": { + "$ref": "#/$defs/duration" + }, + "warmup": { + "$ref": "#/$defs/duration" + }, + "load": { + "$ref": "#/$defs/load" + }, + "signing": { + "$ref": "#/$defs/signingMode" + }, + "signatureTimeWindow": { + "type": "boolean" + }, + "runs": { + "type": "integer", + "minimum": 1 + } + } + }, + "actorGroup": { + "type": "object", + "additionalProperties": false, + "required": [ + "signatureStandards" + ], + "properties": { + "name": { + "type": "string" + }, + "count": { + "type": "integer", + "minimum": 1 + }, + "signatureStandards": { + "type": "array", + "uniqueItems": true, + "minItems": 1, + "items": { + "$ref": "#/$defs/signatureStandard" + }, + "contains": { + "enum": [ + "draft-cavage-http-signatures-12", + "rfc9421" + ] + }, + "minContains": 1, + "maxContains": 1, + "description": "Exactly one HTTP request signature scheme, plus optional document signature schemes." + } + } + }, + "generateDirective": { + "type": "object", + "additionalProperties": false, + "required": [ + "generate" + ], + "properties": { + "generate": { + "enum": [ + "lorem" + ] + }, + "size": { + "$ref": "#/$defs/size" + } + } + }, + "content": { + "oneOf": [ + { + "type": "string" + }, + { + "$ref": "#/$defs/generateDirective" + } + ] + }, + "objectSpec": { + "type": "object", + "properties": { + "type": { + "$ref": "#/$defs/scalarOrListString" + }, + "content": { + "$ref": "#/$defs/content" + } + } + }, + "activitySpec": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "$ref": "#/$defs/scalarOrListString" + }, + "embedObject": { + "type": "boolean" + }, + "object": { + "$ref": "#/$defs/objectSpec" + } + } + }, + "objectSource": { + "oneOf": [ + { + "$ref": "#/$defs/scalarOrListString" + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "seed" + ], + "properties": { + "seed": { + "$ref": "#/$defs/scalarOrListString" + }, + "collection": { + "$ref": "#/$defs/scalarOrListString" + }, + "limit": { + "type": "integer", + "minimum": 1 + }, + "type": { + "$ref": "#/$defs/scalarOrListString" + } + } + } + ] + }, + "expectSeverity": { + "enum": [ + "warn", + "fail" + ] + }, + "expectValue": { + "oneOf": [ + { + "type": "string", + "description": "An assertion such as '>= 99%' or '< 100ms'." + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "assert" + ], + "properties": { + "assert": { + "type": "string" + }, + "severity": { + "$ref": "#/$defs/expectSeverity" + } + } + } + ] + }, + "mixEntry": { + "type": "object", + "additionalProperties": false, + "required": [ + "scenario", + "weight" + ], + "properties": { + "scenario": { + "type": "string" + }, + "weight": { + "type": "number", + "exclusiveMinimum": 0 + } + } + }, + "scenario": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "type" + ], + "properties": { + "name": { + "type": "string" + }, + "type": { + "enum": [ + "inbox", + "webfinger", + "actor", + "object", + "fanout", + "collection", + "failure", + "mixed" + ] + }, + "load": { + "$ref": "#/$defs/load" + }, + "duration": { + "$ref": "#/$defs/duration" + }, + "warmup": { + "$ref": "#/$defs/duration" + }, + "signing": { + "$ref": "#/$defs/signingMode" + }, + "signatureTimeWindow": { + "type": "boolean" + }, + "runs": { + "type": "integer", + "minimum": 1 + }, + "expect": { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/expectValue" + } + }, + "recipient": { + "$ref": "#/$defs/scalarOrListString" + }, + "inbox": { + "type": "string" + }, + "activity": { + "$ref": "#/$defs/activitySpec" + }, + "authenticated": { + "type": "boolean" + }, + "collection": { + "$ref": "#/$defs/scalarOrListString" + }, + "source": { + "$ref": "#/$defs/objectSource" + }, + "sender": { + "type": "string" + }, + "followers": { + "type": "integer", + "minimum": 1 + }, + "trigger": { + "type": "object" + }, + "sinkBehavior": { + "type": "object" + }, + "queueDrainTimeout": { + "$ref": "#/$defs/duration" + }, + "fault": { + "$ref": "#/$defs/scalarOrListString" + }, + "mix": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/mixEntry" + } + }, + "sinkBase": { + "type": "string" + } + }, + "allOf": [ + { + "if": { + "properties": { + "type": { + "const": "inbox" + } + } + }, + "then": { + "required": [ + "recipient" + ], + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max", + "signatureVerification.p50", + "signatureVerification.p95", + "signatureVerification.p99" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "type": { + "const": "webfinger" + } + } + }, + "then": { + "required": [ + "recipient" + ], + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "type": { + "const": "actor" + } + } + }, + "then": { + "required": [ + "recipient" + ], + "allOf": [ + { + "if": { + "required": [ + "authenticated" + ], + "properties": { + "authenticated": { + "const": true + } + } + }, + "then": { + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max", + "signatureVerification.p50", + "signatureVerification.p95", + "signatureVerification.p99" + ] + } + } + } + }, + "else": { + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max" + ] + } + } + } + } + } + ] + } + }, + { + "if": { + "properties": { + "type": { + "const": "object" + } + } + }, + "then": { + "required": [ + "source" + ], + "allOf": [ + { + "if": { + "required": [ + "authenticated" + ], + "properties": { + "authenticated": { + "const": true + } + } + }, + "then": { + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max", + "signatureVerification.p50", + "signatureVerification.p95", + "signatureVerification.p99" + ] + } + } + } + }, + "else": { + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max" + ] + } + } + } + } + } + ] + } + }, + { + "if": { + "properties": { + "type": { + "const": "collection" + } + } + }, + "then": { + "required": [ + "recipient" + ], + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "type": { + "const": "fanout" + } + } + }, + "then": { + "required": [ + "sender" + ], + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "deliveryThroughput", + "errors.total", + "errors.4xx", + "errors.5xx", + "queueDrain.p50", + "queueDrain.p95", + "queueDrain.p99" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "type": { + "const": "failure" + } + } + }, + "then": { + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "type": { + "const": "mixed" + } + } + }, + "then": { + "required": [ + "mix" + ], + "properties": { + "expect": { + "propertyNames": { + "enum": [ + "successRate", + "throughputPerSec", + "errors.total", + "errors.4xx", + "errors.5xx", + "latency.p50", + "latency.p95", + "latency.p99", + "latency.mean", + "latency.max", + "deliveryThroughput" + ] + } + } + } + } + } + ] + } + } +}