Skip to content

flake: overview.e2e sees an extra runtime-app-reload generation frame (1-in-5 on main, Node 26) — dominant local-ci gate tax #111

Description

@ScriptedAlchemy

Summary

packages/workbench/tests/overview.e2e.test.ts :: offers the host-owned MCP playground handoff... fails intermittently with an extra runtime-app-reload generation frame beyond what the test expects. Reproduced 1-in-5 on unmodified main under Node 26 inside a local-ci leg worktree on a loaded machine.

This is currently the dominant tax on the merge loop: at roughly 20% per leg across the three verify legs, a fully green pnpm check:local-ci run is close to a coin flip, and the last two landings (#86, #109) each burned 4-5 gate attempts on pre-existing load-sensitive flakes rather than regressions.

Evidence

Likely home of the defect

The generation frames come from the provider-owned Runtime App reload channel introduced by #73. Two candidate shapes:

  1. The channel emits a duplicate/extra generation frame under load (e.g. a coalescing window that reopens, or an initial-state frame racing a real reload), in which case the channel should deduplicate by generation ordinal before broadcast.
  2. The channel is behaving as designed (frames are at-least-once) and the test's exactly-once assumption is wrong, in which case the assertion should be relative/monotonic over generation ordinals rather than counting frames — the same fix shape PR fix: root-cause the CI flakes instead of rerunning them #20 applied to dev-provider.integration.test.ts.

Deciding which requires capturing the frame sequence on a failing run (the SSE hub already carries sequence/replay-gap information that should make this cheap to log).

Acceptance

  • The failing assertion either consumes generation frames monotonically (order- and duplicate-tolerant) or the channel provably emits one frame per generation under induced load, with a regression test.
  • 20 consecutive runs of the overview e2e on a loaded machine (load comparable to a 4-leg local-ci run) pass on Node 22.19/24/26.
  • Extend the existing HMR/dev e2e rather than adding a parallel test file (repo convention).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingFramework test harnesses and integration evidence

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions