Skip to content

[rushd][WS3] Daemon lifecycle, reload & warm-set #5898

Description

@bmiddha

Own the daemon's whole lifecycle and memory footprint: client auto-start, fingerprint-driven reload tiers, install/update self-restart with socket handoff, version-mismatch restart, idle auto-shutdown, and stale-socket/crash recovery — plus best-effort warm-set management (LRU idle eviction under a memory budget, telemetry-weighted warm selection, and its config surface) that never affects build correctness.

Depends on: #5895; #5896; #5897

Scope

  • Client auto-start. In rush-client-core, paired with host readiness signaling: a start-lock to avoid a thundering herd, a detached spawn that survives the starting client, awaiting the ready signal, and backoff on failure.
  • Reload tiers. After each EXCLUSIVE command, compute an inputs fingerprint and pick a tier — 0 hot (no reload), 1 soft reload (re-read config/graph), or 2 hard restart (new process).
  • install/update self-restart. Run the mutation, send the requesting client its exit code first, then drain, kill runners, spawn a successor, and hand off the socket (clients connecting during handoff transparently retry).
  • Version-mismatch restart. Trigger a Tier-2 restart on version skew (client expects Y, daemon runs X) and retry the request.
  • Idle shutdown & crash recovery. Shut down after an idle timeout (active runners inhibit shutdown); detect a stale socket/PID from a crashed daemon and auto-start a replacement.
  • Warm-set eviction. Auto-evict idle operations LRU under a budget (closeRunnersAsync(cold) + watcher teardown + record drop), never evicting active-and-subscribed operations.
  • Telemetry-weighted warming. Warm-select operations from the graph's per-op telemetry (score = (timeSaved × frequency) / residentMemory), sharing one ranking with eviction so warming and eviction never disagree.
  • Warm-set config. Expose warmIdleTimeoutSeconds, warmMemoryBudgetMB, warmSetMaxProjects, and autoWarmByTelemetry; all warm-set behavior is best-effort and never correctness-affecting.

Acceptance criteria

  • N concurrent first-invocations start exactly one daemon (start-lock holds under a race); the daemon spawns detached and survives the starting client; start failures back off with an actionable error (falling back to in-process where applicable).
  • Known input deltas map to the expected reload tier (no change → 0, config change → 1, version/engine change → 2); the fingerprint is stable for unchanged inputs and changes when relevant inputs change.
  • On install/update the requesting client receives its exit code before the restart begins, the successor takes over the socket with no dropped client (in-flight connections retry and succeed), and the warm graph is rebuilt against post-install/update state.
  • A forced version skew triggers a Tier-2 restart into the expected version and the client's request then succeeds, with no orphaned old-version daemon left behind.
  • The daemon exits after the configured idle timeout (an in-flight runner prevents shutdown until it finishes); a stale socket/PID is detected and a replacement auto-starts on the next request.
  • Eviction keeps resident memory under the configured budget by dropping LRU cold operations and never evicts active-and-subscribed operations (invariant asserted under load).
  • A single deterministic ranking drives both warm selection and eviction, preferentially warming/retaining higher-value operations (high timeSaved × frequency, low memory); warm-set behavior is best-effort and correctness is unaffected when telemetry is absent.
  • All four warm-set knobs are schema-validated with documented defaults, reject invalid/out-of-range values, and take effect at runtime; changing any knob only affects footprint/latency, never build correctness.
  • Lifecycle, concurrency, and eviction paths are covered by tests; tests green in CI.

Part of #5894.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    Needs triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions