fix(server): observe persisted sandbox watch updates

[elezar]

Summary

Make WatchSandbox status-follow streams observe sandbox changes persisted by another gateway process. The existing in-memory notification bus remains the fast path; a bounded store reconciliation closes the cross-process consistency gap.

Related Issue

Related to #1868. This is a generally applicable consistency fix extracted from the HA review.

Changes

• Poll the authoritative sandbox record once per second only while follow_status is enabled.
• Delay the first poll until one interval after the initial snapshot and skip missed ticks to avoid redundant reads and catch-up bursts.
• Emit updates only when the persisted resource_version changes.
• Keep streams alive across transient poll failures, logging only the first failure in each consecutive failure period.
• Add a focused test that persists a status change without notifying the local in-memory bus.

Testing

• mise run pre-commit passes (run inside nix develop)
• Unit tests added/updated
• E2E tests added/updated (not applicable)

Focused regression test:

• nix develop -c cargo test -p openshell-server watch_sandbox_observes_store_update_without_local_notification --lib

Operational tradeoff

Each active WatchSandbox stream with follow_status=true performs one indexed single-record store read per second. This intentionally favors a small, isolated consistency fix over adding a database-specific notification mechanism. Streams that only follow logs or platform events do not poll. A future shared-store watch abstraction could coalesce reads if sustained watcher volume makes this material.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; no public interface, configuration, or stable subsystem boundary changed)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[elezar]

gator-agent

PR Review Status

Validation: Maintainer-authored, small concentrated server consistency fix related to #1868; project-valid for OpenShell.
Head SHA: 1d24eff77d8313924173a28014f1ffa0f4e5b39a

Review findings:

• Two warning-level findings remain and are posted inline.

Docs: Not needed; this changes internal server watch consistency and does not change CLI, configuration, Fern docs workflows, or public policy/provider setup.

Checks: Required gates are currently green: DCOAssistant, OpenShell / Branch Checks, OpenShell / Helm Lint, OpenShell / E2E, and OpenShell / GPU E2E.

Next state: gator:in-review

[elezar]

gator-agent

Warning: The bus path updates last_sandbox_resource_version but does not dedupe before sending. If the store poll observes a local write before the broadcast receiver runs, the later bus event can emit the same resource_version again. Please apply the same guard used in the poll branch before sending, for example:

let resource_version = sandbox.get_resource_version();
if last_sandbox_resource_version == Some(resource_version) {
    continue;
}
last_sandbox_resource_version = Some(resource_version);

[elezar]

gator-agent

Warning: The new persisted-update path only stops on Ready, but WatchSandboxRequest.stop_on_terminal is documented as “READY or ERROR.” A remote process can now persist Error, the stream will emit it, then keep polling. Please use a shared terminal check such as matches!(phase, SandboxPhase::Ready | SandboxPhase::Error) in all stop checks, and add a regression for Provisioning -> Error via store polling.