fix(supervisor): opt-in reconcile of drifted sandbox state ownership (#2336)

[TonyLuo-NV]

What & why

Fixes (partially, as a contained stop-gap) #2336.

On the rootless Podman/Docker drivers OpenShell attaches no pinned user-namespace mapping and bind-mounts host state as plain rbinds, so after a host reboot shifts the rootless subuid base a persisted sandbox state tree can come back owned by an unrelated host user (root/systemd-network/dnsmasq) with restrictive modes. The sandbox can then no longer read its own state (readonly sqlite, EACCES). Today prepare_filesystem only chowns newly-created read_write paths, so an existing tree is never repaired. (Downstream report: NVIDIA/NemoClaw#6972.)

Change

Add an opt-in, default-off toggle OPENSHELL_RECONCILE_SANDBOX_OWNERSHIP. When set (1/true), prepare_filesystem recursively re-owns an existing sandbox-writable (read_write) path whose owner no longer matches the configured sandbox identity, so a drifted tree becomes readable again under the current mapping.

Deliberately conservative — it does not change existing behavior or architecture:

• Default off → the create-only chown contract is byte-for-byte unchanged. The existing test prepare_filesystem_skips_chown_for_existing_read_write_paths still passes.
• A path already owned by the target identity is skipped after a single stat, so healthy launches are a no-op.
• Only paths the policy already declares sandbox-writable are touched.
• Symlinks are never followed (reuses the existing chown_sandbox_home symlink refusal).
• The userns/idmap model is untouched.

This is a mitigation that heals the symptom each launch; it does not pin the mapping. The durable fix (idmapped mounts / deterministic --uidmap/--gidmap) is the architectural work tracked in #2336 and left to maintainers — hence draft, to confirm the toggle approach is wanted before polishing docs.

Tests

cargo test -p openshell-supervisor-process -p openshell-core — all pass, including 5 new root-free unit tests:

• path_owner_matches_detects_current_and_mismatched_owner
• path_owner_matches_treats_symlink_as_matching
• reconcile_read_write_ownership_is_noop_when_already_owned
• reconcile_read_write_ownership_skips_missing_paths
• reconcile_read_write_ownership_triggers_on_mismatch

and the preserved prepare_filesystem_skips_chown_for_existing_read_write_paths. cargo fmt --check and cargo clippy --all-targets clean on both crates.

Open questions for maintainers

• Is an opt-in env toggle the preferred stop-gap, or should this wait for the idmapped-mount fix in bug: sandbox bind-mount ownership breaks after host reboot (unpinned rootless userns mapping, no launch-time reconcile) #2336?
• Should the reconcile also cover /sandbox (currently gated on OPENSHELL_SANDBOX_UID) for the rootless drivers?
• Naming/placement of the env var.

Signed-off-by: Tony Luo xialuo@nvidia.com

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

[TonyLuo-NV]

Not stale — still waiting on maintainer review. Rebased onto current main to clear a merge conflict; cargo test, cargo fmt --check, and cargo clippy --all-targets are green on openshell-core + openshell-supervisor-process, and the earlier mirrored Branch Checks run passed.

Summary for a reviewer: this is a default-off opt-in (OPENSHELL_RECONCILE_SANDBOX_OWNERSHIP) that repairs a persisted sandbox state tree whose ownership drifted after a host reboot shifted the rootless userns subuid base (root cause: #2336; downstream report: NVIDIA/NemoClaw#6972). With the toggle unset, behavior is byte-for-byte unchanged — the create-only chown contract and its existing test are preserved.

The open question from the description still stands, and it's the only thing blocking a decision: is an opt-in toggle the preferred stop-gap, or should this wait for the durable idmapped-mount / pinned uid-map fix in #2336? Happy to close this in favor of the durable fix if that's the direction — just needs a maintainer call.

[TonyLuo-NV]

The red Rust (linux-amd64-cpu8) on this PR looks unrelated to the change — a maintainer rerun would likely clear it.

Evidence:

• Exactly one failure out of 1228: grpc::sandbox::tests::watch_producer_releases_request_span_when_client_disconnects in openshell-server, failing with Elapsed(()) — it polls at 10ms under a 5s wall-clock timeout, so it's timing-sensitive on a loaded runner.
• This PR touches only openshell-core and openshell-supervisor-process; the openshell-core change is a single added const with no runtime effect. Both of those crates' suites passed in the same run (365 and 101 tests).
• The failing test is new — added in feat(observability): export gateway traces over OTLP #2534 (feat(gateway): export traces over OTLP).

Caveat, so I'm not overstating it: I could not reproduce locally to confirm, because openshell-server needs the z3 system library that isn't installed on my machine (z3.h not found) — so this is inference from the run, not a local repro.

I don't have rerun rights on this repo (Must have admin rights to Repository), so I can't retrigger it myself. If someone can rerun the failed job — or tell me it's a real regression — I'll act on it either way.