test(e2e): coordinate podman cleanup with creation

[pimlock]

Summary

Prevent automatic Podman E2E sandbox cleanup from overlapping sandbox creation in another test. The Podman lane still permits concurrent creates, while teardown is exclusive against creation; other compute backends retain their existing lifecycle concurrency.

This was exposed by an intermittent Podman E2E failure and confirmed with a standalone Podman reproducer: deleting a container using an image volume can race with another container attaching the same image.

Related Issue

No issue required: this is a localized E2E harness reliability fix with the investigation and design tradeoffs documented here.

Changes

• Add a Podman-specific read/write lifecycle gate to SandboxGuard
  • creates take a shared lock, preserving concurrent-create coverage
  • cleanup takes an exclusive lock, preventing teardown from overlapping creation or another cleanup
• Join fallback cleanup from Drop instead of detaching a thread that can escape into a later test or be terminated at process exit
• Bound best-effort sandbox deletion to 60 seconds
• Leave Docker, Kubernetes, and VM lifecycle concurrency unchanged
• Keep production compute-driver lifecycle concurrency unchanged

Potential issue

The Podman driver mounts the supervisor OCI image directly into every sandbox using an image volume. A standalone Podman reproducer confirmed that one container removal can race with another container attaching the same image volume. In OpenShell, that can leave the new sandbox with an incomplete supervisor mount.

The E2E harness amplified this unlikely runtime race. SandboxGuard::drop() previously spawned deletion in a detached thread and returned immediately, so teardown from a completed test could overlap creation in another test. The detached thread could also be terminated when an integration-test process exited.

The read/write lifecycle gate is therefore enabled only when OPENSHELL_E2E_DRIVER=podman. Joined, bounded fallback cleanup remains generic test-harness hygiene. The PR does not add a production mutex.

Longer-term Podman and Docker consistency

The more durable production design may be to make Podman deliver the supervisor binary in the same way as Docker: extract the binary from the supervisor image, cache it by immutable image ID, and bind-mount the cached file read-only into each sandbox instead of mounting the complete OCI image.

Podman exposes the required stopped-container archive operation through its Docker-compatible API:
https://docs.podman.io/en/v3.0/_static/api-static.html

Implementing that design would require:

• adding binary response streaming and the container archive endpoint to PodmanClient
• pulling and inspecting the supervisor image and keying the cache by immutable image ID
• creating a temporary stopped container and extracting the supervisor from the returned tar archive
• sharing or extracting Docker driver utilities for archive parsing, ELF validation, atomic cache writes, and digest-keyed paths
• preserving mutable-tag refresh behavior
• mounting the driver-owned cache file read-only with the appropriate SELinux label
• defining cache permissions and stale-cache cleanup
• validating host-path behavior with Podman Machine and remote Podman, where bind sources resolve on the Podman server rather than necessarily on the API client

If host-cache binds are not portable to Podman Machine, a digest-keyed, driver-owned Podman named volume could provide a daemon-side alternative.

This larger change should be reviewed with the original Podman driver author because image-volume sideloading was an intentional part of that driver design.

Alternatives considered

• Production-wide mutex: prevents the race but serializes all sandbox creates and deletes. It is also process-local, so multiple gateways connected to the same Podman service would not coordinate.
• Production read/write lock: preserves concurrent creates but still changes supported production lifecycle concurrency and remains process-local.
• Single-threaded Podman E2E: simple, but slows the complete suite and removes useful concurrent-create coverage.
• Targeted production retry: could preserve concurrency, but requires precise classification of the transient Podman error and safe handling of partially created containers.
• Digest-keyed Podman named volume: keeps supervisor delivery daemon-side and avoids per-sandbox image mounts, but adds population, versioning, and garbage-collection logic.
• Upstream Podman fix: the standalone reproducer can support an upstream report, but OpenShell still needs stable CI while affected Podman versions remain supported.

Testing

• mise run pre-commit passes
• E2E harness unit tests pass: 21 passed
• All E2E test binaries compile with e2e-podman
• E2E harness Clippy passes with warnings denied
• Standalone Podman reproducer confirms the image-volume attach/remove race
• Full OpenShell Podman E2E run was not available in this worktree

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; production architecture is unchanged)

[pimlock]

Standalone Podman reproduction: confirmed

I reproduced the underlying failure using Podman only—no OpenShell process or code was involved.

Environment

• Ubuntu 24.04, rootless Podman 4.9.3
• SQLite state and native overlay storage
• crun 1.14.1
• Podman REST service with concurrent remote Podman clients, matching the driver transport
• Exact supervisor image from the failing workflow: ghcr.io/nvidia/openshell/supervisor:02e890ccf1995bf8671a81ec84baf44328b7f5b8
• Alpine 3.20 as the otherwise-unrelated sandbox image

Trigger

1. Start eight containers that mount the supervisor with type=image at /opt/openshell/bin.
2. Pre-create eight replacement containers with the same image mount.
3. For each old container, issue the same two-phase sequence used by the Podman driver:
   • podman stop --time 10 <old>
   • podman rm --force --volumes <old>
4. Start the replacements approximately 10 seconds after the stops begin, so creation overlaps the stop-timeout transition and final removal.
5. Repeat in batches while varying the start delay around the timeout boundary.

Starting concurrently with the beginning of immediate teardown did not reproduce across thousands of overlaps. The important window is when the stop timeout expires and Podman transitions into final cleanup.

On batch 8, two replacement starts failed with the exact E2E signature:

crun: mount `/home/pmlocek/.local/share/containers/storage/
overlay-containers/<new-container-id>/userdata/overlay/<id>/merge`
to `opt/openshell/bin`: No such file or directory:
OCI runtime attempted to invoke a command that was not found

The missing path belonged to the new container being started, not one of the containers being removed.

Conclusion

This confirms an upstream Podman/Libpod image-mount lifecycle race. Podman generated an OCI specification referencing a Podman-owned image-overlay path that had disappeared before crun consumed it. crun is reporting the invalid source path; it does not own or manage that path.

The E2E lifecycle gate in this PR remains useful because it fixes detached cross-test teardown and prevents the suite from hitting this runtime race. However, the standalone reproduction updates two statements in the current PR description:

• the behavior is not specific to how OpenShell uses image volumes;
• an upstream Podman report is now actionable with a reduced reproducer.

Production remains potentially exposed when sandbox stop/removal overlaps another sandbox image-volume start. The longer-term supervisor-delivery alternatives described in the PR would avoid that Podman path entirely.

[github-actions]

Label test:e2e applied for 71f7d24. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[pimlock]

Closing this for now while we investigate a minimal driver-side compatibility fix that works with the older Podman version shipped by Ubuntu 24.04 without introducing global lifecycle locking.

Ubuntu 24.04 installs Podman 4.9.3 from its official repositories. Although Ubuntu 24.04 remains supported, Podman 4.9 has been upstream EOL for more than two years (Podman lifecycle). This leaves users of the stock Ubuntu package on a substantially older runtime.

The original failure was reproduced independently of OpenShell as a Podman image-volume lifecycle race: removing one container can interfere with another container attaching the same image volume. This PR attempted to avoid that race in the E2E harness by coordinating cleanup with creation while retaining concurrent sandbox creation.
The latest CI run exposed another failure during concurrent Podman operations. It did not have the original missing-supervisor mount signature: the container started, but its supervisor could not reach the gateway and exited after exhausting policy-fetch retries. This shows that the proposed test-level coordination is not a sufficiently complete or principled solution for Podman 4.9.3.

Serializing every Podman sandbox create and delete in the test suite would likely make CI more reliable, but it would also remove useful concurrency coverage and would not protect production deployments. We would prefer a narrow driver-side solution—for example, detecting and safely recovering from the affected transient Podman states with bounded cleanup and retry—while preserving concurrency for unaffected operations.

We also need to clarify the support contract for Ubuntu 24.04 with its stock Podman package and validate the behavior against a current upstream Podman release. We can reopen or replace this PR once we have a focused fix that addresses the runtime behavior rather than masking it only in tests.