diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 82ce9650cd..07de687103 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -1,11 +1,11 @@ --- name: debug-openshell-cluster -description: Debug why an OpenShell gateway deployment is unhealthy, unreachable, or unable to create sandboxes. Use when the user has a gateway health failure, Docker/Podman runtime issue, Helm install failure, Kubernetes scheduling issue, TLS secret issue, VM driver issue, or sandbox startup problem. Trigger keywords - debug gateway, gateway failing, deployment failing, helm install failing, cluster health, gateway health, gateway not starting, health check failed, sandbox pending, docker driver, podman driver, vm driver. +description: Debug why an OpenShell gateway deployment is unhealthy, unreachable, or unable to create sandboxes. Use for gateway health failures, Docker/Podman runtime issues, Helm failures, Kubernetes scheduling, TLS or auth, gateway interceptors, supervisor middleware startup or runtime failures, external compute-driver sockets, VM drivers, or sandbox startup. Trigger keywords - debug gateway, gateway failing, deployment failing, helm install failing, cluster health, gateway health, gateway not starting, health check failed, sandbox pending, docker driver, podman driver, kubernetes driver, external driver, compute driver socket, gateway interceptor, supervisor middleware, middleware failed, vm driver. --- # Debug OpenShell Gateway Deployment -Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, or the experimental VM driver. +Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, the experimental VM driver, or an operator-managed out-of-tree compute driver. Use `openshell` first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver: `docker`, `podman`, `kubectl`/`helm`, or VM driver logs. @@ -15,7 +15,7 @@ The target deployment flow is: 1. Operator starts or deploys the gateway with system packages, systemd, Helm, or a development task. The CLI does not start, stop, or destroy gateway services. 2. Operator configures the compute driver. -3. Operator provides TLS and SSH relay material for the deployment mode. +3. Operator provides the CLI and supervisor authentication material required by the deployment mode: edge or OIDC user auth, optional CLI mTLS, and gateway-minted sandbox JWTs. 4. The CLI registers a reachable gateway endpoint with `openshell gateway add`. 5. The gateway creates sandboxes through the selected compute driver. @@ -25,7 +25,7 @@ For local evaluation only, TLS may be disabled and the gateway can be reached th - The `openshell` CLI must be available for endpoint checks. - Know the active gateway name and endpoint, or be able to inspect local gateway metadata. -- Know the compute platform: Docker, Podman, Kubernetes, or VM. +- Know the compute platform: Docker, Podman, Kubernetes, VM, or an out-of-tree driver. - For Kubernetes: `kubectl` must target the cluster that hosts OpenShell and Helm version 3 or later must be available. - For Docker or Podman: the runtime socket must be reachable from the gateway host. @@ -36,15 +36,24 @@ Run diagnostics in order and stop once the root cause is clear. ### Step 1: Check CLI Reachability ```bash +openshell gateway list --output json openshell gateway info openshell status ``` +For a one-off endpoint check that bypasses stored gateway selection and metadata: + +```bash +openshell --gateway-endpoint status +``` + Common findings: - `No active gateway`: register one with `openshell gateway add `. - Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active. -- TLS/certificate errors: CLI mTLS bundle does not match the gateway CA, or the gateway is running with unexpected TLS settings. +- TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener. +- `Unauthenticated` from an edge or OIDC gateway: refresh stored credentials with `openshell gateway login [name]`, then retry. Use `gateway logout` only when intentionally clearing local credentials. +- A direct development endpoint with a private or self-signed certificate can be isolated with `--gateway-endpoint --gateway-insecure`; do not persist or recommend insecure verification for shared gateways. ### Step 2: Identify the Compute Platform @@ -56,8 +65,48 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t | Podman | Podman socket, rootless networking, sandbox containers, image pulls. | | Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. | | VM | VM driver logs, rootfs availability, host virtualization support. | +| Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. | + +### Step 3: Check Gateway Startup Dependencies + +Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready. + +For out-of-tree compute drivers, confirm the custom driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts: + +```bash +rg -n 'compute_drivers|socket_path' /etc/openshell/gateway.toml +stat /run/openshell/.sock +journalctl -u --no-pager --lines=200 +journalctl -u openshell-gateway --no-pager --lines=200 +``` + +The custom driver name must not be a reserved built-in name (`docker`, `podman`, `kubernetes`, or `vm`). The socket must be accessible only to the intended gateway identity. Check gateway logs for connection errors, `GetCapabilities` failures, or an unexpected advertised driver name. The gateway does not create or supervise out-of-tree driver processes or sockets. + +For configured gateway interceptors, inspect `[[openshell.gateway.interceptors]]`, their Unix or network endpoints, and gateway startup logs: + +```bash +rg -n 'interceptors|provider_profile_sources|grpc_endpoint|binding_policy|failure_policy' /etc/openshell/gateway.toml +stat /run/openshell/interceptors/.sock +journalctl -u --no-pager --lines=200 +journalctl -u openshell-gateway --no-pager --lines=200 +``` + +The gateway calls each interceptor's `Describe` RPC and validates its manifest at startup. Check for unreachable endpoints, invalid RPC/phase bindings, strict `allowlist` or `exact` mismatches, and `post_commit` bindings that resolve to `fail_closed`. If `provider_profile_sources` names an interceptor, that interceptor must advertise provider-profile capability and return a valid, duplicate-free catalog. A selected interceptor-only source is authoritative; include `builtin` or `user` sources explicitly when composition is intended. + +For operator-run supervisor middleware, inspect `[[openshell.supervisor.middleware]]`, service reachability, and both gateway and supervisor logs: + +```bash +rg -n 'supervisor|middleware|grpc_endpoint|max_body_bytes|timeout' /etc/openshell/gateway.toml +journalctl -u --no-pager --lines=200 +journalctl -u openshell-gateway --no-pager --lines=200 +openshell logs --tail --source sandbox +``` + +The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if `Describe` is unavailable, a manifest exposes duplicate `HttpRequest/pre_credentials` bindings, the registration claims the reserved `openshell/` namespace, or body and timeout limits are invalid. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its `network_middlewares` config. + +At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the request, while `fail_open` bypasses only that stage and emits a detection finding. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event. -### Step 3: Check Docker-Backed Gateways +### Step 4: Check Docker-Backed Gateways ```bash docker info @@ -110,7 +159,7 @@ For source checkout development, restart the local gateway with: mise run gateway:docker ``` -### Step 4: Check Podman-Backed Gateways +### Step 5: Check Podman-Backed Gateways ```bash podman info @@ -126,7 +175,7 @@ Common findings: - Sandbox image missing or pull denied: verify image reference and registry credentials. - Supervisor cannot call back: check callback endpoint and gateway logs. -### Step 5: Check Kubernetes Helm Gateways +### Step 6: Check Kubernetes Helm Gateways ```bash helm -n openshell status openshell @@ -319,7 +368,7 @@ kubectl -n logs -c openshell-supervisor-networ kubectl -n logs -c agent --tail=200 ``` -### Step 6: Check VM-Backed Gateways +### Step 7: Check VM-Backed Gateways Use the VM driver logs and host diagnostics available in the user's environment. Verify: @@ -346,6 +395,13 @@ openshell logs | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | +| Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login `, gateway auth logs | +| Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy | +| Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `builtin` and `user` when intended | +| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response | +| Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields | +| HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` | +| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | @@ -359,7 +415,7 @@ When handing results back to the user, include: - Compute platform and driver. - Gateway process or workload status. - Recent gateway log summary. -- Missing or malformed TLS or SSH relay material. +- Missing or malformed TLS, OIDC/mTLS, or sandbox JWT material. - Service exposure status. - Sandbox workload status. - The exact command that failed and the shortest fix. diff --git a/.agents/skills/launch-openshell-gator/SKILL.md b/.agents/skills/launch-openshell-gator/SKILL.md index bad9441567..20f902f085 100644 --- a/.agents/skills/launch-openshell-gator/SKILL.md +++ b/.agents/skills/launch-openshell-gator/SKILL.md @@ -123,11 +123,11 @@ pr_number="" [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } ``` -Use a restricted sandbox-name character set: +Use the portable Kubernetes DNS-1123 sandbox-name format even when the selected gateway currently uses another driver: ```bash sandbox_name="gator-pr-${pr_number}-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } ``` For local image contexts passed to `--from`, use an agent-created path such as `mktemp -d`; do not pass raw user-supplied paths without validating that they are expected local Dockerfile contexts. @@ -144,7 +144,7 @@ pr_number="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } sandbox_name="gator-pr-${pr_number}-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } ./scripts/agents/run.sh \ --agent gator \ @@ -165,7 +165,7 @@ issue_number="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } [[ "$issue_number" =~ ^[0-9]+$ ]] || { echo "invalid issue number" >&2; exit 1; } sandbox_name="gator-issue-${issue_number}-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } ./scripts/agents/run.sh \ --agent gator \ @@ -186,7 +186,7 @@ issue_number="" [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } [[ "$issue_number" =~ ^[0-9]+$ ]] || { echo "invalid issue number" >&2; exit 1; } sandbox_name="gator-pr-${pr_number}-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } ./scripts/agents/run.sh \ --agent gator \ @@ -207,7 +207,7 @@ pr_number="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } sandbox_name="gator-pr-${pr_number}-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } ./scripts/agents/run.sh \ --agent gator \ @@ -228,7 +228,7 @@ pr_number="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } sandbox_name="gator-pr-${pr_number}-gpt56sol-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } CODEX_MODEL=gpt-5.6-sol \ ./scripts/agents/run.sh \ @@ -250,7 +250,7 @@ pr_number="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "invalid PR number" >&2; exit 1; } sandbox_name="gator-pr-${pr_number}-gpt56sol-supervised" -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } tmp_context="$(mktemp -d "${TMPDIR:-/tmp}/gator-codex-XXXXXX")" cp -R scripts/agents/gator/. "$tmp_context"/ @@ -290,7 +290,7 @@ Read that file directly. Important markers: gateway_name="" sandbox_name="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } openshell --gateway "$gateway_name" sandbox list openshell --gateway "$gateway_name" sandbox get "$sandbox_name" @@ -318,7 +318,7 @@ Before deleting, check that the sandbox is truly stale or that the operator aske gateway_name="" sandbox_name="" [[ "$gateway_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid gateway name" >&2; exit 1; } -[[ "$sandbox_name" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "invalid sandbox name" >&2; exit 1; } +[[ "$sandbox_name" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]] || { echo "invalid sandbox name" >&2; exit 1; } openshell --gateway "$gateway_name" sandbox delete "$sandbox_name" ./scripts/agents/run.sh \ diff --git a/.claude/agents/principal-engineer-reviewer.md b/.claude/agents/principal-engineer-reviewer.md index 8badf491e0..a7926dbf02 100644 --- a/.claude/agents/principal-engineer-reviewer.md +++ b/.claude/agents/principal-engineer-reviewer.md @@ -58,6 +58,7 @@ When reviewing code or diffs: 5. Reference specific files and line numbers (`file_path:line_number`). 6. When suggesting a change, show the concrete fix — don't just describe it. 7. If something is good, say so briefly. Positive signal is useful too. +8. When behavior, commands, or development workflows change, consult the `sync-agent-infra` maintenance map and verify that related skills were updated. Apply its full consistency checklist when the changes add, remove, or rename skills or crates; change workflow relationships or skill coverage; modify issue or PR templates; or change agent cross-references. Report missing companion updates or drift as a warning. When reviewing plans or architecture documents: diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index b01cdb7718..5a80331ba1 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -31,7 +31,7 @@ jobs: outputs: should_run: ${{ steps.gate.outputs.should_run }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - id: gate uses: ./.github/actions/pr-gate @@ -47,7 +47,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Mark workspace as safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -71,7 +71,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install tools run: mise install --locked @@ -97,7 +97,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Configure GHA sccache backend uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 @@ -154,7 +154,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install tools run: mise install --locked @@ -185,7 +185,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install tools run: mise install --locked diff --git a/.github/workflows/branch-docs.yml b/.github/workflows/branch-docs.yml index 48ac80d004..9333a6ee1e 100644 --- a/.github/workflows/branch-docs.yml +++ b/.github/workflows/branch-docs.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Check Fern preview availability id: fern-preview diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 17aa068581..b6810fb2c4 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -28,7 +28,7 @@ jobs: run_kubernetes_ha_e2e: ${{ steps.labels.outputs.run_kubernetes_ha_e2e }} run_any_e2e: ${{ steps.labels.outputs.run_any_e2e }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - id: gate uses: ./.github/actions/pr-gate - id: labels diff --git a/.github/workflows/ci-image.yml b/.github/workflows/ci-image.yml index caeebe4efd..06b4d380dd 100644 --- a/.github/workflows/ci-image.yml +++ b/.github/workflows/ci-image.yml @@ -35,7 +35,7 @@ jobs: runs-on: ${{ matrix.runner }} timeout-minutes: 60 steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Log in to GitHub Container Registry uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4 diff --git a/.github/workflows/dco.yml b/.github/workflows/dco.yml index 53339f4bf9..1bb69a15d9 100644 --- a/.github/workflows/dco.yml +++ b/.github/workflows/dco.yml @@ -6,6 +6,9 @@ on: pull_request_target: types: [opened, closed, synchronize] +env: + DCO_SIGN_COMMENT: "I have read the DCO document and I hereby sign the DCO." + permissions: actions: write checks: none @@ -29,7 +32,7 @@ jobs: - name: "DCO Assistant" if: | (github.event.comment.body == 'recheck' || - github.event.comment.body == 'I have read the Contributor Agreement including DCO and I hereby sign the Contributor Agreement and DCO') || + github.event.comment.body == env.DCO_SIGN_COMMENT) || github.event_name == 'pull_request_target' uses: contributor-assistant/github-action@ca4a40a7d1004f18d9960b404b97e5f30a505a08 env: @@ -46,6 +49,6 @@ jobs: [Developer Certificate of Origin](https://github.com/NVIDIA/OpenShell/blob/main/DCO) before we can accept your contribution. You can sign the DCO by adding a comment below using this text: - custom-pr-sign-comment: "I have read the DCO document and I hereby sign the DCO." + custom-pr-sign-comment: ${{ env.DCO_SIGN_COMMENT }} lock-pullrequest-aftermerge: false use-dco-flag: true diff --git a/.github/workflows/deb-package.yml b/.github/workflows/deb-package.yml index 7700e27213..3929203909 100644 --- a/.github/workflows/deb-package.yml +++ b/.github/workflows/deb-package.yml @@ -42,7 +42,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index da8c07c0a4..d0d600e58b 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -206,7 +206,7 @@ jobs: DOCKER_PUSH: ${{ inputs.push && '1' || '0' }} DOCKER_PLATFORM: ${{ matrix.platform }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} fetch-depth: 0 diff --git a/.github/workflows/driver-vm-linux.yml b/.github/workflows/driver-vm-linux.yml index 7a15271d05..bee48704de 100644 --- a/.github/workflows/driver-vm-linux.yml +++ b/.github/workflows/driver-vm-linux.yml @@ -44,7 +44,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} @@ -120,7 +120,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} fetch-depth: 0 diff --git a/.github/workflows/driver-vm-macos.yml b/.github/workflows/driver-vm-macos.yml index 28b9297838..e80ebf37cb 100644 --- a/.github/workflows/driver-vm-macos.yml +++ b/.github/workflows/driver-vm-macos.yml @@ -32,7 +32,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} @@ -79,7 +79,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} fetch-depth: 0 @@ -135,7 +135,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] }} fetch-depth: 0 diff --git a/.github/workflows/e2e-gpu-test.yaml b/.github/workflows/e2e-gpu-test.yaml index 3695764dfb..be2b66e1d4 100644 --- a/.github/workflows/e2e-gpu-test.yaml +++ b/.github/workflows/e2e-gpu-test.yaml @@ -67,7 +67,7 @@ jobs: # probe below and by the e2e tests in e2e/rust/tests/gpu/device_selection.rs. OPENSHELL_E2E_GPU_PROBE_IMAGE: "nvcr.io/nvidia/base/ubuntu:noble-20251013" steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Use prebuilt OpenShell CLI if: inputs.cli-artifact-prefix != '' diff --git a/.github/workflows/e2e-kubernetes-test.yml b/.github/workflows/e2e-kubernetes-test.yml index 23399fc2a1..c40aed6e82 100644 --- a/.github/workflows/e2e-kubernetes-test.yml +++ b/.github/workflows/e2e-kubernetes-test.yml @@ -71,7 +71,7 @@ jobs: MISE_VERSION: ${{ inputs.mise-version }} KIND_CLUSTER_NAME: kube-e2e-${{ github.run_id }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} diff --git a/.github/workflows/e2e-label-help.yml b/.github/workflows/e2e-label-help.yml index 1190bcd3d3..4c3a1dfe6f 100644 --- a/.github/workflows/e2e-label-help.yml +++ b/.github/workflows/e2e-label-help.yml @@ -59,7 +59,8 @@ jobs: esac mirror_ref="pull-request/$PR_NUMBER" - mirror_sha=$(gh api "repos/$GH_REPO/branches/$mirror_ref" --jq '.commit.sha' 2>/dev/null || echo "") + mirror_sha=$(gh api "repos/$GH_REPO/git/ref/heads/$mirror_ref" --jq '.object.sha' 2>/dev/null || true) + if [[ ! "$mirror_sha" =~ ^[0-9a-f]{40}$ ]]; then mirror_sha=""; fi short_pr=${PR_HEAD_SHA:0:7} if [ -z "$mirror_sha" ]; then diff --git a/.github/workflows/e2e-test.yml b/.github/workflows/e2e-test.yml index 561b6c84d6..d8f33f7016 100644 --- a/.github/workflows/e2e-test.yml +++ b/.github/workflows/e2e-test.yml @@ -74,7 +74,7 @@ jobs: OPENSHELL_REGISTRY_USERNAME: ${{ github.actor }} OPENSHELL_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} persist-credentials: false @@ -93,7 +93,7 @@ jobs: - name: Check out MCP conformance tests if: matrix.suite == 'mcp' - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: repository: modelcontextprotocol/conformance # Pin after v0.1.16 to include the tools_call client scenario fix. @@ -149,7 +149,7 @@ jobs: OPENSHELL_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} persist-credentials: false @@ -257,7 +257,7 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} persist-credentials: false @@ -310,6 +310,12 @@ jobs: socat \ zstd + - name: Validate VM host tools + run: | + command -v mke2fs + command -v mkfs.ext4 + command -v debugfs + - name: Install mise run: | curl https://mise.run | MISE_VERSION=v2026.4.25 sh diff --git a/.github/workflows/helm-lint.yml b/.github/workflows/helm-lint.yml index b57346101b..0ca4473d90 100644 --- a/.github/workflows/helm-lint.yml +++ b/.github/workflows/helm-lint.yml @@ -32,7 +32,7 @@ jobs: outputs: should_run: ${{ steps.gate.outputs.should_run }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - id: gate uses: ./.github/actions/pr-gate @@ -53,7 +53,7 @@ jobs: shell: bash run: echo "should_run=true" >> "$GITHUB_OUTPUT" - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 if: github.event_name == 'push' - id: merge-base @@ -86,7 +86,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install tools run: mise install --locked diff --git a/.github/workflows/publish-docs-website.yml b/.github/workflows/publish-docs-website.yml index 16363f6e34..dfb24b9cea 100644 --- a/.github/workflows/publish-docs-website.yml +++ b/.github/workflows/publish-docs-website.yml @@ -36,7 +36,7 @@ jobs: timeout-minutes: 15 steps: - name: Checkout docs website branch - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: docs-website diff --git a/.github/workflows/release-auto-tag.yml b/.github/workflows/release-auto-tag.yml index c0c84b0a3b..8290de4896 100644 --- a/.github/workflows/release-auto-tag.yml +++ b/.github/workflows/release-auto-tag.yml @@ -20,7 +20,7 @@ jobs: create-tag: runs-on: ubuntu-latest steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index b4c57fc417..eb57da9da3 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -33,7 +33,7 @@ jobs: rpm_version: ${{ steps.v.outputs.rpm_version }} rpm_release: ${{ steps.v.outputs.rpm_release }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -127,7 +127,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -177,7 +177,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -241,7 +241,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -318,7 +318,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -389,7 +389,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -481,7 +481,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -555,7 +555,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 @@ -776,7 +776,7 @@ jobs: outputs: wheel_filenames: ${{ steps.wheel_filenames.outputs.wheel_filenames }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Download all CLI artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 @@ -1013,7 +1013,7 @@ jobs: permissions: packages: write steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: ./.github/actions/release-helm-oci with: diff --git a/.github/workflows/release-tag.yml b/.github/workflows/release-tag.yml index 0ff190ab0b..d2fc6b8102 100644 --- a/.github/workflows/release-tag.yml +++ b/.github/workflows/release-tag.yml @@ -48,7 +48,7 @@ jobs: # Commit resolved from RELEASE_TAG, used for image tags and downstream metadata source_sha: ${{ steps.v.outputs.source_sha }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -156,7 +156,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -207,7 +207,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -272,7 +272,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -350,7 +350,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -422,7 +422,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -522,7 +522,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -599,7 +599,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -857,7 +857,7 @@ jobs: outputs: wheel_filenames: ${{ steps.wheel_filenames.outputs.wheel_filenames }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} @@ -1033,7 +1033,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 15 steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} @@ -1061,7 +1061,7 @@ jobs: permissions: packages: write steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} diff --git a/.github/workflows/release-vm-kernel.yml b/.github/workflows/release-vm-kernel.yml index cf49e8687c..0d7bd31f33 100644 --- a/.github/workflows/release-vm-kernel.yml +++ b/.github/workflows/release-vm-kernel.yml @@ -47,7 +47,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -97,7 +97,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -130,7 +130,7 @@ jobs: env: RUSTC_WRAPPER: "" steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install dependencies run: | @@ -176,7 +176,7 @@ jobs: attestations: write artifact-metadata: write steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Download all runtime artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 diff --git a/.github/workflows/rpm-package.yml b/.github/workflows/rpm-package.yml index dfafcd9339..591d10e619 100644 --- a/.github/workflows/rpm-package.yml +++ b/.github/workflows/rpm-package.yml @@ -60,7 +60,7 @@ jobs: pandoc python3-devel git-core \ cargo-rpm-macros - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.checkout-ref }} fetch-depth: 0 diff --git a/.github/workflows/rust-cache-seed.yml b/.github/workflows/rust-cache-seed.yml index deb33eb835..1e76384543 100644 --- a/.github/workflows/rust-cache-seed.yml +++ b/.github/workflows/rust-cache-seed.yml @@ -36,7 +36,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install tools run: mise install --locked diff --git a/.github/workflows/rust-native-build.yml b/.github/workflows/rust-native-build.yml index 89993ba12d..8ea4df2f3a 100644 --- a/.github/workflows/rust-native-build.yml +++ b/.github/workflows/rust-native-build.yml @@ -86,7 +86,7 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs['checkout-ref'] || github.sha }} fetch-depth: 0 diff --git a/.github/workflows/snap-package.yml b/.github/workflows/snap-package.yml index 22e7302d5c..ed248b79e5 100644 --- a/.github/workflows/snap-package.yml +++ b/.github/workflows/snap-package.yml @@ -44,7 +44,7 @@ jobs: timeout-minutes: 60 environment: ${{ inputs.github-environment }} steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.checkout-ref }} fetch-depth: 0 diff --git a/.github/workflows/sync-docs.yml b/.github/workflows/sync-docs.yml index 481fd85cf5..586444e26e 100644 --- a/.github/workflows/sync-docs.yml +++ b/.github/workflows/sync-docs.yml @@ -52,7 +52,7 @@ jobs: timeout-minutes: 20 steps: - name: Checkout automation - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: path: automation @@ -79,21 +79,21 @@ jobs: - name: Checkout source docs if: ${{ inputs.operation == 'sync' }} - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.source_ref }} fetch-depth: 0 path: source - name: Checkout docs website branch - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: docs-website fetch-depth: 0 path: docs-website - name: Install uv - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 + uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 with: version: "0.10.12" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ad20267ca6..0465f88f7d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -74,6 +74,7 @@ Skills live in `.agents/skills/`. Your agent's harness can discover and load the | Contributing | `create-github-issue` | Create well-structured GitHub issues | | Contributing | `create-github-pr` | Create pull requests with proper conventions | | Reviewing | `review-github-pr` | Summarize PR diffs and key design decisions | +| Reviewing | `review-security-changes` | Review code changes for security vulnerabilities and boundary regressions | | Reviewing | `review-security-issue` | Assess security issues for severity and remediation | | Reviewing | `fix-security-issue` | Implement an approved security remediation plan | | Reviewing | `watch-github-actions` | Monitor CI pipeline status and logs | diff --git a/Cargo.lock b/Cargo.lock index b412467610..31e2104987 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3855,6 +3855,7 @@ dependencies = [ "openshell-core", "openshell-policy", "openshell-providers", + "openshell-sdk", "openshell-tui", "owo-colors", "prost-types", @@ -3889,6 +3890,7 @@ dependencies = [ "glob", "ipnet", "miette", + "nix", "prost", "prost-types", "protobuf-src", @@ -4074,6 +4076,7 @@ version = "0.0.0" dependencies = [ "glob", "openshell-core", + "openshell-policy", "serde", "serde_json", "serde_yml", @@ -4143,6 +4146,7 @@ dependencies = [ "reqwest 0.12.28", "rustls 0.23.38", "serde", + "serde_json", "thiserror 2.0.18", "tokio", "tokio-rustls 0.26.4", @@ -4151,6 +4155,7 @@ dependencies = [ "tonic", "tower 0.5.3", "tracing", + "wiremock", ] [[package]] @@ -4354,6 +4359,7 @@ version = "0.0.0" dependencies = [ "base64 0.22.1", "crossterm 0.28.1", + "indexmap", "miette", "openshell-bootstrap", "openshell-core", diff --git a/Cargo.toml b/Cargo.toml index 7081fe97e2..4ec6a0d44f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -110,6 +110,7 @@ pin-project-lite = "0.2" tokio-stream = "0.1" protobuf-src = "1.1.0" url = "2" +indexmap = "2" # Database sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "sqlite", "migrate"] } diff --git a/architecture/build.md b/architecture/build.md index 70b6bf5dc6..eb8672f15a 100644 --- a/architecture/build.md +++ b/architecture/build.md @@ -138,9 +138,11 @@ do not infer from kube context. ## Python Wheel Packaging The generated protobuf/gRPC stubs under `python/openshell/_proto/` are gitignored -build outputs of `mise run python:proto`. maturin honors `.gitignore` when -collecting `python-source` files, so native builds (Linux CI, local -`pip install .`) would drop them and ship an unimportable wheel. `pyproject.toml` +build outputs of `mise run python:proto`. The task uses `uv run --frozen` to +synchronize the current worktree's `.venv` from `uv.lock` before generation. +maturin honors `.gitignore` when collecting `python-source` files, so native +builds (Linux CI, local `pip install .`) would drop them and ship an unimportable +wheel. `pyproject.toml` pins them back in with `[tool.maturin].include` globs. The release workflows install each Linux wheel in a clean image and import `openshell.sandbox` as a smoke check. diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index ec42277ab6..f211f906d6 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -30,6 +30,34 @@ The gateway records driver identity and version from the startup capability response. Elevated gateway info reports that initialized driver snapshot instead of re-querying drivers on each request. +## Deletion Lifecycle + +Delete requests use per-sandbox gates to serialize delete attempts. A request +resolves the name once and remains bound to that stable ID. The only +combined lock order is delete gate, then the gateway-wide state guard; external +driver calls run without the global guard. + +Delete gates are process-local and do not coordinate gateway replicas. They +serialize attempts rather than share results: if one attempt fails and recovery +restores a deletable state, a request waiting on the gate may retry the driver. +Persisted resource-version checks remain the cross-replica safety boundary. + +Watcher events do not acquire delete gates. Exact resource-version checks allow +them to interleave safely: status snapshots are no-ops for `Deleting` rows, +deleted events are idempotent, and snapshots for absent rows are ignored. + +An accepted delete (`deleted = true`) is finalized by the watcher. If the +backend is already absent (`deleted = false`), the request removes gateway state +synchronously. Sandbox row removal remains bound to the stable ID and resource +version. Settings retain their existing best-effort name-based cleanup; SSH +sessions, indexes, and watch/log buses are cleaned after confirmed removal. + +The request acquires both locks before starting owned work, so cancellation +while queued does not leave a delete armed. After that commitment point, the +owned task prevents cancellation from stranding a mutation. A gateway restart +does not resume a persisted `Deleting` operation. If the backend completed the +delete, reconciliation removes the row; otherwise it can remain `Deleting`. + ## Runtime Summary | Runtime | Best fit | Sandbox boundary | Notes | diff --git a/architecture/gateway.md b/architecture/gateway.md index cca6599281..c8b323ea13 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -137,6 +137,21 @@ Supported auth modes: | Cloudflare JWT | Edge-authenticated deployments where Cloudflare Access supplies identity. | | OIDC | Bearer-token auth for users, with browser PKCE or client credentials login. | +The CLI persists the scopes requested during OIDC login in gateway metadata and +reuses them when refreshing an access token. This preserves the intended API +resource selection for identity providers that bind access-token audiences to +OAuth scopes. + +Gateway health and user authentication are separate probes. `OpenShell.Health` +remains unauthenticated so deployment and load-balancer health checks do not +depend on user credentials. The CLI uses the existing, side-effect-free +`OpenShell.GetGatewayInfo` capability query as its protected authentication +probe. `Unauthenticated` means the credentials were rejected, while +`PermissionDenied` proves authentication succeeded before the caller failed +the capability query's admin authorization check. The CLI combines the health +and capability results so a reachable gateway with an expired or rejected +token is reported as connected but unauthenticated. + Sandbox supervisor RPCs authenticate with explicit sandbox credentials; mTLS does not grant sandbox identity. Kubernetes deployments use the gateway-minted JWT bootstrap path: the supervisor starts with a projected diff --git a/architecture/sandbox.md b/architecture/sandbox.md index d2cb44d7b8..4f95e1ef69 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -53,7 +53,10 @@ paths, such as proxy support files or GPU device paths when a GPU is present. All ordinary agent egress is routed through the sandbox proxy. The proxy identifies the calling binary, checks trust-on-first-use binary identity, rejects -unsafe internal destinations, and evaluates the active policy. +unsafe internal destinations, and evaluates the active policy. On Linux, it +maps an accepted proxy connection back to the workload socket by matching the +complete local-to-remote TCP tuple before resolving every process that owns the +socket inode. For inspected HTTP traffic, the proxy can enforce REST method/path rules, WebSocket upgrade and text-message rules, GraphQL operation rules, and MCP method, tool, and supported params rules or generic JSON-RPC method rules @@ -103,6 +106,70 @@ handled by the inference interception path: External inference endpoints that do not use `inference.local` are treated like ordinary network traffic and must be allowed by policy. +In proxy-required networks, the supervisor chains upstream TLS tunnels through +a corporate forward proxy with HTTP CONNECT instead of connecting directly, +once policy and SSRF checks pass. Only TLS (CONNECT) egress is chained: +plain-HTTP requests always dial the destination directly, because forwarding +plain HTTP through a proxy requires absolute-form request forwarding rather +than CONNECT tunneling and is out of scope. The proxy configuration is an +operator-owned boundary delivered on the supervisor's command line +(`--upstream-proxy` and friends) by the compute driver; sandbox and template +environment — and `ENV` values baked into the sandbox image — cannot +influence it, since none of these can alter the argv the driver sets. The +conventional `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` variables a sandbox +controls are ignored on this path. Operator `NO_PROXY` destinations and +loopback always dial directly; add driver-injected host aliases (e.g. +`host.containers.internal`) to the operator `NO_PROXY` list when the corporate +proxy cannot reach the container host. `NO_PROXY` matching is port-aware and +resolution-aware: an entry with a `:port` qualifier only bypasses that port, +and IP/CIDR entries also match hostnames through their validated resolved +addresses, with the direct dial limited to the addresses the entry contains. Only `http://` proxy URLs in explicit +`http://host:port` form are supported — the scheme and port are both +required, and a path, query, or fragment is rejected. Local DNS resolution +and SSRF validation still run before the proxied dial, and the CONNECT +target sent to the corporate proxy is a validated resolved address, so the +proxy performs no DNS resolution of its own and the tunnel stays bound to +the answer that passed SSRF and `allowed_ips` validation. The hostname still +travels inside the tunnel (TLS SNI, application `Host`). In split-horizon +networks, point the gateway host at the corporate resolver so internal names +validate to their internal addresses; the `proxy_connect_by_hostname` +opt-in exists as a +last resort for proxies whose ACLs filter on hostnames and reject IP CONNECT +targets — with it, the proxy resolves the name itself and its ACLs become +the effective egress control for proxied TLS. (Resolving through the proxy's +own DNS view, e.g. DoH tunneled via CONNECT, is a possible future +enhancement and out of scope.) The workload child's proxy variables are +unaffected — they are always rewritten to point at the local policy proxy. + +The configuration is fail-closed: a setting that is present but invalid — an +empty value, an unsupported or malformed proxy URL, an unreadable auth file, +a malformed credential, or an auth file or `NO_PROXY` list set while no proxy +URL is configured — is fatal to supervisor startup instead of being treated +as unset, so a misconfiguration can never silently degrade to direct dialing +or unauthenticated proxy access. Only an omitted argument means "no proxy". +The driver validates the same rules at sandbox-create time through +validators shared with the supervisor +(`openshell_core::driver_utils::parse_upstream_proxy_url` and +`parse_upstream_proxy_credential`). + +Proxy credentials are never embedded in the URL: an inline `user:pass@` is +rejected because it would be stored in `gateway.toml` and exposed in container +metadata. Operators supply credentials via `proxy_auth_file`; the driver +stages them as a root-only secret mounted at a fixed path and passes only +that path on the supervisor's command line. The supervisor reads the +file and builds the `Proxy-Authorization: Basic` header; a credential that is +empty, contains control characters, or is not in `user:pass` form is fatal on +both sides. + +The Basic header travels over the plain-TCP connection to the `http://` proxy, +so it is readable on the network path between sandbox host and proxy. +Configuring `proxy_auth_file` therefore requires the explicit opt-in +`proxy_auth_allow_insecure = true`. Both the +driver (at sandbox-create time) and the supervisor (at startup) reject an +auth file without the acknowledgement, and the acknowledgement without an +auth file, so credentials are never sent in cleartext without an explicit +operator decision. + ## Credentials Provider credentials are stored at the gateway and fetched by the supervisor at diff --git a/architecture/security-policy.md b/architecture/security-policy.md index c91ba445eb..b4f0bdb912 100644 --- a/architecture/security-policy.md +++ b/architecture/security-policy.md @@ -122,17 +122,25 @@ through the proposal loop instead of treating the denial as terminal. policy plus the sandbox's attached-provider credential set, then computes the delta of findings between the current baseline and the merged policy. 3. **Auto-approval gate (proposer-agnostic, opt-in).** Auto-approval fires - when *both* (a) the prover delta is empty (`prover: no new findings`) AND - (b) the `proposal_approval_mode` setting resolves to `"auto"` — gateway - scope wins, sandbox scope is the per-sandbox override, default is - `"manual"`. When both hold, the gateway internally invokes the approve - path with actor identity `system:auto`. The audit event uses - `CONFIG:APPROVED` and carries `auto=true`, `source=`, - `prover_delta=empty`, and `resolved_from=` as unmapped - fields, with message text `"auto-approved: no new prover findings"` — - never `safe`. The opt-in gate preserves OpenShell's default-deny - posture: with no setting at either scope, every proposal lands in - `pending` for human review, even when the prover sees no findings. + only when *all three* conditions hold: (a) `proposal_approval_mode` + resolves to `"auto"` — gateway scope wins, sandbox scope is the + per-sandbox override, default is `"manual"`; (b) the prover delta is empty + (`prover: no new findings`); and (c) the security notes recomputed from + the chunk's current proposed rule are empty (see + [Security-notes gate](#security-notes-gate)). Before merging, the gateway + reloads the stored chunk and reruns both checks on its current rule. This is + important after edits and mechanistic deduplication: the stored rule, not a + duplicate incoming payload or stale persisted analysis, controls the + decision. The recalculated prover verdict is decision-local rather than + persisted, so `validation_result` reads can still show the submit-time + verdict after an edit or deduplication. Decode, prover, or merge failures + leave the chunk pending. The audit event uses `CONFIG:APPROVED` and carries + `auto=true`, `source=`, `prover_delta=empty`, and + `resolved_from=` as unmapped fields, with message text + `"auto-approved: no new prover findings"` — never `safe`. The opt-in gate + preserves OpenShell's default-deny posture: with no setting at either + scope, every proposal lands in `pending` for human review, even + when the prover sees no findings. 4. **Implicit supersede.** On any successful submission, the gateway scans the sandbox's pending chunks for matches on `(host, port, binary)` and auto-rejects the older ones with reason `"superseded by chunk X"`. This @@ -152,6 +160,30 @@ through the proposal loop instead of treating the denial as terminal. policy. 6. **Escalation.** Anything else lands in `pending` for human review. +### Security-notes gate + +Separately from the prover, each chunk carries advisory `security_notes`. +Reads, bulk approval, and auto-approval regenerate them from the current +stored proposed rule instead of trusting a persisted value that may be stale +after an edit. Non-empty notes block auto-approval and make +`ApproveAllDraftChunks` skip the chunk unless `include_security_flagged` is +set. The chunk stays `pending`; an explicit human approval can still merge a +flagged chunk. + +Private/internal destinations are advisory, not blocking. A literal endpoint +IP, `allowed_ips` entry, or CIDR intersection in RFC 1918, CGNAT +`100.64.0.0/10`, IPv6 ULA `fc00::/7`, or another special-use range covered by +`openshell-core` `net::is_internal_net` produces a note. A hostless rule +carrying `allowed_ips` earns an extra note because it can match any hostname +resolving into the range. + +Always-blocked destinations are separate from this advisory classification. +Loopback, link-local, and unspecified IPs/CIDRs, plus `localhost` and known +metadata endpoint hostnames, are excluded from security notes. Submit and edit +may store such a draft, but existing merge validation rejects it when an +approval attempts to add it to policy; runtime SSRF protections remain the +final enforcement boundary. + ## What the prover decides The prover answers four formal questions about each proposed policy diff --git a/crates/openshell-cli/Cargo.toml b/crates/openshell-cli/Cargo.toml index 1cbfd7d474..d7b8fd502c 100644 --- a/crates/openshell-cli/Cargo.toml +++ b/crates/openshell-cli/Cargo.toml @@ -19,6 +19,7 @@ openshell-bootstrap = { path = "../openshell-bootstrap" } openshell-core = { path = "../openshell-core", default-features = false } openshell-policy = { path = "../openshell-policy" } openshell-providers = { path = "../openshell-providers" } +openshell-sdk = { path = "../openshell-sdk" } openshell-tui = { path = "../openshell-tui" } serde = { workspace = true } serde_json = { workspace = true } diff --git a/crates/openshell-cli/src/commands/common.rs b/crates/openshell-cli/src/commands/common.rs new file mode 100644 index 0000000000..7fa5cd1fec --- /dev/null +++ b/crates/openshell-cli/src/commands/common.rs @@ -0,0 +1,950 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared helpers, types, and parsing utilities used across CLI command groups. + +use chrono::DateTime; +use dialoguer::{Confirm, theme::ColorfulTheme}; +use indicatif::{MultiProgress, ProgressBar, ProgressStyle}; +use miette::{IntoDiagnostic, Result}; +use openshell_core::progress::{ + PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, + PROGRESS_COMPLETE_STEP_KEY, PROGRESS_STEP_PULLING_IMAGE, PROGRESS_STEP_REQUESTING_SANDBOX, + PROGRESS_STEP_STARTING_SANDBOX, +}; +use openshell_core::proto::{ + PlatformEvent, SandboxPhase, SandboxPolicy, SettingValue, setting_value, +}; +use openshell_core::settings::{self, SettingValueKind}; +use owo_colors::OwoColorize; +use std::collections::HashMap; +use std::io::IsTerminal; +use std::process::Command; +use std::time::{Duration, Instant}; + +// --------------------------------------------------------------------------- +// View types +// --------------------------------------------------------------------------- + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum PolicyGetView { + Metadata, + Base, + Full, +} + +impl PolicyGetView { + pub fn from_flags(base: bool, full: bool) -> Self { + match (base, full) { + (true, _) => Self::Base, + (false, true) => Self::Full, + (false, false) => Self::Metadata, + } + } + + pub(crate) fn includes_policy(self) -> bool { + matches!(self, Self::Base | Self::Full) + } +} + +// --------------------------------------------------------------------------- +// Formatting / display helpers +// --------------------------------------------------------------------------- + +/// Convert a sandbox phase integer to a human-readable string. +pub fn phase_name(phase: i32) -> &'static str { + match SandboxPhase::try_from(phase) { + Ok(SandboxPhase::Unspecified) => "Unspecified", + Ok(SandboxPhase::Provisioning) => "Provisioning", + Ok(SandboxPhase::Ready) => "Ready", + Ok(SandboxPhase::Error) => "Error", + Ok(SandboxPhase::Deleting) => "Deleting", + Ok(SandboxPhase::Unknown) | Err(_) => "Unknown", + } +} + +/// Format milliseconds since Unix epoch as a `YYYY-MM-DD HH:MM:SS` UTC string. +pub fn format_epoch_ms(ms: i64) -> String { + use std::time::UNIX_EPOCH; + + let Ok(ms_u64) = u64::try_from(ms) else { + return "-".to_string(); + }; + let Ok(time) = UNIX_EPOCH + .checked_add(Duration::from_millis(ms_u64)) + .ok_or(()) + else { + return "-".to_string(); + }; + let Ok(dur) = time.duration_since(UNIX_EPOCH) else { + return "-".to_string(); + }; + + let secs = dur.as_secs(); + let days = secs / 86400; + let time_of_day = secs % 86400; + let hours = time_of_day / 3600; + let minutes = (time_of_day % 3600) / 60; + let seconds = time_of_day % 60; + + let (y, m, d) = civil_from_days(days); + format!("{y:04}-{m:02}-{d:02} {hours:02}:{minutes:02}:{seconds:02}") +} + +/// Convert days since 1970-01-01 to (year, month, day). +/// Algorithm from Howard Hinnant's `chrono`-compatible date library. +fn civil_from_days(days: u64) -> (i64, u64, u64) { + let z = days.cast_signed() + 719_468; + let era = if z >= 0 { z } else { z - 146_096 } / 146_097; + let doe = (z - era * 146_097).cast_unsigned(); + let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146_096) / 365; + let y = yoe.cast_signed() + era * 400; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let d = doy - (153 * mp + 2) / 5 + 1; + let m = if mp < 10 { mp + 3 } else { mp - 9 }; + let y = if m <= 2 { y + 1 } else { y }; + (y, m, d) +} + +pub fn format_optional_epoch_ms(ms: i64) -> String { + if ms > 0 { + format_epoch_ms(ms) + } else { + "-".to_string() + } +} + +/// Format a duration as a compact elapsed time string, e.g. `(3s)` or `(1m 12s)`. +pub fn format_elapsed(d: Duration) -> String { + let secs = d.as_secs(); + if secs < 60 { + format!("({secs}s)") + } else { + let mins = secs / 60; + let rem = secs % 60; + format!("({mins}m {rem}s)") + } +} + +/// Format a total elapsed time for non-interactive mode timestamps. +pub fn format_timestamp(d: Duration) -> String { + let secs = d.as_secs_f64(); + format!("[{secs:.1}s]") +} + +/// Format a millisecond timestamp into a readable string. +pub fn format_timestamp_ms(ms: i64) -> String { + if ms <= 0 { + return "-".to_string(); + } + let secs = ms / 1000; + let mins = (secs / 60) % 60; + let hours = (secs / 3600) % 24; + let days = secs / 86400; + if days > 0 { + format!("{days}d {hours:02}:{mins:02}") + } else { + format!("{hours:02}:{mins:02}") + } +} + +pub fn truncate_status_field(value: &str, max_chars: usize) -> String { + if value.is_empty() { + return "-".to_string(); + } + let mut chars = value.chars(); + let truncated = chars.by_ref().take(max_chars).collect::(); + if chars.next().is_some() { + format!("{truncated}...") + } else { + truncated + } +} + +pub fn truncate_display(value: &str, max_width: usize) -> String { + if value.chars().count() <= max_width { + return value.to_string(); + } + + let keep = max_width.saturating_sub(3); + let mut truncated = value.chars().take(keep).collect::(); + truncated.push_str("..."); + truncated +} + +pub fn short_hash(hash: &str) -> &str { + if hash.len() >= 12 { &hash[..12] } else { hash } +} + +pub fn non_empty_or<'a>(value: &'a str, fallback: &'a str) -> &'a str { + if value.is_empty() { fallback } else { value } +} + +pub fn format_setting_value(value: Option<&SettingValue>) -> String { + let Some(value) = value.and_then(|v| v.value.as_ref()) else { + return "".to_string(); + }; + match value { + setting_value::Value::StringValue(v) => v.clone(), + setting_value::Value::BoolValue(v) => v.to_string(), + setting_value::Value::IntValue(v) => v.to_string(), + setting_value::Value::BytesValue(v) => format!("", v.len()), + } +} + +// --------------------------------------------------------------------------- +// YAML / policy display +// --------------------------------------------------------------------------- + +pub fn print_yaml_line(line: &str) { + let trimmed = line.trim_start(); + let indent = &line[..line.len() - trimmed.len()]; + + if let Some(rest) = trimmed.strip_prefix("- ") { + print!("{indent}"); + print!("{}", "- ".dimmed()); + print!("{rest}"); + println!(); + return; + } + + if let Some(colon_pos) = trimmed.find(':') { + let key = &trimmed[..colon_pos]; + let after_colon = &trimmed[colon_pos + 1..]; + + print!("{indent}"); + print!("{}", key.dimmed()); + print!("{}", ":".dimmed()); + + if after_colon.is_empty() { + // Key with nested content (no value on this line) + } else if let Some(value) = after_colon.strip_prefix(' ') { + print!(" {value}"); + } else { + print!("{after_colon}"); + } + println!(); + return; + } + + println!("{line}"); +} + +/// Print sandbox policy as YAML with dimmed keys. +pub fn print_sandbox_policy(policy: &SandboxPolicy) { + println!("{}", "Policy:".cyan().bold()); + println!(); + if let Ok(yaml_str) = openshell_policy::serialize_sandbox_policy(policy) { + for line in yaml_str.lines() { + if line == "---" { + continue; + } + print!(" "); + print_yaml_line(line); + } + } +} + +pub fn print_policy_merge_warnings(warnings: &[openshell_policy::PolicyMergeWarning]) { + for warning in warnings { + eprintln!("{} {}", "!".yellow().bold(), warning); + } +} + +// --------------------------------------------------------------------------- +// Provisioning display +// --------------------------------------------------------------------------- + +/// Known provisioning steps derived from Kubernetes events and sandbox lifecycle. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum ProvisioningStep { + /// Sandbox CRD created, waiting for pod to be scheduled. + RequestingSandbox, + /// Pulling the sandbox container image. + PullingSandboxImage, + /// Container is starting up. + StartingSandbox, +} + +impl ProvisioningStep { + /// Human-readable label for a completed step. + pub fn completed_label(self) -> &'static str { + match self { + Self::RequestingSandbox => "Sandbox allocated", + Self::PullingSandboxImage => "Image pulled", + Self::StartingSandbox => "Sandbox ready", + } + } + + /// Human-readable label for an in-progress step (shown on the spinner). + pub fn active_label(self) -> &'static str { + match self { + Self::RequestingSandbox => "Requesting sandbox...", + Self::PullingSandboxImage => "Pulling image...", + Self::StartingSandbox => "Starting sandbox...", + } + } +} + +/// Live-updating display showing a provisioning step checklist with spinner. +/// +/// Completed steps are printed as static `✓ Step` lines. The current +/// in-progress step is shown on a spinner with elapsed time. +pub struct ProvisioningDisplay { + mp: MultiProgress, + spinner: ProgressBar, + /// Blank line below the spinner so progress doesn't sit flush against + /// the bottom of the terminal. + spacer: ProgressBar, + /// Steps that have been completed, in order. + completed_steps: Vec, + /// Progress bars for completed steps (so they can be cleared). + completed_bars: Vec, + /// The currently active step label (shown on the spinner). + active_label: String, + /// Detail text shown next to the active step (e.g. image name). + active_detail: String, + /// When the current active step started (for elapsed time). + step_start: Instant, +} + +impl ProvisioningDisplay { + pub fn new() -> Self { + let mp = MultiProgress::new(); + + let spinner = mp.add(ProgressBar::new_spinner()); + spinner.set_style( + ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") + .unwrap_or_else(|_| ProgressStyle::default_spinner()), + ); + spinner.enable_steady_tick(Duration::from_millis(120)); + + let spacer = mp.add(ProgressBar::new(0)); + spacer.set_style( + ProgressStyle::with_template("{msg}").unwrap_or_else(|_| ProgressStyle::default_bar()), + ); + spacer.set_message(""); + + let now = Instant::now(); + Self { + mp, + spinner, + spacer, + completed_steps: Vec::new(), + completed_bars: Vec::new(), + active_label: ProvisioningStep::RequestingSandbox + .active_label() + .to_string(), + active_detail: String::new(), + step_start: now, + } + } + + /// Record a completed provisioning step with a custom label. + pub fn complete_step_with_label(&mut self, step: ProvisioningStep, label: &str) { + if self.completed_steps.contains(&step) { + return; + } + self.completed_steps.push(step); + + let elapsed = self.step_start.elapsed(); + let elapsed_str = format_elapsed(elapsed); + + let bar = self.mp.insert_before(&self.spinner, ProgressBar::new(0)); + bar.set_style( + ProgressStyle::with_template("{msg}").unwrap_or_else(|_| ProgressStyle::default_bar()), + ); + bar.set_message(format!( + "{} {} {}", + "\u{2713}".green().bold(), + label, + elapsed_str.dimmed() + )); + bar.finish(); + self.completed_bars.push(bar); + + self.step_start = Instant::now(); + self.spinner.reset_elapsed(); + self.active_detail.clear(); + } + + /// Set the active (in-progress) step shown on the spinner. + pub fn set_active(&mut self, label: &str) { + self.active_label = label.to_string(); + self.active_detail.clear(); + self.spinner.reset_elapsed(); + self.step_start = Instant::now(); + self.update_spinner(); + } + + /// Set the active step from a known provisioning step enum. + pub fn set_active_step(&mut self, step: ProvisioningStep) { + self.set_active(step.active_label()); + } + + /// Set detail text shown alongside the active step (e.g. image name). + pub fn set_active_detail(&mut self, detail: &str) { + self.active_detail = detail.to_string(); + self.update_spinner(); + } + + fn update_spinner(&self) { + let msg = if self.active_detail.is_empty() { + self.active_label.clone() + } else { + format!("{} {}", self.active_label, self.active_detail.dimmed()) + }; + self.spinner.set_message(msg); + } + + /// Finish with an error message shown on the last step line. + pub fn finish_error(&self, msg: &str) { + let _ = self + .mp + .println(format!("{} {}", "\u{2717}".red().bold(), msg.red())); + self.spinner.finish_and_clear(); + } + + /// Print a line above the progress bars (for static header content). + pub fn println(&self, msg: &str) { + let _ = self.mp.println(msg); + } + + /// Clear all progress output (spinner, spacer, and completed step lines). + pub fn clear(&self) { + self.spacer.finish_and_clear(); + self.spinner.finish_and_clear(); + for bar in &self.completed_bars { + bar.finish_and_clear(); + } + } +} + +// --------------------------------------------------------------------------- +// Provisioning progress event handling +// --------------------------------------------------------------------------- + +pub fn progress_step_from_metadata(value: &str) -> Option { + match value { + PROGRESS_STEP_REQUESTING_SANDBOX => Some(ProvisioningStep::RequestingSandbox), + PROGRESS_STEP_PULLING_IMAGE => Some(ProvisioningStep::PullingSandboxImage), + PROGRESS_STEP_STARTING_SANDBOX => Some(ProvisioningStep::StartingSandbox), + _ => None, + } +} + +pub fn noninteractive_active_label(step: ProvisioningStep) -> String { + step.active_label().trim_end_matches('.').to_string() +} + +pub fn handle_platform_progress_event( + event: &PlatformEvent, + display: &mut Option, + provision_start: Instant, +) -> bool { + let completed_step = event + .metadata + .get(PROGRESS_COMPLETE_STEP_KEY) + .and_then(|step| progress_step_from_metadata(step)); + let active_step = event + .metadata + .get(PROGRESS_ACTIVE_STEP_KEY) + .and_then(|step| progress_step_from_metadata(step)); + let active_detail = event + .metadata + .get(PROGRESS_ACTIVE_DETAIL_KEY) + .filter(|detail| !detail.is_empty()); + + let handled = completed_step.is_some() || active_step.is_some() || active_detail.is_some(); + if !handled { + return false; + } + + if let Some(step) = completed_step { + let label = event + .metadata + .get(PROGRESS_COMPLETE_LABEL_KEY) + .map_or_else(|| step.completed_label(), String::as_str); + if let Some(d) = display.as_mut() { + d.complete_step_with_label(step, label); + } else { + let ts = format_timestamp(provision_start.elapsed()); + println!("{} {}", ts.dimmed(), label); + } + } + + if let Some(step) = active_step + && let Some(d) = display.as_mut() + { + d.set_active_step(step); + } + + if let Some(detail) = active_detail { + if let Some(d) = display.as_mut() { + d.set_active_detail(detail); + } else { + let ts = format_timestamp(provision_start.elapsed()); + if let Some(step) = active_step { + println!( + "{} {} {}", + ts.dimmed(), + noninteractive_active_label(step), + detail + ); + } else { + println!("{} {}", ts.dimmed(), detail); + } + } + } + + true +} + +pub fn is_provisioning_progress_event(event: &PlatformEvent) -> bool { + if event.metadata.contains_key(PROGRESS_COMPLETE_STEP_KEY) + || event.metadata.contains_key(PROGRESS_ACTIVE_STEP_KEY) + || event.metadata.contains_key(PROGRESS_ACTIVE_DETAIL_KEY) + { + return true; + } + + event.source == "vm" + && matches!( + event.reason.as_str(), + "PullingLayer" + | "ResolvingImage" + | "AuthenticatingRegistry" + | "FetchingManifest" + | "CacheHit" + | "CacheMiss" + | "WaitingForImageCacheLock" + | "ExportingRootfs" + | "PreparingRootfs" + | "CreatingRootDisk" + | "PreparingOverlay" + | "Started" + ) +} + +// --------------------------------------------------------------------------- +// Sandbox header +// --------------------------------------------------------------------------- + +pub fn print_sandbox_header( + sandbox: &openshell_core::proto::Sandbox, + display: Option<&ProvisioningDisplay>, +) { + use openshell_core::ObjectName; + + let lines = [ + String::new(), + format!( + "{} {}", + "Created sandbox:".cyan().bold(), + sandbox.object_name().bold() + ), + String::new(), + ]; + match display { + Some(d) => { + for line in lines { + d.println(&line); + } + } + None => { + for line in lines { + println!("{line}"); + } + } + } +} + +// --------------------------------------------------------------------------- +// Sandbox provisioning helpers +// --------------------------------------------------------------------------- + +pub fn ready_false_condition_message( + status: Option<&openshell_core::proto::SandboxStatus>, +) -> Option { + let condition = status?.conditions.iter().find(|condition| { + condition.r#type == "Ready" && condition.status.eq_ignore_ascii_case("false") + })?; + + if condition.message.is_empty() { + if condition.reason.is_empty() { + None + } else { + Some(condition.reason.clone()) + } + } else if condition.reason.is_empty() { + Some(condition.message.clone()) + } else { + Some(format!("{}: {}", condition.reason, condition.message)) + } +} + +pub fn provisioning_timeout_message( + timeout_secs: u64, + resource_requirements: Option<&openshell_core::proto::ResourceRequirements>, + condition_message: Option<&str>, +) -> String { + let mut message = format!("sandbox provisioning timed out after {timeout_secs}s"); + + if let Some(condition_message) = condition_message.filter(|msg| !msg.is_empty()) { + message.push_str(". Last reported status: "); + message.push_str(condition_message); + } + + if resource_requirements.is_some_and(|requirements| requirements.gpu.is_some()) { + message.push_str( + ". Hint: this may be because the available GPU is already in use by another sandbox.", + ); + } + + message +} + +// --------------------------------------------------------------------------- +// Settings helpers +// --------------------------------------------------------------------------- + +pub fn parse_cli_setting_value(key: &str, raw_value: &str) -> Result { + let setting = settings::setting_for_key(key).ok_or_else(|| { + miette::miette!( + "unknown setting key '{}'. Allowed keys: {}", + key, + settings::registered_keys_csv() + ) + })?; + + let value = match setting.kind { + SettingValueKind::String => { + // Reject typos client-side so `openshell settings set ... + // proposal_approval_mode autom` errors immediately instead of + // round-tripping through the server. The server enforces the + // same check independently for non-CLI callers. + setting + .validate_string_value(raw_value) + .map_err(|allowed| { + miette::miette!( + "invalid value '{}' for key '{}'; expected one of: {}", + raw_value, + key, + allowed.join(", ") + ) + })?; + setting_value::Value::StringValue(raw_value.to_string()) + } + SettingValueKind::Int => { + let parsed = raw_value.trim().parse::().map_err(|_| { + miette::miette!( + "invalid int value '{}' for key '{}'; expected base-10 integer", + raw_value, + key + ) + })?; + setting_value::Value::IntValue(parsed) + } + SettingValueKind::Bool => { + let parsed = settings::parse_bool_like(raw_value).ok_or_else(|| { + miette::miette!( + "invalid bool value '{}' for key '{}'; expected one of: true,false,yes,no,1,0", + raw_value, + key + ) + })?; + setting_value::Value::BoolValue(parsed) + } + }; + + Ok(SettingValue { value: Some(value) }) +} + +pub fn confirm_global_setting_takeover(key: &str, yes: bool) -> Result<()> { + if yes { + return Ok(()); + } + + if !std::io::stdin().is_terminal() || !std::io::stdout().is_terminal() { + return Err(miette::miette!( + "global setting updates require confirmation; pass --yes in non-interactive mode" + )); + } + + let proceed = Confirm::with_theme(&ColorfulTheme::default()) + .with_prompt(format!( + "Setting '{key}' globally will disable sandbox-level management for this key. Continue?" + )) + .default(false) + .interact() + .into_diagnostic()?; + + if !proceed { + return Err(miette::miette!("aborted by user")); + } + + Ok(()) +} + +pub fn confirm_global_setting_delete(key: &str, yes: bool) -> Result<()> { + if yes { + return Ok(()); + } + + if !std::io::stdin().is_terminal() || !std::io::stdout().is_terminal() { + return Err(miette::miette!( + "global setting deletes require confirmation; pass --yes in non-interactive mode" + )); + } + + let proceed = Confirm::with_theme(&ColorfulTheme::default()) + .with_prompt(format!( + "Deleting global setting '{key}' re-enables sandbox-level management for this key. Continue?" + )) + .default(false) + .interact() + .into_diagnostic()?; + + if !proceed { + return Err(miette::miette!("aborted by user")); + } + + Ok(()) +} + +/// Parse a duration string like "5m", "1h", "30s" into milliseconds. +pub fn parse_duration_to_ms(s: &str) -> Result { + let s = s.trim(); + if s.is_empty() { + return Err(miette::miette!("empty duration string")); + } + let (num_str, unit) = s.split_at(s.len() - 1); + let num: i64 = num_str + .parse() + .map_err(|_| miette::miette!("invalid duration: {s} (expected e.g. 5m, 1h, 30s)"))?; + let multiplier = match unit { + "s" => 1_000, + "m" => 60_000, + "h" => 3_600_000, + _ => { + return Err(miette::miette!( + "unknown duration unit: {unit} (use s, m, or h)" + )); + } + }; + Ok(num * multiplier) +} + +// --------------------------------------------------------------------------- +// Parsing utilities +// --------------------------------------------------------------------------- + +pub fn parse_key_value_pairs(items: &[String], flag: &str) -> Result> { + let mut map = HashMap::new(); + + for item in items { + let Some((key, value)) = item.split_once('=') else { + return Err(miette::miette!("{flag} expects KEY=VALUE, got '{item}'")); + }; + + let key = key.trim(); + if key.is_empty() { + return Err(miette::miette!("{flag} key cannot be empty")); + } + + map.insert(key.to_string(), value.to_string()); + } + + Ok(map) +} + +pub fn parse_env_pairs(items: &[String]) -> Result> { + let map = parse_key_value_pairs(items, "--env")?; + for key in map.keys() { + if !is_valid_env_name(key) { + return Err(miette::miette!( + "--env key must match [A-Za-z_][A-Za-z0-9_]*; got '{key}'" + )); + } + if key.starts_with("OPENSHELL_") { + return Err(miette::miette!( + "--env keys starting with OPENSHELL_ are reserved; got '{key}'" + )); + } + } + Ok(map) +} + +/// Resolve `--secret-material-env KEY[=ENVVAR]` values from the CLI process +/// environment (`ENVVAR` defaults to `KEY`) so secrets never transit argv. +pub fn parse_secret_material_env_pairs(items: &[String]) -> Result> { + let mut map = HashMap::new(); + + for item in items { + let (key, env_name) = match item.split_once('=') { + Some((key, env_name)) => (key.trim(), env_name.trim()), + None => (item.trim(), item.trim()), + }; + if key.is_empty() { + return Err(miette::miette!("--secret-material-env key cannot be empty")); + } + if env_name.is_empty() { + return Err(miette::miette!( + "--secret-material-env {key} names an empty environment variable" + )); + } + + let value = std::env::var(env_name).map_err(|_| { + miette::miette!( + "--secret-material-env {key} requires local env var '{env_name}' to be set to a non-empty value" + ) + })?; + if value.trim().is_empty() { + return Err(miette::miette!( + "--secret-material-env {key} requires local env var '{env_name}' to be set to a non-empty value" + )); + } + + if map.contains_key(key) { + return Err(miette::miette!( + "--secret-material-env key '{key}' supplied more than once" + )); + } + map.insert(key.to_string(), value); + } + + Ok(map) +} + +pub fn is_valid_env_name(key: &str) -> bool { + let mut bytes = key.bytes(); + let Some(first) = bytes.next() else { + return false; + }; + if !(first == b'_' || first.is_ascii_alphabetic()) { + return false; + } + bytes.all(|b| b == b'_' || b.is_ascii_alphanumeric()) +} + +pub fn parse_credential_pairs(items: &[String]) -> Result> { + let mut map = HashMap::new(); + + for item in items { + if let Some((key, value)) = item.split_once('=') { + let key = key.trim(); + if key.is_empty() { + return Err(miette::miette!("--credential key cannot be empty")); + } + map.insert(key.to_string(), value.to_string()); + continue; + } + + let key = item.trim(); + if key.is_empty() { + return Err(miette::miette!("--credential key cannot be empty")); + } + + let value = std::env::var(key).map_err(|_| { + miette::miette!( + "--credential {key} requires local env var '{key}' to be set to a non-empty value" + ) + })?; + + if value.trim().is_empty() { + return Err(miette::miette!( + "--credential {key} requires local env var '{key}' to be set to a non-empty value" + )); + } + + map.insert(key.to_string(), value); + } + + Ok(map) +} + +pub fn parse_credential_expiry_cli_value(value: &str) -> std::result::Result { + parse_credential_expiry_value(value, None).map_err(|err| err.to_string()) +} + +fn credential_expiry_value_error(key: Option<&str>, detail: &str) -> miette::Report { + key.map_or_else( + || miette::miette!("--credential-expires-at value {detail}"), + |key| miette::miette!("--credential-expires-at value for '{key}' {detail}"), + ) +} + +pub fn parse_credential_expiry_value(value: &str, key: Option<&str>) -> Result { + let value = value.trim(); + if value.is_empty() { + return Err(credential_expiry_value_error(key, "cannot be empty")); + } + + if let Ok(value_ms) = value.parse::() { + if value_ms < 0 { + return Err(credential_expiry_value_error( + key, + "must be greater than or equal to 0", + )); + } + return Ok(value_ms); + } + + let parsed = DateTime::parse_from_rfc3339(value).map_err(|_| { + credential_expiry_value_error( + key, + "must be a Unix epoch millisecond timestamp or RFC3339 timestamp", + ) + })?; + let value_ms = parsed.timestamp_millis(); + if value_ms < 0 { + return Err(credential_expiry_value_error( + key, + "must be greater than or equal to 0", + )); + } + + Ok(value_ms) +} + +pub fn parse_credential_expiry_pairs(items: &[String]) -> Result> { + let mut map = HashMap::new(); + + for item in items { + let Some((key, value)) = item.split_once('=') else { + return Err(miette::miette!( + "--credential-expires-at expects KEY=TIMESTAMP, got '{item}'" + )); + }; + let key = key.trim(); + if key.is_empty() { + return Err(miette::miette!( + "--credential-expires-at key cannot be empty" + )); + } + let value = parse_credential_expiry_value(value, Some(key))?; + map.insert(key.to_string(), value); + } + + Ok(map) +} + +// --------------------------------------------------------------------------- +// Git environment helpers +// --------------------------------------------------------------------------- + +pub fn scrub_git_env(command: &mut Command) -> &mut Command { + for key in [ + "GIT_DIR", + "GIT_WORK_TREE", + "GIT_INDEX_FILE", + "GIT_PREFIX", + "GIT_COMMON_DIR", + "GIT_OBJECT_DIRECTORY", + "GIT_ALTERNATE_OBJECT_DIRECTORIES", + ] { + command.env_remove(key); + } + command +} diff --git a/crates/openshell-cli/src/commands/mod.rs b/crates/openshell-cli/src/commands/mod.rs new file mode 100644 index 0000000000..ab20f224db --- /dev/null +++ b/crates/openshell-cli/src/commands/mod.rs @@ -0,0 +1,4 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +pub mod common; diff --git a/crates/openshell-cli/src/completers.rs b/crates/openshell-cli/src/completers.rs index 572d601eaa..9d3b88d594 100644 --- a/crates/openshell-cli/src/completers.rs +++ b/crates/openshell-cli/src/completers.rs @@ -140,7 +140,13 @@ async fn completion_grpc_client( Some("oidc") => { if let Some(bundle) = load_oidc_token(gateway_name) { if is_token_expired(&bundle) { - match oidc_refresh_token(&bundle, tls_opts.gateway_insecure).await { + match oidc_refresh_token( + &bundle, + meta.oidc_scopes.as_deref(), + tls_opts.gateway_insecure, + ) + .await + { Ok(refreshed) => { let _ = store_oidc_token(gateway_name, &refreshed); tls_opts.oidc_token = Some(refreshed.access_token); diff --git a/crates/openshell-cli/src/lib.rs b/crates/openshell-cli/src/lib.rs index 156668951a..ecdf54f28f 100644 --- a/crates/openshell-cli/src/lib.rs +++ b/crates/openshell-cli/src/lib.rs @@ -9,6 +9,7 @@ pub(crate) static TEST_ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); pub mod auth; +pub(crate) mod commands; pub mod completers; pub mod edge_tunnel; pub mod oidc_auth; diff --git a/crates/openshell-cli/src/main.rs b/crates/openshell-cli/src/main.rs index df4e983e5b..00942d79ec 100644 --- a/crates/openshell-cli/src/main.rs +++ b/crates/openshell-cli/src/main.rs @@ -157,19 +157,30 @@ fn resolve_gateway_name(gateway_flag: &Option) -> Option { /// and setting it on `TlsOptions`. For OIDC, automatically refreshes the token /// if it's near expiry. fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) { - let Some(meta) = get_gateway_metadata(gateway_name) else { - return; - }; + let _ = apply_auth_with_status(tls, gateway_name); +} + +/// Apply stored authentication and return a user-facing preparation failure, +/// if the CLI already knows the credentials cannot be used. +fn apply_auth_with_status(tls: &mut TlsOptions, gateway_name: &str) -> Option { + let meta = get_gateway_metadata(gateway_name)?; match meta.auth_mode.as_deref() { Some("cloudflare_jwt") => { if let Some(token) = load_edge_token(gateway_name) { tls.edge_token = Some(token); + None + } else { + Some(format!( + "edge credentials are missing; run `openshell gateway login {gateway_name}`" + )) } } Some("oidc") => { let Some(bundle) = openshell_bootstrap::oidc_token::load_oidc_token(gateway_name) else { - return; + return Some(format!( + "OIDC credentials are missing; run `openshell gateway login {gateway_name}`" + )); }; if openshell_bootstrap::oidc_token::is_token_expired(&bundle) { let insecure = std::env::var("OPENSHELL_GATEWAY_INSECURE") @@ -178,7 +189,11 @@ fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) { // so the async refresh can run within the sync apply_auth call. match tokio::task::block_in_place(|| { tokio::runtime::Handle::current().block_on( - openshell_cli::oidc_auth::oidc_refresh_token(&bundle, insecure), + openshell_cli::oidc_auth::oidc_refresh_token( + &bundle, + meta.oidc_scopes.as_deref(), + insecure, + ), ) }) { Ok(refreshed) => { @@ -187,19 +202,24 @@ fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) { &refreshed, ); tls.oidc_token = Some(refreshed.access_token); + None } Err(e) => { tracing::warn!("OIDC token refresh failed: {e}"); // Use the expired token anyway — server will reject it // with a clear error prompting re-login. tls.oidc_token = Some(bundle.access_token); + Some(format!( + "OIDC token refresh failed; run `openshell gateway login {gateway_name}`" + )) } } } else { tls.oidc_token = Some(bundle.access_token); + None } } - _ => {} + _ => None, } } @@ -562,7 +582,11 @@ enum Commands { /// Show gateway status and information. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Status, + Status { + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, + }, /// Manage inference configuration. #[command(after_help = INFERENCE_EXAMPLES, help_template = SUBCOMMAND_HELP_TEMPLATE)] @@ -1414,6 +1438,10 @@ enum SandboxCommands { #[arg(long, value_parser = ["manual", "auto"], default_value = "manual")] approval_mode: String, + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table, conflicts_with_all = ["editor", "command", "no_keep", "forward"])] + output: OutputFormat, + /// Command to run after "--" (defaults to an interactive shell). #[arg(last = true, allow_hyphen_values = true)] command: Vec, @@ -1427,8 +1455,12 @@ enum SandboxCommands { name: Option, /// Print only the active policy YAML (same policy as the default view; stdout only). - #[arg(long)] + #[arg(long, conflicts_with = "output")] policy_only: bool, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, }, /// List sandboxes. @@ -2250,11 +2282,24 @@ async fn main() -> Result<()> { // ----------------------------------------------------------- // Top-level status // ----------------------------------------------------------- - Some(Commands::Status) => { + Some(Commands::Status { output }) => { if let Ok(ctx) = resolve_gateway(&cli.gateway, &cli.gateway_endpoint) { let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); - run::gateway_status(&ctx.name, &ctx.endpoint, &tls).await?; + let auth_error = apply_auth_with_status(&mut tls, &ctx.name); + run::gateway_status( + &ctx.name, + &ctx.endpoint, + output.as_str(), + &tls, + auth_error.as_deref(), + ) + .await?; + } else if openshell_cli::output::print_output_single( + output.as_str(), + &(), + |()| serde_json::json!({"status": "not_configured"}), + )? { + // Structured output handled. } else { println!("{}", "Gateway Status".cyan().bold()); println!(); @@ -2865,6 +2910,7 @@ async fn main() -> Result<()> { labels, envs, approval_mode, + output, command, } => { // Resolve --tty / --no-tty into an Option override. @@ -2951,6 +2997,7 @@ async fn main() -> Result<()> { labels: labels_map, environment: env_map, approval_mode: &approval_mode, + output: output.as_str(), }, &cli.workspace, &tls, @@ -3010,10 +3057,21 @@ async fn main() -> Result<()> { | SandboxCommands::Download { .. } => { unreachable!() } - SandboxCommands::Get { name, policy_only } => { + SandboxCommands::Get { + name, + policy_only, + output, + } => { let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; - run::sandbox_get(endpoint, &name, policy_only, &cli.workspace, &tls) - .await?; + run::sandbox_get( + endpoint, + &name, + policy_only, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; } SandboxCommands::List { limit, @@ -3863,7 +3921,7 @@ mod tests { .expect("global gateway flag should parse with subcommands"); assert_eq!(cli.gateway.as_deref(), Some("demo")); - assert!(matches!(cli.command, Some(Commands::Status))); + assert!(matches!(cli.command, Some(Commands::Status { .. }))); } #[test] @@ -4150,6 +4208,25 @@ mod tests { }); } + #[test] + fn apply_auth_reports_missing_edge_credentials() { + let tmp = tempfile::tempdir().unwrap(); + with_tmp_xdg(tmp.path(), || { + store_gateway_metadata( + "edge-gateway", + &edge_metadata("edge-gateway", "https://gw.example.com"), + ) + .unwrap(); + + let mut tls = TlsOptions::default(); + let error = apply_auth_with_status(&mut tls, "edge-gateway") + .expect("missing credentials should be reported"); + + assert!(error.contains("edge credentials are missing")); + assert!(!tls.is_bearer_auth()); + }); + } + /// Verify the flag names the TUI uses to build its `ProxyCommand` are /// accepted by the `SshProxy` subcommand and land in the right fields. /// This catches drift when CLI flags are renamed or restructured. @@ -4986,6 +5063,25 @@ mod tests { ); } + #[test] + fn sandbox_create_output_conflicts_with_side_effect_args() { + for (label, extra_args) in [ + ("--editor", &["--editor", "code"][..]), + ("trailing command", &["--", "claude"][..]), + ("--no-keep", &["--no-keep"][..]), + ("--forward", &["--forward", "8080"][..]), + ] { + let args = ["openshell", "sandbox", "create", "--output", "json"] + .into_iter() + .chain(extra_args.iter().copied()); + let result = Cli::try_parse_from(args); + assert!( + result.is_err(), + "structured output should conflict with {label}" + ); + } + } + #[test] fn sandbox_create_resource_flags_parse() { let cli = Cli::try_parse_from([ diff --git a/crates/openshell-cli/src/oidc_auth.rs b/crates/openshell-cli/src/oidc_auth.rs index 379a531127..2aacdb0c9c 100644 --- a/crates/openshell-cli/src/oidc_auth.rs +++ b/crates/openshell-cli/src/oidc_auth.rs @@ -17,9 +17,10 @@ use miette::{IntoDiagnostic, Result}; use oauth2::basic::BasicClient; use oauth2::{ AuthType, AuthUrl, AuthorizationCode, ClientId, ClientSecret, CsrfToken, PkceCodeChallenge, - RedirectUrl, RefreshToken, Scope, TokenResponse, TokenUrl, + RedirectUrl, Scope, TokenResponse, TokenUrl, }; use openshell_bootstrap::oidc_token::OidcTokenBundle; +use openshell_sdk::oidc::RefreshTokenInput; use serde::Deserialize; use std::convert::Infallible; use std::sync::{Arc, Mutex}; @@ -227,10 +228,14 @@ pub async fn oidc_client_credentials_flow( /// Refresh an OIDC token using the `refresh_token` grant. /// +/// Reuses the configured login scopes when supplied so providers can select +/// the same API resource for the refreshed access token. +/// /// Preserves the existing refresh token if the server does not return a new /// one (per OAuth 2.0 spec, the refresh response may omit `refresh_token`). pub async fn oidc_refresh_token( bundle: &OidcTokenBundle, + scopes: Option<&str>, insecure: bool, ) -> Result { let refresh_token = bundle.refresh_token.as_deref().ok_or_else(|| { @@ -239,24 +244,37 @@ pub async fn oidc_refresh_token( ) })?; - let discovery = discover(&bundle.issuer, insecure).await?; - - let client = BasicClient::new(ClientId::new(bundle.client_id.clone())) - .set_token_uri(TokenUrl::new(discovery.token_endpoint).into_diagnostic()?); - - let http = http_client(insecure); - let token_response = client - .exchange_refresh_token(&RefreshToken::new(refresh_token.to_string())) - .request_async(&http) + let scopes = scopes.map_or_else(Vec::new, |scopes| { + scopes.split_whitespace().map(str::to_owned).collect() + }); + let input = RefreshTokenInput::new(refresh_token, &bundle.issuer, &bundle.client_id) + .with_scopes(scopes) + .with_insecure(insecure); + let refreshed = openshell_sdk::oidc::refresh_token(&input) .await - .map_err(|e| miette::miette!("token refresh failed: {e}"))?; + .map_err(miette::Report::new)?; - let mut refreshed = - bundle_from_oauth2_response(&token_response, &bundle.issuer, &bundle.client_id); - if refreshed.refresh_token.is_none() { - refreshed.refresh_token.clone_from(&bundle.refresh_token); + Ok(bundle_from_refresh_output( + bundle, + refreshed.access_token, + refreshed.refresh_token, + refreshed.expires_at, + )) +} + +fn bundle_from_refresh_output( + previous: &OidcTokenBundle, + access_token: String, + refresh_token: Option, + expires_at: Option, +) -> OidcTokenBundle { + OidcTokenBundle { + access_token, + refresh_token: refresh_token.or_else(|| previous.refresh_token.clone()), + expires_at, + issuer: previous.issuer.clone(), + client_id: previous.client_id.clone(), } - Ok(refreshed) } /// Ensure we have a valid OIDC token for the given gateway, refreshing if needed. @@ -279,7 +297,10 @@ pub async fn ensure_valid_oidc_token(gateway_name: &str, insecure: bool) -> Resu gateway = gateway_name, "OIDC token expired, attempting refresh" ); - let refreshed = oidc_refresh_token(&bundle, insecure).await?; + let scopes = openshell_bootstrap::load_gateway_metadata(gateway_name) + .ok() + .and_then(|metadata| metadata.oidc_scopes); + let refreshed = oidc_refresh_token(&bundle, scopes.as_deref(), insecure).await?; openshell_bootstrap::oidc_token::store_oidc_token(gateway_name, &refreshed)?; Ok(refreshed.access_token) } @@ -531,4 +552,24 @@ mod tests { assert_eq!(bundle.client_id, "my-client"); assert!(bundle.expires_at.is_some()); } + + #[test] + fn refresh_output_preserves_omitted_refresh_token() { + let previous = OidcTokenBundle { + access_token: "expired-access".to_string(), + refresh_token: Some("refresh-secret".to_string()), + expires_at: Some(0), + issuer: "https://issuer.example".to_string(), + client_id: "client-id".to_string(), + }; + + let refreshed = + bundle_from_refresh_output(&previous, "refreshed-access".to_string(), None, Some(300)); + + assert_eq!(refreshed.access_token, "refreshed-access"); + assert_eq!(refreshed.refresh_token, previous.refresh_token); + assert_eq!(refreshed.issuer, previous.issuer); + assert_eq!(refreshed.client_id, previous.client_id); + assert_eq!(refreshed.expires_at, Some(300)); + } } diff --git a/crates/openshell-cli/src/run.rs b/crates/openshell-cli/src/run.rs index 7b68487ea1..de346d4377 100644 --- a/crates/openshell-cli/src/run.rs +++ b/crates/openshell-cli/src/run.rs @@ -3,20 +3,34 @@ //! CLI command implementations. +pub use crate::commands::common::{ + PolicyGetView, parse_credential_expiry_cli_value, parse_env_pairs, parse_key_value_pairs, + parse_secret_material_env_pairs, +}; +use crate::commands::common::{ + ProvisioningDisplay, ProvisioningStep, confirm_global_setting_delete, + confirm_global_setting_takeover, format_epoch_ms, format_optional_epoch_ms, + format_setting_value, format_timestamp, format_timestamp_ms, handle_platform_progress_event, + is_provisioning_progress_event, non_empty_or, parse_cli_setting_value, + parse_credential_expiry_pairs, parse_credential_pairs, parse_duration_to_ms, phase_name, + print_policy_merge_warnings, print_sandbox_header, print_sandbox_policy, + provisioning_timeout_message, ready_false_condition_message, scrub_git_env, short_hash, + truncate_display, truncate_status_field, +}; + use crate::policy_update::build_policy_update_plan; use crate::tls::{ TlsOptions, build_insecure_rustls_config, build_rustls_config, grpc_client, grpc_inference_client, require_tls_materials, }; use bytes::Bytes; -use chrono::DateTime; use dialoguer::{Confirm, Select, theme::ColorfulTheme}; use futures::StreamExt; use http_body_util::Full; use hyper::{Request, StatusCode}; use hyper_rustls::HttpsConnectorBuilder; use hyper_util::{client::legacy::Client, rt::TokioExecutor}; -use indicatif::{MultiProgress, ProgressBar, ProgressStyle}; +use indicatif::{ProgressBar, ProgressStyle}; use miette::{IntoDiagnostic, Result, WrapErr, miette}; use openshell_bootstrap::{ GatewayMetadata, clear_active_gateway, clear_last_sandbox_if_matches, @@ -27,11 +41,6 @@ use openshell_bootstrap::{ use openshell_bootstrap::{ GatewayMetadataSource, ListedGateway, gateway_metadata_source, list_gateways_with_source, }; -use openshell_core::progress::{ - PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, - PROGRESS_COMPLETE_STEP_KEY, PROGRESS_STEP_PULLING_IMAGE, PROGRESS_STEP_REQUESTING_SANDBOX, - PROGRESS_STEP_STARTING_SANDBOX, -}; use openshell_core::proto::ProviderProfileCategory; use openshell_core::proto::{ ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, AttachSandboxProviderRequest, @@ -42,21 +51,20 @@ use openshell_core::proto::{ ExposeServiceRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest, GetGatewayInfoRequest, GetInferenceRouteRequest, GetProviderProfileRequest, GetProviderRefreshStatusRequest, GetProviderRequest, GetSandboxConfigRequest, - GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, GetSandboxRequest, GetServiceRequest, - GpuResourceRequirements, HealthRequest, ImportProviderProfilesRequest, - LintProviderProfilesRequest, ListProviderProfilesRequest, ListProvidersRequest, - ListSandboxPoliciesRequest, ListSandboxProvidersRequest, ListSandboxesRequest, - ListServicesRequest, PlatformEvent, PolicySource, PolicyStatus, Provider, + GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, + GetSandboxRequest, GetServiceRequest, GpuResourceRequirements, HealthRequest, + ImportProviderProfilesRequest, LintProviderProfilesRequest, ListProviderProfilesRequest, + ListProvidersRequest, ListSandboxPoliciesRequest, ListSandboxProvidersRequest, + ListSandboxesRequest, ListServicesRequest, PolicySource, PolicyStatus, Provider, ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy, ProviderProfile, ProviderProfileDiagnostic, ProviderProfileImportItem, RejectDraftChunkRequest, ResourceRequirements, RevokeSshSessionRequest, RotateProviderCredentialRequest, Sandbox, SandboxPhase, SandboxPolicy, SandboxSpec, SandboxTemplate, ServiceEndpointResponse, - ServiceStatus, SetInferenceRouteRequest, SettingScope, SettingValue, TcpForwardFrame, - TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, UpdateProviderProfilesRequest, - UpdateProviderRequest, WatchSandboxRequest, exec_sandbox_event, setting_value, - tcp_forward_init, + ServiceStatus, SetInferenceRouteRequest, SettingScope, TcpForwardFrame, TcpForwardInit, + TcpRelayTarget, UpdateConfigRequest, UpdateProviderProfilesRequest, UpdateProviderRequest, + WatchSandboxRequest, exec_sandbox_event, setting_value, tcp_forward_init, }; -use openshell_core::settings::{self, SettingValueKind}; +use openshell_core::settings; use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; use openshell_providers::{ ProviderRegistry, ProviderTypeProfile, RealDiscoveryContext, detect_provider_from_command, @@ -82,27 +90,6 @@ pub use openshell_core::forward::{ ForwardSpec, find_forward_by_port, list_forwards, stop_forward, stop_forwards_for_sandbox, }; -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum PolicyGetView { - Metadata, - Base, - Full, -} - -impl PolicyGetView { - pub fn from_flags(base: bool, full: bool) -> Self { - match (base, full) { - (true, _) => Self::Base, - (false, true) => Self::Full, - (false, false) => Self::Metadata, - } - } - - fn includes_policy(self) -> bool { - matches!(self, Self::Base | Self::Full) - } -} - #[derive(Debug, PartialEq, Eq)] enum SandboxUploadPlan { GitAware { @@ -113,496 +100,352 @@ enum SandboxUploadPlan { GitFilteredEmpty, } -/// Convert a sandbox phase integer to a human-readable string. -fn phase_name(phase: i32) -> &'static str { - match SandboxPhase::try_from(phase) { - Ok(SandboxPhase::Unspecified) => "Unspecified", - Ok(SandboxPhase::Provisioning) => "Provisioning", - Ok(SandboxPhase::Ready) => "Ready", - Ok(SandboxPhase::Error) => "Error", - Ok(SandboxPhase::Deleting) => "Deleting", - Ok(SandboxPhase::Unknown) | Err(_) => "Unknown", - } +#[derive(Debug, Clone)] +struct GatewayInfoView { + gateway: String, + server: String, + auth: Option<&'static str>, + status: String, + version: String, + compute_drivers: Vec, } -fn ready_false_condition_message( - status: Option<&openshell_core::proto::SandboxStatus>, -) -> Option { - let condition = status?.conditions.iter().find(|condition| { - condition.r#type == "Ready" && condition.status.eq_ignore_ascii_case("false") - })?; - - if condition.message.is_empty() { - if condition.reason.is_empty() { - None - } else { - Some(condition.reason.clone()) - } - } else if condition.reason.is_empty() { - Some(condition.message.clone()) - } else { - Some(format!("{}: {}", condition.reason, condition.message)) - } +#[derive(Debug, Clone)] +struct ComputeDriverInfoView { + name: String, + capabilities: ComputeDriverCapabilitiesView, } -fn provisioning_timeout_message( - timeout_secs: u64, - resource_requirements: Option<&ResourceRequirements>, - condition_message: Option<&str>, -) -> String { - let mut message = format!("sandbox provisioning timed out after {timeout_secs}s"); - - if let Some(condition_message) = condition_message.filter(|msg| !msg.is_empty()) { - message.push_str(". Last reported status: "); - message.push_str(condition_message); - } - - if resource_requirements.is_some_and(|requirements| requirements.gpu.is_some()) { - message.push_str( - ". Hint: this may be because the available GPU is already in use by another sandbox.", - ); - } - - message +#[derive(Debug, Clone)] +struct ComputeDriverCapabilitiesView { + driver_name: String, + driver_version: String, } -/// Format milliseconds since Unix epoch as a `YYYY-MM-DD HH:MM:SS` UTC string. -fn format_epoch_ms(ms: i64) -> String { - use std::time::UNIX_EPOCH; - - let Ok(ms_u64) = u64::try_from(ms) else { - return "-".to_string(); - }; - let Ok(time) = UNIX_EPOCH - .checked_add(Duration::from_millis(ms_u64)) - .ok_or(()) - else { - return "-".to_string(); - }; - let Ok(dur) = time.duration_since(UNIX_EPOCH) else { - return "-".to_string(); - }; +enum ProgressOutput { + Interactive(ProvisioningDisplay), + Plain, + Silent, +} - let secs = dur.as_secs(); - let days = secs / 86400; - let time_of_day = secs % 86400; - let hours = time_of_day / 3600; - let minutes = (time_of_day % 3600) / 60; - let seconds = time_of_day % 60; - - // Convert days since epoch to year-month-day using a basic civil calendar algorithm. - let (y, m, d) = civil_from_days(days); - format!("{y:04}-{m:02}-{d:02} {hours:02}:{minutes:02}:{seconds:02}") -} - -/// Convert days since 1970-01-01 to (year, month, day). -/// Algorithm from Howard Hinnant's `chrono`-compatible date library. -fn civil_from_days(days: u64) -> (i64, u64, u64) { - let z = days.cast_signed() + 719_468; - let era = if z >= 0 { z } else { z - 146_096 } / 146_097; - let doe = (z - era * 146_097).cast_unsigned(); - let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146_096) / 365; - let y = yoe.cast_signed() + era * 400; - let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); - let mp = (5 * doy + 2) / 153; - let d = doy - (153 * mp + 2) / 5 + 1; - let m = if mp < 10 { mp + 3 } else { mp - 9 }; - let y = if m <= 2 { y + 1 } else { y }; - (y, m, d) -} - -/// Known provisioning steps derived from Kubernetes events and sandbox lifecycle. -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] -enum ProvisioningStep { - /// Sandbox CRD created, waiting for pod to be scheduled. - RequestingSandbox, - /// Pulling the sandbox container image. - PullingSandboxImage, - /// Container is starting up. - StartingSandbox, -} - -impl ProvisioningStep { - /// Human-readable label for a completed step. - fn completed_label(self) -> &'static str { +impl ProgressOutput { + fn as_interactive_mut(&mut self) -> Option<&mut ProvisioningDisplay> { match self { - Self::RequestingSandbox => "Sandbox allocated", - Self::PullingSandboxImage => "Image pulled", - Self::StartingSandbox => "Sandbox ready", + Self::Interactive(d) => Some(d), + _ => None, } } - /// Human-readable label for an in-progress step (shown on the spinner). - fn active_label(self) -> &'static str { + fn as_interactive(&self) -> Option<&ProvisioningDisplay> { match self { - Self::RequestingSandbox => "Requesting sandbox...", - Self::PullingSandboxImage => "Pulling image...", - Self::StartingSandbox => "Starting sandbox...", + Self::Interactive(d) => Some(d), + _ => None, } } -} -/// Live-updating display showing a provisioning step checklist with spinner. -/// -/// Completed steps are printed as static `✓ Step` lines. The current -/// in-progress step is shown on a spinner with elapsed time. -struct ProvisioningDisplay { - mp: MultiProgress, - spinner: ProgressBar, - /// Blank line below the spinner so progress doesn't sit flush against - /// the bottom of the terminal. - spacer: ProgressBar, - /// Steps that have been completed, in order. - completed_steps: Vec, - /// Progress bars for completed steps (so they can be cleared). - completed_bars: Vec, - /// The currently active step label (shown on the spinner). - active_label: String, - /// Detail text shown next to the active step (e.g. image name). - active_detail: String, - /// When the current active step started (for elapsed time). - step_start: Instant, -} - -impl ProvisioningDisplay { - fn new() -> Self { - let mp = MultiProgress::new(); - - let spinner = mp.add(ProgressBar::new_spinner()); - spinner.set_style( - ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") - .unwrap_or_else(|_| ProgressStyle::default_spinner()), - ); - spinner.enable_steady_tick(Duration::from_millis(120)); - - // Always keep a blank line below the spinner so the progress area - // doesn't sit flush against the bottom of the terminal. - let spacer = mp.add(ProgressBar::new(0)); - spacer.set_style( - ProgressStyle::with_template("{msg}").unwrap_or_else(|_| ProgressStyle::default_bar()), - ); - spacer.set_message(""); - - let now = Instant::now(); - Self { - mp, - spinner, - spacer, - completed_steps: Vec::new(), - completed_bars: Vec::new(), - active_label: ProvisioningStep::RequestingSandbox - .active_label() - .to_string(), - active_detail: String::new(), - step_start: now, - } + fn is_plain(&self) -> bool { + matches!(self, Self::Plain) } +} - /// Record a completed provisioning step with a custom label. - fn complete_step_with_label(&mut self, step: ProvisioningStep, label: &str) { - // Don't duplicate steps we've already printed. - if self.completed_steps.contains(&step) { - return; +/// Show gateway status. +#[allow(clippy::branches_sharing_code)] +pub async fn gateway_status( + gateway_name: &str, + server: &str, + output: &str, + tls: &TlsOptions, + auth_preparation_error: Option<&str>, +) -> Result<()> { + // Build status data before any output. + let is_bearer = tls.is_bearer_auth(); + let (status_str, version, error, http_status, authentication): ( + &str, + Option, + Option, + Option, + GatewayAuthenticationState, + ) = match grpc_client(server, tls).await { + Ok(mut client) => match client.health(HealthRequest {}).await { + Ok(response) => { + let health = response.into_inner(); + let auth = match auth_preparation_error { + Some(e) => GatewayAuthenticationState::Failed(e.to_string()), + None => gateway_authentication_state( + client + .get_gateway_info(GetGatewayInfoRequest {}) + .await + .map(|_| ()), + tls, + server, + ), + }; + ("connected", Some(health.version), None, None, auth) + } + Err(e) => { + let auth = auth_preparation_error.map_or_else( + || { + GatewayAuthenticationState::Unverified( + "gRPC health check failed".to_string(), + ) + }, + |err| GatewayAuthenticationState::Failed(err.to_string()), + ); + http_health_check(server, tls).await?.map_or_else( + || ("error", None, Some(e.to_string()), None, auth.clone()), + |http| { + let hs = Some(http.to_string()); + if http.is_success() { + ( + "connected_http", + None, + Some(e.to_string()), + hs, + auth.clone(), + ) + } else { + ("error", None, Some(e.to_string()), hs, auth.clone()) + } + }, + ) + } + }, + Err(e) => { + let auth = auth_preparation_error.map_or_else( + || GatewayAuthenticationState::Unverified("gateway unreachable".to_string()), + |err| GatewayAuthenticationState::Failed(err.to_string()), + ); + http_health_check(server, tls).await?.map_or_else( + || { + ( + "disconnected", + None, + Some(e.to_string()), + None, + auth.clone(), + ) + }, + |http| { + let hs = Some(http.to_string()); + if http.is_success() { + ( + "connected_http", + None, + Some(e.to_string()), + hs, + auth.clone(), + ) + } else { + ("disconnected", None, Some(e.to_string()), hs, auth.clone()) + } + }, + ) } - self.completed_steps.push(step); - - let elapsed = self.step_start.elapsed(); - let elapsed_str = format_elapsed(elapsed); - - // Use a progress bar instead of println so we can clear it later. - let bar = self.mp.insert_before(&self.spinner, ProgressBar::new(0)); - bar.set_style( - ProgressStyle::with_template("{msg}").unwrap_or_else(|_| ProgressStyle::default_bar()), - ); - bar.set_message(format!( - "{} {} {}", - "\u{2713}".green().bold(), - label, - elapsed_str.dimmed() - )); - bar.finish(); - self.completed_bars.push(bar); - - // Reset step timer for the next step. - self.step_start = Instant::now(); - self.spinner.reset_elapsed(); - self.active_detail.clear(); - } - - /// Set the active (in-progress) step shown on the spinner. - fn set_active(&mut self, label: &str) { - self.active_label = label.to_string(); - self.active_detail.clear(); - // Reset the spinner's elapsed time for the new step. - self.spinner.reset_elapsed(); - self.step_start = Instant::now(); - self.update_spinner(); - } - - /// Set the active step from a known provisioning step enum. - fn set_active_step(&mut self, step: ProvisioningStep) { - self.set_active(step.active_label()); - } - - /// Set detail text shown alongside the active step (e.g. image name). - fn set_active_detail(&mut self, detail: &str) { - self.active_detail = detail.to_string(); - self.update_spinner(); - } - - fn update_spinner(&self) { - let msg = if self.active_detail.is_empty() { - self.active_label.clone() - } else { - format!("{} {}", self.active_label, self.active_detail.dimmed()) - }; - self.spinner.set_message(msg); - } + }; - /// Finish with an error message shown on the last step line. - fn finish_error(&self, msg: &str) { - let _ = self - .mp - .println(format!("{} {}", "\u{2717}".red().bold(), msg.red())); - self.spinner.finish_and_clear(); + let json_data = status_to_json( + gateway_name, + server, + is_bearer, + status_str, + &version, + &error, + &http_status, + &authentication, + ); + if crate::output::print_output_single(output, &json_data, Clone::clone)? { + return Ok(()); } - /// Print a line above the progress bars (for static header content). - fn println(&self, msg: &str) { - let _ = self.mp.println(msg); - } + // Human-readable output. + println!("{}", "Server Status".cyan().bold()); + println!(); + println!(" {} {}", "Gateway:".dimmed(), gateway_name); + println!(" {} {}", "Server:".dimmed(), server); - /// Clear all progress output (spinner, spacer, and completed step lines). - fn clear(&self) { - self.spacer.finish_and_clear(); - self.spinner.finish_and_clear(); - for bar in &self.completed_bars { - bar.finish_and_clear(); + match status_str { + "connected" => { + println!(" {} {}", "Status:".dimmed(), "Connected".green()); + print_gateway_authentication_state(&authentication); + if let Some(ref v) = version { + println!(" {} {}", "Version:".dimmed(), v); + } } - } -} - -/// Format a duration as a compact elapsed time string, e.g. `(3s)` or `(1m 12s)`. -fn format_elapsed(d: Duration) -> String { - let secs = d.as_secs(); - if secs < 60 { - format!("({secs}s)") - } else { - let mins = secs / 60; - let rem = secs % 60; - format!("({mins}m {rem}s)") - } -} - -/// Format a total elapsed time for non-interactive mode timestamps. -fn format_timestamp(d: Duration) -> String { - let secs = d.as_secs_f64(); - format!("[{secs:.1}s]") -} - -fn progress_step_from_metadata(value: &str) -> Option { - match value { - PROGRESS_STEP_REQUESTING_SANDBOX => Some(ProvisioningStep::RequestingSandbox), - PROGRESS_STEP_PULLING_IMAGE => Some(ProvisioningStep::PullingSandboxImage), - PROGRESS_STEP_STARTING_SANDBOX => Some(ProvisioningStep::StartingSandbox), - _ => None, - } -} - -fn noninteractive_active_label(step: ProvisioningStep) -> String { - step.active_label().trim_end_matches('.').to_string() -} - -fn handle_platform_progress_event( - event: &PlatformEvent, - display: &mut Option, - provision_start: Instant, -) -> bool { - let completed_step = event - .metadata - .get(PROGRESS_COMPLETE_STEP_KEY) - .and_then(|step| progress_step_from_metadata(step)); - let active_step = event - .metadata - .get(PROGRESS_ACTIVE_STEP_KEY) - .and_then(|step| progress_step_from_metadata(step)); - let active_detail = event - .metadata - .get(PROGRESS_ACTIVE_DETAIL_KEY) - .filter(|detail| !detail.is_empty()); - - let handled = completed_step.is_some() || active_step.is_some() || active_detail.is_some(); - if !handled { - return false; - } - - if let Some(step) = completed_step { - let label = event - .metadata - .get(PROGRESS_COMPLETE_LABEL_KEY) - .map_or_else(|| step.completed_label(), String::as_str); - if let Some(d) = display.as_mut() { - d.complete_step_with_label(step, label); - } else { - let ts = format_timestamp(provision_start.elapsed()); - println!("{} {}", ts.dimmed(), label); + "connected_http" => { + println!(" {} {}", "Status:".dimmed(), "Connected (HTTP)".yellow()); + if let Some(ref hs) = http_status { + println!(" {} {}", "HTTP: ".dimmed(), hs); + } + if let Some(ref e) = error { + println!(" {} {}", "gRPC error:".dimmed(), e); + } + print_gateway_authentication_state(&authentication); } - } - - if let Some(step) = active_step - && let Some(d) = display.as_mut() - { - d.set_active_step(step); - } - - if let Some(detail) = active_detail { - if let Some(d) = display.as_mut() { - d.set_active_detail(detail); - } else { - let ts = format_timestamp(provision_start.elapsed()); - if let Some(step) = active_step { - println!( - "{} {} {}", - ts.dimmed(), - noninteractive_active_label(step), - detail - ); - } else { - println!("{} {}", ts.dimmed(), detail); + "error" => { + println!(" {} {}", "Status:".dimmed(), "Error".red()); + if let Some(ref hs) = http_status { + println!(" {} {}", "HTTP:".dimmed(), hs); + if let Some(ref e) = error { + println!(" {} {}", "gRPC error:".dimmed(), e); + } + } else if let Some(ref e) = error { + println!(" {} {}", "Error:".dimmed(), e); } + print_gateway_authentication_state(&authentication); + } + _ => { + println!(" {} {}", "Status:".dimmed(), "Disconnected".red()); + if let Some(ref hs) = http_status { + println!(" {} {}", "HTTP:".dimmed(), hs); + } + if let Some(ref e) = error { + println!(" {} {}", "Error:".dimmed(), e); + } + print_gateway_authentication_state(&authentication); } } - true + Ok(()) } -fn is_provisioning_progress_event(event: &PlatformEvent) -> bool { - if event.metadata.contains_key(PROGRESS_COMPLETE_STEP_KEY) - || event.metadata.contains_key(PROGRESS_ACTIVE_STEP_KEY) - || event.metadata.contains_key(PROGRESS_ACTIVE_DETAIL_KEY) - { - return true; - } - - event.source == "vm" - && matches!( - event.reason.as_str(), - "PullingLayer" - | "ResolvingImage" - | "AuthenticatingRegistry" - | "FetchingManifest" - | "CacheHit" - | "CacheMiss" - | "WaitingForImageCacheLock" - | "ExportingRootfs" - | "PreparingRootfs" - | "CreatingRootDisk" - | "PreparingOverlay" - | "Started" - ) +#[derive(Debug, Clone, PartialEq, Eq)] +enum GatewayAuthenticationState { + Authenticated(&'static str), + NotRequired(&'static str), + Failed(String), + Unverified(String), } -fn print_sandbox_header(sandbox: &Sandbox, display: Option<&ProvisioningDisplay>) { - let lines = [ - String::new(), - format!( - "{} {}", - "Created sandbox:".cyan().bold(), - sandbox.object_name().bold() - ), - String::new(), - ]; - match display { - Some(d) => { - for line in lines { - d.println(&line); - } +fn gateway_authentication_state( + result: std::result::Result<(), Status>, + tls: &TlsOptions, + server: &str, +) -> GatewayAuthenticationState { + match result { + Ok(()) if tls.oidc_token.is_some() => GatewayAuthenticationState::Authenticated("OIDC"), + Ok(()) if tls.edge_token.is_some() => GatewayAuthenticationState::Authenticated("edge"), + Ok(()) if server.starts_with("https://") => { + GatewayAuthenticationState::Authenticated("mTLS transport") + } + Ok(()) => GatewayAuthenticationState::NotRequired("gateway"), + Err(status) if status.code() == Code::Unauthenticated => { + GatewayAuthenticationState::Failed(status.message().to_string()) + } + Err(status) if status.code() == Code::PermissionDenied => { + let provider = if tls.oidc_token.is_some() { + "OIDC; authorization denied" + } else if tls.edge_token.is_some() { + "edge; authorization denied" + } else if server.starts_with("https://") { + "mTLS transport; authorization denied" + } else { + "authorization denied" + }; + GatewayAuthenticationState::Authenticated(provider) } - None => { - for line in lines { - println!("{line}"); - } + Err(status) if status.code() == Code::Unimplemented && tls.edge_token.is_some() => { + GatewayAuthenticationState::Authenticated("edge") + } + Err(status) + if status.code() == Code::Unimplemented + && !tls.is_bearer_auth() + && server.starts_with("http://") => + { + GatewayAuthenticationState::NotRequired("gateway") } + Err(status) + if status.code() == Code::Unimplemented + && !tls.is_bearer_auth() + && server.starts_with("https://") => + { + GatewayAuthenticationState::Authenticated("mTLS transport") + } + Err(status) if status.code() == Code::Unimplemented => { + GatewayAuthenticationState::Unverified( + "gateway does not support authentication checks".to_string(), + ) + } + Err(status) => GatewayAuthenticationState::Unverified(status.message().to_string()), } } -#[derive(Debug, Clone)] -struct GatewayInfoView { - gateway: String, - server: String, - auth: Option<&'static str>, - status: String, - version: String, - compute_drivers: Vec, -} - -#[derive(Debug, Clone)] -struct ComputeDriverInfoView { - name: String, - capabilities: ComputeDriverCapabilitiesView, +fn print_gateway_authentication_state(state: &GatewayAuthenticationState) { + match state { + GatewayAuthenticationState::Authenticated(provider) => println!( + " {} {} ({provider})", + "Authentication:".dimmed(), + "Authenticated".green() + ), + GatewayAuthenticationState::NotRequired(mode) => println!( + " {} {} ({mode})", + "Authentication:".dimmed(), + "Not required".green() + ), + GatewayAuthenticationState::Failed(error) => println!( + " {} {} ({error})", + "Authentication:".dimmed(), + "Failed".red() + ), + GatewayAuthenticationState::Unverified(reason) => println!( + " {} {} ({reason})", + "Authentication:".dimmed(), + "Unverified".yellow() + ), + } } -#[derive(Debug, Clone)] -struct ComputeDriverCapabilitiesView { - driver_name: String, - driver_version: String, +fn authentication_state_to_json(state: &GatewayAuthenticationState) -> serde_json::Value { + match state { + GatewayAuthenticationState::Authenticated(provider) => serde_json::json!({ + "status": "authenticated", + "provider": provider, + }), + GatewayAuthenticationState::NotRequired(mode) => serde_json::json!({ + "status": "not_required", + "mode": mode, + }), + GatewayAuthenticationState::Failed(error) => serde_json::json!({ + "status": "failed", + "error": error, + }), + GatewayAuthenticationState::Unverified(reason) => serde_json::json!({ + "status": "unverified", + "reason": reason, + }), + } } -/// Show gateway status. -#[allow(clippy::branches_sharing_code)] -pub async fn gateway_status(gateway_name: &str, server: &str, tls: &TlsOptions) -> Result<()> { - println!("{}", "Server Status".cyan().bold()); - println!(); - println!(" {} {}", "Gateway:".dimmed(), gateway_name); - println!(" {} {}", "Server:".dimmed(), server); - if tls.is_bearer_auth() { - println!(" {} Edge (bearer token)", "Auth:".dimmed()); +#[allow(clippy::too_many_arguments)] +fn status_to_json( + gateway_name: &str, + server: &str, + is_bearer: bool, + status: &str, + version: &Option, + error: &Option, + http_status: &Option, + authentication: &GatewayAuthenticationState, +) -> serde_json::Value { + let mut obj = serde_json::json!({ + "gateway": gateway_name, + "server": server, + "status": status, + "authentication": authentication_state_to_json(authentication), + }); + let map = obj.as_object_mut().expect("json! returns object"); + if is_bearer { + map.insert("auth".into(), serde_json::json!("edge_bearer")); } - - // Try to connect and get health - match grpc_client(server, tls).await { - Ok(mut client) => match client.health(HealthRequest {}).await { - Ok(response) => { - let health = response.into_inner(); - println!(" {} {}", "Status:".dimmed(), "Connected".green()); - println!(" {} {}", "Version:".dimmed(), health.version); - } - Err(e) => { - if let Some(status) = http_health_check(server, tls).await? { - if status.is_success() { - println!(" {} {}", "Status:".dimmed(), "Connected (HTTP)".yellow()); - println!(" {} {}", "HTTP: ".dimmed(), status); - println!(" {} {}", "gRPC error:".dimmed(), e); - } else { - println!(" {} {}", "Status:".dimmed(), "Error".red()); - println!(" {} {}", "HTTP:".dimmed(), status); - println!(" {} {}", "gRPC error:".dimmed(), e); - } - } else { - println!(" {} {}", "Status:".dimmed(), "Error".red()); - println!(" {} {}", "Error:".dimmed(), e); - } - } - }, - Err(e) => { - if let Some(status) = http_health_check(server, tls).await? { - if status.is_success() { - println!(" {} {}", "Status:".dimmed(), "Connected (HTTP)".yellow()); - println!(" {} {}", "HTTP:".dimmed(), status); - println!(" {} {}", "gRPC error:".dimmed(), e); - } else { - println!(" {} {}", "Status:".dimmed(), "Disconnected".red()); - println!(" {} {}", "HTTP:".dimmed(), status); - println!(" {} {}", "Error:".dimmed(), e); - } - } else { - println!(" {} {}", "Status:".dimmed(), "Disconnected".red()); - println!(" {} {}", "Error:".dimmed(), e); - } - } + if let Some(v) = version { + map.insert("version".into(), serde_json::json!(v)); } - - Ok(()) + if let Some(hs) = http_status { + map.insert("http_status".into(), serde_json::json!(hs)); + } + if let Some(e) = error { + map.insert("error".into(), serde_json::json!(e)); + } + obj } fn gateway_service_status_name(status: i32) -> &'static str { @@ -1930,6 +1773,7 @@ pub struct SandboxCreateConfig<'a> { pub labels: HashMap, pub environment: HashMap, pub approval_mode: &'a str, + pub output: &'a str, } impl Default for SandboxCreateConfig<'_> { @@ -1953,6 +1797,7 @@ impl Default for SandboxCreateConfig<'_> { labels: HashMap::new(), environment: HashMap::new(), approval_mode: "manual", + output: "table", } } } @@ -1984,6 +1829,7 @@ pub async fn sandbox_create( labels, environment, approval_mode, + output, } = config; if editor.is_some() && !command.is_empty() { @@ -2133,23 +1979,36 @@ pub async fn sandbox_create( } } + let structured_output = output != "table"; + // Set up display — interactive terminals get a step-based checklist with - // spinners; non-interactive (pipes / CI) get timestamped lines. - let mut display = if interactive { - Some(ProvisioningDisplay::new()) + // spinners; non-interactive (pipes / CI) get timestamped lines; + // structured output suppresses all stdout progress. + let mut display = if structured_output { + ProgressOutput::Silent + } else if interactive { + ProgressOutput::Interactive(ProvisioningDisplay::new()) } else { - None + ProgressOutput::Plain }; - // Print header - print_sandbox_header(&sandbox, display.as_ref()); - - // Set initial active step on the spinner. - if let Some(d) = display.as_mut() { - d.set_active_step(ProvisioningStep::RequestingSandbox); + if structured_output { + eprintln!("Provisioning sandbox (structured output on stdout)..."); } else { - let ts = format_timestamp(Duration::ZERO); - println!(" {} Requesting compute...", ts.dimmed()); + // Print header + print_sandbox_header(&sandbox, display.as_interactive()); + + // Set initial active step on the spinner. + match &mut display { + ProgressOutput::Interactive(d) => { + d.set_active_step(ProvisioningStep::RequestingSandbox); + } + ProgressOutput::Plain => { + let ts = format_timestamp(Duration::ZERO); + println!(" {} Requesting compute...", ts.dimmed()); + } + ProgressOutput::Silent => {} + } } // Non-interactive mode: track start time for timestamps. @@ -2183,6 +2042,7 @@ pub async fn sandbox_create( .into_inner(); let mut last_phase = sandbox.phase(); + let mut last_sandbox = sandbox.clone(); let mut last_error_reason = String::new(); let mut last_condition_message = ready_false_condition_message(sandbox.status.as_ref()); // Track whether we have seen a non-Ready phase during the watch. @@ -2209,10 +2069,12 @@ pub async fn sandbox_create( resource_requirements.as_ref(), last_condition_message.as_deref(), ); - if let Some(d) = display.as_mut() { + if let Some(d) = display.as_interactive_mut() { d.finish_error(&timeout_message); } - println!(); + if display.is_plain() { + println!(); + } return Err(miette::miette!(timeout_message)); } @@ -2228,10 +2090,12 @@ pub async fn sandbox_create( resource_requirements.as_ref(), last_condition_message.as_deref(), ); - if let Some(d) = display.as_mut() { + if let Some(d) = display.as_interactive_mut() { d.finish_error(&timeout_message); } - println!(); + if display.is_plain() { + println!(); + } return Err(miette::miette!(timeout_message)); } }; @@ -2241,6 +2105,7 @@ pub async fn sandbox_create( Some(openshell_core::proto::sandbox_stream_event::Payload::Sandbox(s)) => { let phase = SandboxPhase::try_from(s.phase()).unwrap_or(SandboxPhase::Unknown); last_phase = s.phase(); + last_sandbox = s.clone(); if let Some(message) = ready_false_condition_message(s.status.as_ref()) { last_condition_message = Some(message); } @@ -2268,7 +2133,7 @@ pub async fn sandbox_create( // Only accept Ready as terminal after we've observed a // non-Ready phase, proving the controller has reconciled. if saw_non_ready && phase == SandboxPhase::Ready { - if let Some(d) = display.as_mut() { + if let Some(d) = display.as_interactive_mut() { d.clear(); } break; @@ -2282,7 +2147,24 @@ pub async fn sandbox_create( } Some(openshell_core::proto::sandbox_stream_event::Payload::Event(ev)) => { let extends_timeout = is_provisioning_progress_event(&ev); - if handle_platform_progress_event(&ev, &mut display, provision_start) { + // Silent mode suppresses all progress output; only update + // the deadline when applicable. + let handled = match &mut display { + ProgressOutput::Interactive(d) => { + let mut opt = Some(std::mem::replace(d, ProvisioningDisplay::new())); + let h = handle_platform_progress_event(&ev, &mut opt, provision_start); + if let Some(inner) = opt { + *d = inner; + } + h + } + ProgressOutput::Plain => { + let mut opt: Option = None; + handle_platform_progress_event(&ev, &mut opt, provision_start) + } + ProgressOutput::Silent => false, + }; + if handled { if extends_timeout { provisioning_idle_deadline = Instant::now() + provision_timeout; } @@ -2292,19 +2174,21 @@ pub async fn sandbox_create( provisioning_idle_deadline = Instant::now() + provision_timeout; } - if let Some(d) = display.as_mut() { - // Unknown events: show as detail on the current spinner. - if !ev.message.is_empty() { - d.set_active_detail(&ev.message); - } + if let Some(d) = display.as_interactive_mut() + && !ev.message.is_empty() + { + d.set_active_detail(&ev.message); } } Some(openshell_core::proto::sandbox_stream_event::Payload::Warning(w)) => { - if let Some(d) = display.as_mut() { - d.println(&format!(" {} {}", "!".yellow().bold(), w.message.yellow())); - } else { - let ts = format_timestamp(provision_start.elapsed()); - eprintln!(" {} {} {}", ts.dimmed(), "WARN".yellow(), w.message); + match &display { + ProgressOutput::Interactive(d) => { + d.println(&format!(" {} {}", "!".yellow().bold(), w.message.yellow())); + } + ProgressOutput::Plain | ProgressOutput::Silent => { + let ts = format_timestamp(provision_start.elapsed()); + eprintln!(" {} {} {}", ts.dimmed(), "WARN".yellow(), w.message); + } } } Some(openshell_core::proto::sandbox_stream_event::Payload::DraftPolicyUpdate(_)) @@ -2317,7 +2201,7 @@ pub async fn sandbox_create( // If we exited the loop without hitting the Ready break, finish the display. let final_phase = SandboxPhase::try_from(last_phase).unwrap_or(SandboxPhase::Unknown); if final_phase != SandboxPhase::Ready - && let Some(d) = display.as_mut() + && let Some(d) = display.as_interactive_mut() { if final_phase == SandboxPhase::Error { let msg = if last_error_reason.is_empty() { @@ -2427,6 +2311,11 @@ pub async fn sandbox_create( ); } + if structured_output { + crate::output::print_output_single(output, &last_sandbox, sandbox_to_json)?; + return Ok(()); + } + if let Some(editor) = editor { let ssh_gateway_name = effective_tls.gateway_name().unwrap_or(gateway_name); sandbox_connect_editor( @@ -2765,6 +2654,7 @@ pub async fn sandbox_get( server: &str, name: &str, policy_only: bool, + output: &str, workspace: &str, tls: &TlsOptions, ) -> Result<()> { @@ -2806,6 +2696,11 @@ pub async fn sandbox_get( return Ok(()); } + let detail_json = sandbox_detail_to_json(&sandbox, &config)?; + if crate::output::print_output_single(output, &detail_json, Clone::clone)? { + return Ok(()); + } + println!("{}", "Sandbox:".cyan().bold()); println!(); let id = if sandbox.object_id().is_empty() { @@ -3450,63 +3345,6 @@ async fn sandbox_exec_interactive_grpc( Ok(exit_code) } -/// Print a single YAML line with dimmed keys and regular values. -fn print_yaml_line(line: &str) { - // Find leading whitespace - let trimmed = line.trim_start(); - let indent = &line[..line.len() - trimmed.len()]; - - // Handle list items - if let Some(rest) = trimmed.strip_prefix("- ") { - print!("{indent}"); - print!("{}", "- ".dimmed()); - print!("{rest}"); - println!(); - return; - } - - // Handle key: value pairs - if let Some(colon_pos) = trimmed.find(':') { - let key = &trimmed[..colon_pos]; - let after_colon = &trimmed[colon_pos + 1..]; - - print!("{indent}"); - print!("{}", key.dimmed()); - print!("{}", ":".dimmed()); - - if after_colon.is_empty() { - // Key with nested content (no value on this line) - } else if let Some(value) = after_colon.strip_prefix(' ') { - // Key: value - print!(" {value}"); - } else { - // Shouldn't happen in valid YAML, but handle it - print!("{after_colon}"); - } - println!(); - return; - } - - // Plain line (shouldn't happen often in YAML) - println!("{line}"); -} - -/// Print sandbox policy as YAML with dimmed keys. -fn print_sandbox_policy(policy: &SandboxPolicy) { - println!("{}", "Policy:".cyan().bold()); - println!(); - if let Ok(yaml_str) = openshell_policy::serialize_sandbox_policy(policy) { - // Indent the YAML output and skip the initial "---" line - for line in yaml_str.lines() { - if line == "---" { - continue; - } - print!(" "); - print_yaml_line(line); - } - } -} - /// List sandboxes. #[allow(clippy::too_many_arguments)] pub async fn sandbox_list( @@ -3658,6 +3496,48 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value { }) } +fn sandbox_detail_to_json( + sandbox: &Sandbox, + config: &GetSandboxConfigResponse, +) -> Result { + let mut value = sandbox_to_json(sandbox); + let obj = value + .as_object_mut() + .expect("sandbox_to_json returns object"); + + let policy_source = if config.policy_source == PolicySource::Global as i32 { + "global" + } else { + "sandbox" + }; + obj.insert("policy_source".into(), serde_json::json!(policy_source)); + + let policy_from_global = config.policy_source == PolicySource::Global as i32; + let revision = if policy_from_global { + if config.global_policy_version > 0 { + Some(config.global_policy_version) + } else if config.version > 0 { + Some(config.version) + } else { + None + } + } else if config.version > 0 { + Some(config.version) + } else { + None + }; + obj.insert("revision".into(), serde_json::json!(revision)); + + let policy_json = match config.policy.as_ref() { + Some(p) => openshell_policy::sandbox_policy_to_json_value(p) + .wrap_err("failed to convert policy to JSON")?, + None => serde_json::Value::Null, + }; + obj.insert("policy".into(), policy_json); + + Ok(value) +} + pub async fn sandbox_provider_list( server: &str, name: &str, @@ -4137,285 +4017,95 @@ async fn auto_create_provider( r#type: provider_type.to_string(), credentials: discovered.credentials.clone(), config: discovered.config.clone(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: workspace.to_string(), - }), - workspace: workspace.to_string(), - }; - - let response = client.create_provider(request).await.map_err(|status| { - miette::miette!("failed to create provider '{exact_name}': {status}") - })?; - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - eprintln!( - "{} Created provider {} ({}) from existing local state", - "✓".green().bold(), - provider.object_name(), - provider.r#type - ); - if seen_names.insert(provider.object_name().to_string()) { - configured_names.push(provider.object_name().to_string()); - } - } else { - // Inferred type: try type as name, then suffixed variants. - let mut created = false; - for attempt in 0..5 { - let name = if attempt == 0 { - provider_type.to_string() - } else { - format!("{provider_type}-{attempt}") - }; - - let request = CreateProviderRequest { - provider: Some(Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: name.clone(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.to_string(), - credentials: discovered.credentials.clone(), - config: discovered.config.clone(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: workspace.to_string(), - }), - workspace: workspace.to_string(), - }; - - match client.create_provider(request).await { - Ok(response) => { - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - eprintln!( - "{} Created provider {} ({}) from existing local state", - "✓".green().bold(), - provider.object_name(), - provider.r#type - ); - if seen_names.insert(provider.object_name().to_string()) { - configured_names.push(provider.object_name().to_string()); - } - created = true; - break; - } - Err(status) if status.code() == Code::AlreadyExists => {} - Err(status) => { - return Err(miette::miette!( - "failed to create provider for type '{provider_type}': {status}" - )); - } - } - } - - if !created { - return Err(miette::miette!( - "failed to create provider for type '{provider_type}' after name retries" - )); - } - } - - eprintln!(); - Ok(()) -} - -pub fn parse_key_value_pairs(items: &[String], flag: &str) -> Result> { - let mut map = HashMap::new(); - - for item in items { - let Some((key, value)) = item.split_once('=') else { - return Err(miette::miette!("{flag} expects KEY=VALUE, got '{item}'")); - }; - - let key = key.trim(); - if key.is_empty() { - return Err(miette::miette!("{flag} key cannot be empty")); - } - - map.insert(key.to_string(), value.to_string()); - } - - Ok(map) -} - -pub fn parse_env_pairs(items: &[String]) -> Result> { - let map = parse_key_value_pairs(items, "--env")?; - for key in map.keys() { - if !is_valid_env_name(key) { - return Err(miette::miette!( - "--env key must match [A-Za-z_][A-Za-z0-9_]*; got '{key}'" - )); - } - if key.starts_with("OPENSHELL_") { - return Err(miette::miette!( - "--env keys starting with OPENSHELL_ are reserved; got '{key}'" - )); - } - } - Ok(map) -} - -/// Resolve `--secret-material-env KEY[=ENVVAR]` values from the CLI process -/// environment (`ENVVAR` defaults to `KEY`) so secrets never transit argv. -pub fn parse_secret_material_env_pairs(items: &[String]) -> Result> { - let mut map = HashMap::new(); - - for item in items { - let (key, env_name) = match item.split_once('=') { - Some((key, env_name)) => (key.trim(), env_name.trim()), - None => (item.trim(), item.trim()), - }; - if key.is_empty() { - return Err(miette::miette!("--secret-material-env key cannot be empty")); - } - if env_name.is_empty() { - return Err(miette::miette!( - "--secret-material-env {key} names an empty environment variable" - )); - } - - let value = std::env::var(env_name).map_err(|_| { - miette::miette!( - "--secret-material-env {key} requires local env var '{env_name}' to be set to a non-empty value" - ) - })?; - if value.trim().is_empty() { - return Err(miette::miette!( - "--secret-material-env {key} requires local env var '{env_name}' to be set to a non-empty value" - )); - } - - if map.contains_key(key) { - return Err(miette::miette!( - "--secret-material-env key '{key}' supplied more than once" - )); - } - map.insert(key.to_string(), value); - } - - Ok(map) -} - -fn is_valid_env_name(key: &str) -> bool { - let mut bytes = key.bytes(); - let Some(first) = bytes.next() else { - return false; - }; - if !(first == b'_' || first.is_ascii_alphabetic()) { - return false; - } - bytes.all(|b| b == b'_' || b.is_ascii_alphanumeric()) -} - -fn parse_credential_pairs(items: &[String]) -> Result> { - let mut map = HashMap::new(); - - for item in items { - if let Some((key, value)) = item.split_once('=') { - let key = key.trim(); - if key.is_empty() { - return Err(miette::miette!("--credential key cannot be empty")); - } - map.insert(key.to_string(), value.to_string()); - continue; - } - - let key = item.trim(); - if key.is_empty() { - return Err(miette::miette!("--credential key cannot be empty")); - } - - let value = std::env::var(key).map_err(|_| { - miette::miette!( - "--credential {key} requires local env var '{key}' to be set to a non-empty value" - ) - })?; - - if value.trim().is_empty() { - return Err(miette::miette!( - "--credential {key} requires local env var '{key}' to be set to a non-empty value" - )); - } - - map.insert(key.to_string(), value); - } - - Ok(map) -} - -pub fn parse_credential_expiry_cli_value(value: &str) -> std::result::Result { - parse_credential_expiry_value(value, None).map_err(|err| err.to_string()) -} - -fn credential_expiry_value_error(key: Option<&str>, detail: &str) -> miette::Report { - key.map_or_else( - || miette::miette!("--credential-expires-at value {detail}"), - |key| miette::miette!("--credential-expires-at value for '{key}' {detail}"), - ) -} - -fn parse_credential_expiry_value(value: &str, key: Option<&str>) -> Result { - let value = value.trim(); - if value.is_empty() { - return Err(credential_expiry_value_error(key, "cannot be empty")); - } + credential_expires_at_ms: HashMap::new(), + profile_workspace: workspace.to_string(), + }), + workspace: workspace.to_string(), + }; - if let Ok(value_ms) = value.parse::() { - if value_ms < 0 { - return Err(credential_expiry_value_error( - key, - "must be greater than or equal to 0", - )); + let response = client.create_provider(request).await.map_err(|status| { + miette::miette!("failed to create provider '{exact_name}': {status}") + })?; + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + eprintln!( + "{} Created provider {} ({}) from existing local state", + "✓".green().bold(), + provider.object_name(), + provider.r#type + ); + if seen_names.insert(provider.object_name().to_string()) { + configured_names.push(provider.object_name().to_string()); } - return Ok(value_ms); - } - - let parsed = DateTime::parse_from_rfc3339(value).map_err(|_| { - credential_expiry_value_error( - key, - "must be a Unix epoch millisecond timestamp or RFC3339 timestamp", - ) - })?; - let value_ms = parsed.timestamp_millis(); - if value_ms < 0 { - return Err(credential_expiry_value_error( - key, - "must be greater than or equal to 0", - )); - } + } else { + // Inferred type: try type as name, then suffixed variants. + let mut created = false; + for attempt in 0..5 { + let name = if attempt == 0 { + provider_type.to_string() + } else { + format!("{provider_type}-{attempt}") + }; - Ok(value_ms) -} + let request = CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.clone(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + r#type: provider_type.to_string(), + credentials: discovered.credentials.clone(), + config: discovered.config.clone(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: workspace.to_string(), + }), + workspace: workspace.to_string(), + }; -fn parse_credential_expiry_pairs(items: &[String]) -> Result> { - let mut map = HashMap::new(); + match client.create_provider(request).await { + Ok(response) => { + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + eprintln!( + "{} Created provider {} ({}) from existing local state", + "✓".green().bold(), + provider.object_name(), + provider.r#type + ); + if seen_names.insert(provider.object_name().to_string()) { + configured_names.push(provider.object_name().to_string()); + } + created = true; + break; + } + Err(status) if status.code() == Code::AlreadyExists => {} + Err(status) => { + return Err(miette::miette!( + "failed to create provider for type '{provider_type}': {status}" + )); + } + } + } - for item in items { - let Some((key, value)) = item.split_once('=') else { - return Err(miette::miette!( - "--credential-expires-at expects KEY=TIMESTAMP, got '{item}'" - )); - }; - let key = key.trim(); - if key.is_empty() { + if !created { return Err(miette::miette!( - "--credential-expires-at key cannot be empty" + "failed to create provider for type '{provider_type}' after name retries" )); } - let value = parse_credential_expiry_value(value, Some(key))?; - map.insert(key.to_string(), value); } - Ok(map) + eprintln!(); + Ok(()) } pub async fn service_expose( @@ -5903,27 +5593,6 @@ fn refresh_status_row(status: &ProviderCredentialRefreshStatus) -> String { ) } -fn format_optional_epoch_ms(ms: i64) -> String { - if ms > 0 { - format_epoch_ms(ms) - } else { - "-".to_string() - } -} - -fn truncate_status_field(value: &str, max_chars: usize) -> String { - if value.is_empty() { - return "-".to_string(); - } - let mut chars = value.chars(); - let truncated = chars.by_ref().take(max_chars).collect::(); - if chars.next().is_some() { - format!("{truncated}...") - } else { - truncated - } -} - fn provider_refresh_strategy_name(strategy: ProviderCredentialRefreshStrategy) -> &'static str { match strategy { ProviderCredentialRefreshStrategy::Static => "static", @@ -6006,6 +5675,17 @@ fn load_profile_import_item( } .map_err(|err| profile_file_diagnostic(&source, err.to_string()))?; + let pre_lower = profile.validate_before_lowering(&source); + if let Some(diag) = pre_lower.into_iter().find(|d| d.severity == "error") { + return Err(ProviderProfileDiagnostic { + source: diag.source, + profile_id: diag.profile_id, + field: diag.field, + message: diag.message, + severity: diag.severity, + }); + } + Ok(ProviderProfileImportItem { profile: Some(profile.to_proto()), source, @@ -6161,17 +5841,6 @@ fn print_provider_type_row( ); } -fn truncate_display(value: &str, max_width: usize) -> String { - if value.chars().count() <= max_width { - return value.to_string(); - } - - let keep = max_width.saturating_sub(3); - let mut truncated = value.chars().take(keep).collect::(); - truncated.push_str("..."); - truncated -} - #[allow(clippy::too_many_arguments)] pub async fn provider_update( server: &str, @@ -7119,174 +6788,10 @@ pub async fn sandbox_upload( Ok(()) } -fn scrub_git_env(command: &mut Command) -> &mut Command { - for key in [ - "GIT_DIR", - "GIT_WORK_TREE", - "GIT_INDEX_FILE", - "GIT_PREFIX", - "GIT_COMMON_DIR", - "GIT_OBJECT_DIRECTORY", - "GIT_ALTERNATE_OBJECT_DIRECTORIES", - ] { - command.env_remove(key); - } - command -} - // --------------------------------------------------------------------------- // Sandbox policy commands // --------------------------------------------------------------------------- -/// Parse a duration string like "5m", "1h", "30s" into milliseconds. -fn parse_duration_to_ms(s: &str) -> Result { - let s = s.trim(); - if s.is_empty() { - return Err(miette::miette!("empty duration string")); - } - let (num_str, unit) = s.split_at(s.len() - 1); - let num: i64 = num_str - .parse() - .map_err(|_| miette::miette!("invalid duration: {s} (expected e.g. 5m, 1h, 30s)"))?; - let multiplier = match unit { - "s" => 1_000, - "m" => 60_000, - "h" => 3_600_000, - _ => { - return Err(miette::miette!( - "unknown duration unit: {unit} (use s, m, or h)" - )); - } - }; - Ok(num * multiplier) -} - -fn confirm_global_setting_takeover(key: &str, yes: bool) -> Result<()> { - if yes { - return Ok(()); - } - - if !std::io::stdin().is_terminal() || !std::io::stdout().is_terminal() { - return Err(miette::miette!( - "global setting updates require confirmation; pass --yes in non-interactive mode" - )); - } - - let proceed = Confirm::with_theme(&ColorfulTheme::default()) - .with_prompt(format!( - "Setting '{key}' globally will disable sandbox-level management for this key. Continue?" - )) - .default(false) - .interact() - .into_diagnostic()?; - - if !proceed { - return Err(miette::miette!("aborted by user")); - } - - Ok(()) -} - -fn confirm_global_setting_delete(key: &str, yes: bool) -> Result<()> { - if yes { - return Ok(()); - } - - if !std::io::stdin().is_terminal() || !std::io::stdout().is_terminal() { - return Err(miette::miette!( - "global setting deletes require confirmation; pass --yes in non-interactive mode" - )); - } - - let proceed = Confirm::with_theme(&ColorfulTheme::default()) - .with_prompt(format!( - "Deleting global setting '{key}' re-enables sandbox-level management for this key. Continue?" - )) - .default(false) - .interact() - .into_diagnostic()?; - - if !proceed { - return Err(miette::miette!("aborted by user")); - } - - Ok(()) -} - -fn parse_cli_setting_value(key: &str, raw_value: &str) -> Result { - let setting = settings::setting_for_key(key).ok_or_else(|| { - miette::miette!( - "unknown setting key '{}'. Allowed keys: {}", - key, - settings::registered_keys_csv() - ) - })?; - - let value = match setting.kind { - SettingValueKind::String => { - // Reject typos client-side so `openshell settings set ... - // proposal_approval_mode autom` errors immediately instead of - // round-tripping through the server. The server enforces the - // same check independently for non-CLI callers. - setting - .validate_string_value(raw_value) - .map_err(|allowed| { - miette::miette!( - "invalid value '{}' for key '{}'; expected one of: {}", - raw_value, - key, - allowed.join(", ") - ) - })?; - setting_value::Value::StringValue(raw_value.to_string()) - } - SettingValueKind::Int => { - let parsed = raw_value.trim().parse::().map_err(|_| { - miette::miette!( - "invalid int value '{}' for key '{}'; expected base-10 integer", - raw_value, - key - ) - })?; - setting_value::Value::IntValue(parsed) - } - SettingValueKind::Bool => { - let parsed = settings::parse_bool_like(raw_value).ok_or_else(|| { - miette::miette!( - "invalid bool value '{}' for key '{}'; expected one of: true,false,yes,no,1,0", - raw_value, - key - ) - })?; - setting_value::Value::BoolValue(parsed) - } - }; - - Ok(SettingValue { value: Some(value) }) -} - -fn format_setting_value(value: Option<&SettingValue>) -> String { - let Some(value) = value.and_then(|v| v.value.as_ref()) else { - return "".to_string(); - }; - match value { - setting_value::Value::StringValue(v) => v.clone(), - setting_value::Value::BoolValue(v) => v.to_string(), - setting_value::Value::IntValue(v) => v.to_string(), - setting_value::Value::BytesValue(v) => format!("", v.len()), - } -} - -fn short_hash(hash: &str) -> &str { - if hash.len() >= 12 { &hash[..12] } else { hash } -} - -fn print_policy_merge_warnings(warnings: &[openshell_policy::PolicyMergeWarning]) { - for warning in warnings { - eprintln!("{} {}", "!".yellow().bold(), warning); - } -} - pub async fn sandbox_policy_set_global( server: &str, policy_path: &str, @@ -7440,7 +6945,7 @@ pub async fn gateway_settings_get(server: &str, json: bool, tls: &TlsOptions) -> fn settings_to_json_sandbox( name: &str, workspace: &str, - response: &openshell_core::proto::GetSandboxConfigResponse, + response: &GetSandboxConfigResponse, ) -> serde_json::Value { let policy_source = if response.policy_source == PolicySource::Global as i32 { "global" @@ -8859,46 +8364,28 @@ fn format_endpoint(endpoint: &openshell_core::proto::NetworkEndpoint) -> String format!("{host_port} [{}]", tags.join(", ")) } -fn non_empty_or<'a>(value: &'a str, fallback: &'a str) -> &'a str { - if value.is_empty() { fallback } else { value } -} - -/// Format a millisecond timestamp into a readable string. -fn format_timestamp_ms(ms: i64) -> String { - if ms <= 0 { - return "-".to_string(); - } - let secs = ms / 1000; - let mins = (secs / 60) % 60; - let hours = (secs / 3600) % 24; - let days = secs / 86400; - if days > 0 { - format!("{days}d {hours:02}:{mins:02}") - } else { - format!("{hours:02}:{mins:02}") - } -} - #[cfg(test)] mod tests { use super::{ - ComputeDriverCapabilitiesView, ComputeDriverInfoView, GatewayInfoView, PolicyGetView, - ProvisioningStep, TlsOptions, build_sandbox_resource_limits, - dockerfile_sources_supported_for_gateway, format_endpoint, format_gateway_select_header, - format_gateway_select_items, format_provider_attachment_table, gateway_add, - gateway_auth_label, gateway_env_override_warning, gateway_info_to_json, + ComputeDriverCapabilitiesView, ComputeDriverInfoView, GatewayAuthenticationState, + GatewayInfoView, PolicyGetView, ProvisioningStep, TlsOptions, + build_sandbox_resource_limits, dockerfile_sources_supported_for_gateway, format_endpoint, + format_gateway_select_header, format_gateway_select_items, + format_provider_attachment_table, gateway_add, gateway_auth_label, + gateway_authentication_state, gateway_env_override_warning, gateway_info_to_json, gateway_remote_label, gateway_select_with, gateway_to_json, gateway_type_label, git_sync_files, http_health_check, import_local_package_mtls_bundle, inferred_provider_type, mtls_certs_exist_for_gateway, package_managed_tls_dirs, parse_cli_setting_value, parse_credential_expiry_cli_value, parse_credential_expiry_pairs, parse_credential_pairs, parse_driver_config_json, parse_secret_material_env_pairs, - plaintext_gateway_is_remote, policy_revision_to_json, progress_step_from_metadata, + plaintext_gateway_is_remote, policy_revision_to_json, provider_profile_allows_empty_credentials, provisioning_timeout_message, ready_false_condition_message, refresh_status_header, refresh_status_row, resolve_from, sandbox_should_persist, sandbox_upload_plan, service_expose_status_error, service_url_for_gateway, }; use crate::TEST_ENV_LOCK; + use crate::commands::common::progress_step_from_metadata; use hyper::StatusCode; use std::fs; use std::io::{Read, Write}; @@ -8917,13 +8404,84 @@ mod tests { PROGRESS_STEP_STARTING_SANDBOX, }; use openshell_core::proto::{ - GpuResourceRequirements, PolicyStatus, Provider, ProviderCredentialRefresh, - ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy, - ProviderCredentialTokenGrant, ProviderProfile, ProviderProfileCredential, - ResourceRequirements, SandboxCondition, SandboxPolicyRevision, SandboxStatus, - datamodel::v1::ObjectMeta, + GetSandboxConfigResponse, GpuResourceRequirements, PolicySource, PolicyStatus, Provider, + ProviderCredentialRefresh, ProviderCredentialRefreshStatus, + ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrant, ProviderProfile, + ProviderProfileCredential, ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase, + SandboxPolicyRevision, SandboxStatus, datamodel::v1::ObjectMeta, }; + #[test] + fn gateway_status_reports_authenticated_oidc_probe() { + let mut tls = TlsOptions::default(); + tls.oidc_token = Some("token".to_string()); + + let state = gateway_authentication_state(Ok(()), &tls, "https://gateway.example.com"); + + assert_eq!(state, GatewayAuthenticationState::Authenticated("OIDC")); + } + + #[test] + fn gateway_status_treats_authorization_denial_as_authenticated() { + let mut tls = TlsOptions::default(); + tls.oidc_token = Some("token".to_string()); + + let state = gateway_authentication_state( + Err(Status::permission_denied("role 'openshell-admin' required")), + &tls, + "https://gateway.example.com", + ); + + assert_eq!( + state, + GatewayAuthenticationState::Authenticated("OIDC; authorization denied") + ); + } + + #[test] + fn gateway_status_reports_rejected_bearer_token() { + let mut tls = TlsOptions::default(); + tls.oidc_token = Some("expired".to_string()); + + let state = gateway_authentication_state( + Err(Status::unauthenticated("invalid token: ExpiredSignature")), + &tls, + "https://gateway.example.com", + ); + + assert_eq!( + state, + GatewayAuthenticationState::Failed("invalid token: ExpiredSignature".to_string()) + ); + } + + #[test] + fn gateway_status_marks_oidc_unverified_on_older_gateway() { + let mut tls = TlsOptions::default(); + tls.oidc_token = Some("token".to_string()); + + let state = gateway_authentication_state( + Err(Status::unimplemented("unknown service")), + &tls, + "https://gateway.example.com", + ); + + assert_eq!( + state, + GatewayAuthenticationState::Unverified( + "gateway does not support authentication checks".to_string() + ) + ); + } + + #[test] + fn gateway_status_reports_local_plaintext_auth_not_required() { + let state = + gateway_authentication_state(Ok(()), &TlsOptions::default(), "http://127.0.0.1:8080"); + + assert_eq!(state, GatewayAuthenticationState::NotRequired("gateway")); + } + #[test] fn policy_revision_json_includes_revision_provenance() { let revision = SandboxPolicyRevision { @@ -10950,4 +10508,124 @@ mod tests { "raw milliseconds field should not exist" ); } + + #[test] + fn sandbox_detail_to_json_includes_policy_fields() { + let mut sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: "sb-123".to_string(), + name: "test-sb".to_string(), + resource_version: 5, + created_at_ms: 1_609_459_200_000, + ..Default::default() + }), + ..Default::default() + }; + sandbox.set_phase(SandboxPhase::Ready as i32); + sandbox.set_current_policy_version(2); + + let config = GetSandboxConfigResponse { + policy_source: PolicySource::Global as i32, + global_policy_version: 3, + ..Default::default() + }; + + let json = super::sandbox_detail_to_json(&sandbox, &config).unwrap(); + + assert_eq!(json["id"], "sb-123"); + assert_eq!(json["name"], "test-sb"); + assert_eq!(json["phase"], "Ready"); + assert_eq!(json["policy_source"], "global"); + assert_eq!(json["revision"], 3); + assert!(json["policy"].is_null()); + } + + #[test] + fn sandbox_detail_to_json_sandbox_source_without_policy() { + let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: "sb-456".to_string(), + name: "no-policy-sb".to_string(), + ..Default::default() + }), + ..Default::default() + }; + let config = GetSandboxConfigResponse { + policy_source: PolicySource::Sandbox as i32, + version: 0, + ..Default::default() + }; + + let json = super::sandbox_detail_to_json(&sandbox, &config).unwrap(); + + assert_eq!(json["policy_source"], "sandbox"); + assert!(json["revision"].is_null()); + assert!(json["policy"].is_null()); + } + + #[test] + fn status_to_json_connected() { + let auth = GatewayAuthenticationState::Authenticated("mTLS transport"); + let json = super::status_to_json( + "my-gw", + "http://127.0.0.1:8090", + false, + "connected", + &Some("1.2.3".to_string()), + &None, + &None, + &auth, + ); + + assert_eq!(json["gateway"], "my-gw"); + assert_eq!(json["server"], "http://127.0.0.1:8090"); + assert_eq!(json["status"], "connected"); + assert_eq!(json["version"], "1.2.3"); + assert_eq!(json["authentication"]["status"], "authenticated"); + assert!(json.get("auth").is_none()); + assert!(json.get("error").is_none()); + assert!(json.get("http_status").is_none()); + } + + #[test] + fn status_to_json_disconnected_with_error() { + let auth = GatewayAuthenticationState::Unverified("gateway unreachable".to_string()); + let json = super::status_to_json( + "broken-gw", + "http://10.0.0.1:8090", + false, + "disconnected", + &None, + &Some("connection refused".to_string()), + &None, + &auth, + ); + + assert_eq!(json["status"], "disconnected"); + assert_eq!(json["error"], "connection refused"); + assert_eq!(json["authentication"]["status"], "unverified"); + assert!(json.get("version").is_none()); + } + + #[test] + fn status_to_json_connected_http_with_bearer() { + let auth = GatewayAuthenticationState::Failed("token expired".to_string()); + let json = super::status_to_json( + "edge-gw", + "https://edge.example.com", + true, + "connected_http", + &None, + &Some("gRPC unavailable".to_string()), + &Some("200 OK".to_string()), + &auth, + ); + + assert_eq!(json["status"], "connected_http"); + assert_eq!(json["auth"], "edge_bearer"); + assert_eq!(json["error"], "gRPC unavailable"); + assert_eq!(json["http_status"], "200 OK"); + assert_eq!(json["authentication"]["status"], "failed"); + assert!(json.get("version").is_none()); + } } diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index 50bc2bbee9..f8de99a692 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -18,6 +18,7 @@ use openshell_core::proto::{ }; use owo_colors::OwoColorize; use std::fs; +use std::future::Future; use std::io::{IsTerminal, Write}; #[cfg(unix)] use std::os::unix::process::CommandExt; @@ -447,9 +448,24 @@ async fn wait_for_forward_start(child: &mut Child, spec: &ForwardSpec) -> Result /// last probe error is folded into the timeout diagnostic, so a failure reports /// why the listener never opened, not just that it timed out. async fn wait_for_forward_listener(spec: &ForwardSpec, wait_for: Duration) -> Result<()> { + wait_for_forward_listener_with_probe(spec, wait_for, |spec| async move { + probe_forward_listener(&spec).await + }) + .await +} + +async fn wait_for_forward_listener_with_probe( + spec: &ForwardSpec, + wait_for: Duration, + mut probe: F, +) -> Result<()> +where + F: FnMut(ForwardSpec) -> Fut, + Fut: Future>, +{ let deadline = tokio::time::Instant::now() + wait_for; loop { - let probe_error = match probe_forward_listener(spec).await { + let probe_error = match probe(spec.clone()).await { Ok(()) => return Ok(()), Err(err) => err, }; @@ -786,7 +802,7 @@ async fn ssh_tar_upload( "mkdir -p {escaped_dest} && cat | tar xf - -C {escaped_dest}", )) .stdin(Stdio::piped()) - .stdout(Stdio::inherit()) + .stdout(Stdio::null()) .stderr(Stdio::inherit()); let mut child = ssh.spawn().into_diagnostic()?; @@ -1816,17 +1832,17 @@ mod tests { } #[tokio::test] - async fn wait_for_forward_listener_rejects_missing_listener() { - let listener = std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap(); - let port = listener.local_addr().unwrap().port(); - drop(listener); - let spec = ForwardSpec::new(port); + async fn wait_for_forward_listener_reports_failed_probe() { + let spec = ForwardSpec::new(12345); - let err = wait_for_forward_listener(&spec, Duration::from_millis(20)) - .await - .unwrap_err(); + let err = wait_for_forward_listener_with_probe(&spec, Duration::ZERO, |_| { + std::future::ready(Err("connection refused".to_string())) + }) + .await + .unwrap_err(); let text = format!("{err:?}"); assert!(text.contains("local forward listener did not open")); + assert!(text.contains("connection refused")); } #[test] diff --git a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs index fda3d529f3..f42880d9f6 100644 --- a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs +++ b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs @@ -726,7 +726,7 @@ struct TestServer { endpoint: String, tls: TlsOptions, openshell: TestOpenShell, - _dir: TempDir, + dir: TempDir, } async fn run_server() -> TestServer { @@ -775,7 +775,7 @@ async fn run_server() -> TestServer { endpoint, tls, openshell, - _dir: dir, + dir, } } @@ -1908,3 +1908,77 @@ async fn sandbox_create_env_rejects_invalid_key_name() { "error should mention invalid key, got: {msg}" ); } + +async fn run_cli_sandbox_create( + server: &TestServer, + name: &str, + extra_args: &[&str], +) -> std::process::Output { + let xdg_dir = tempfile::tempdir().unwrap(); + let tls_dir = xdg_dir.path().join("openshell/gateways/openshell/mtls"); + fs::create_dir_all(&tls_dir).unwrap(); + for filename in ["ca.crt", "tls.crt", "tls.key"] { + fs::copy(server.dir.path().join(filename), tls_dir.join(filename)).unwrap(); + } + + tokio::process::Command::new(env!("CARGO_BIN_EXE_openshell")) + .args([ + "--gateway", + "openshell", + "--gateway-endpoint", + &server.endpoint, + "sandbox", + "create", + "--name", + name, + "--no-tty", + "--no-auto-providers", + ]) + .args(extra_args) + .env("XDG_CONFIG_HOME", xdg_dir.path()) + .env("HOME", xdg_dir.path()) + .env("OPENSHELL_PROVISION_TIMEOUT", "5") + .output() + .await + .unwrap() +} + +#[tokio::test] +async fn sandbox_create_json_stdout_is_parseable_with_progress_events() { + let server = run_server().await; + server + .openshell + .state + .vm_slow_progress_before_ready + .store(true, Ordering::SeqCst); + + let result = run_cli_sandbox_create(&server, "json-progress", &["--output=json"]).await; + assert!( + result.status.success(), + "sandbox create failed:\n{}", + String::from_utf8_lossy(&result.stderr) + ); + let stdout = String::from_utf8(result.stdout).expect("stdout should be UTF-8"); + serde_json::from_str::(&stdout) + .unwrap_or_else(|err| panic!("stdout should contain only JSON: {err}\n{stdout}")); +} + +#[tokio::test] +async fn sandbox_create_yaml_stdout_is_parseable_with_progress_events() { + let server = run_server().await; + server + .openshell + .state + .vm_slow_progress_before_ready + .store(true, Ordering::SeqCst); + + let result = run_cli_sandbox_create(&server, "yaml-progress", &["--output=yaml"]).await; + assert!( + result.status.success(), + "sandbox create failed:\n{}", + String::from_utf8_lossy(&result.stderr) + ); + let stdout = String::from_utf8(result.stdout).expect("stdout should be UTF-8"); + serde_yml::from_str::(&stdout) + .unwrap_or_else(|err| panic!("stdout should contain only YAML: {err}\n{stdout}")); +} diff --git a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs index 699749f0d1..5fa2c97029 100644 --- a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs +++ b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs @@ -687,9 +687,16 @@ async fn run_server() -> TestServer { async fn sandbox_get_sends_correct_name() { let ts = run_server().await; - run::sandbox_get(&ts.endpoint, "my-sandbox", false, "default", &ts.tls) - .await - .expect("sandbox_get should succeed"); + run::sandbox_get( + &ts.endpoint, + "my-sandbox", + false, + "table", + "default", + &ts.tls, + ) + .await + .expect("sandbox_get should succeed"); let recorded = ts.openshell.state.last_get_name.lock().await.clone(); assert_eq!( @@ -704,9 +711,16 @@ async fn sandbox_get_sends_correct_name() { async fn sandbox_get_policy_only_round_trip() { let ts = run_server().await; - run::sandbox_get(&ts.endpoint, "my-sandbox", true, "default", &ts.tls) - .await - .expect("sandbox_get with policy_only should succeed"); + run::sandbox_get( + &ts.endpoint, + "my-sandbox", + true, + "table", + "default", + &ts.tls, + ) + .await + .expect("sandbox_get with policy_only should succeed"); let recorded = ts.openshell.state.last_get_name.lock().await.clone(); assert_eq!(recorded.as_deref(), Some("my-sandbox")); @@ -730,7 +744,7 @@ async fn sandbox_get_with_persisted_last_sandbox() { assert_eq!(resolved, "persisted-sb"); // Call sandbox_get with the resolved name. - run::sandbox_get(&ts.endpoint, &resolved, false, "default", &ts.tls) + run::sandbox_get(&ts.endpoint, &resolved, false, "table", "default", &ts.tls) .await .expect("sandbox_get should succeed"); @@ -883,9 +897,16 @@ async fn explicit_name_takes_precedence_over_persisted() { // Persist one name, but supply a different one explicitly. save_last_sandbox("my-cluster", "default", "old-sandbox").expect("save should succeed"); - run::sandbox_get(&ts.endpoint, "explicit-sandbox", false, "default", &ts.tls) - .await - .expect("sandbox_get should succeed"); + run::sandbox_get( + &ts.endpoint, + "explicit-sandbox", + false, + "table", + "default", + &ts.tls, + ) + .await + .expect("sandbox_get should succeed"); let recorded = ts.openshell.state.last_get_name.lock().await.clone(); assert_eq!( diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index 53735a6bb8..35a3732cf9 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -28,16 +28,15 @@ base64 = { workspace = true } chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true } reqwest = { workspace = true, features = ["blocking", "rustls-tls-webpki-roots"], optional = true } +[target.'cfg(unix)'.dependencies] +nix = { workspace = true } + [features] default = ["telemetry"] ## Compile in anonymous telemetry emission support. On by default; disable with ## `--no-default-features` (plus any other features you need) for a build that ## contains no telemetry endpoint, no HTTP client, and no emission code at all. telemetry = ["dep:reqwest", "dep:chrono"] -## Expose proposals::test_helpers (`ProposalsFlagGuard`) to downstream test -## code in other crates. Enabled by openshell-sandbox and -## openshell-supervisor-network dev builds. -test-helpers = [] [build-dependencies] tonic-prost-build = { workspace = true } diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index c9f8a86b21..b200eded64 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -171,9 +171,19 @@ pub fn detect_driver() -> Option { } fn is_podman_available() -> bool { - podman_socket_candidates() + detect_podman_socket().is_some() +} + +/// Return the first responsive Podman API socket, or `None` if none respond. +pub fn detect_podman_socket() -> Option { + detect_podman_socket_from_candidates(&podman_socket_candidates()) +} + +fn detect_podman_socket_from_candidates(candidates: &[PathBuf]) -> Option { + candidates .iter() - .any(|path| podman_socket_responds(path)) + .find(|path| podman_socket_responds(path)) + .cloned() } fn podman_socket_candidates() -> Vec { @@ -972,8 +982,9 @@ mod tests { ComputeDriverKind, Config, DEFAULT_SERVICE_ROUTING_DOMAIN, GatewayInterceptorBindingPolicy, GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, GatewayJwtConfig, GatewayProviderProfileSourceConfig, detect_docker_socket_from_candidates, detect_driver, - docker_host_unix_socket_path, docker_socket_responds, is_unix_socket, - normalize_compute_driver_name, podman_socket_candidates_from_env, podman_socket_responds, + detect_podman_socket_from_candidates, docker_host_unix_socket_path, docker_socket_responds, + is_unix_socket, normalize_compute_driver_name, podman_socket_candidates_from_env, + podman_socket_responds, }; #[cfg(unix)] use std::io::{Read as _, Write as _}; @@ -1349,6 +1360,34 @@ mod tests { ))); } + #[cfg(unix)] + #[test] + fn podman_socket_detection_returns_the_responsive_candidate() { + let temp_dir = tempfile::tempdir().expect("create temp dir"); + let inactive_path = temp_dir.path().join("inactive.sock"); + let inactive_listener = UnixListener::bind(&inactive_path).expect("bind inactive socket"); + drop(inactive_listener); + + let responsive_path = temp_dir.path().join("responsive.sock"); + let listener = UnixListener::bind(&responsive_path).expect("bind responsive socket"); + let handle = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("accept podman probe"); + let mut request = [0_u8; 128]; + let _ = stream.read(&mut request).expect("read podman probe"); + stream + .write_all( + b"HTTP/1.1 200 OK\r\nLibpod-Api-Version: 5.8.2\r\nContent-Length: 2\r\n\r\nOK", + ) + .expect("write podman ping response"); + }); + + assert_eq!( + detect_podman_socket_from_candidates(&[inactive_path, responsive_path.clone(),]), + Some(responsive_path) + ); + handle.join().expect("probe server exits"); + } + #[test] #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 fn detect_driver_prefers_kubernetes_when_k8s_env_is_set() { diff --git a/crates/openshell-core/src/driver_utils.rs b/crates/openshell-core/src/driver_utils.rs index e5eba760d8..a5bcc55ad3 100644 --- a/crates/openshell-core/src/driver_utils.rs +++ b/crates/openshell-core/src/driver_utils.rs @@ -30,6 +30,13 @@ pub const LABEL_SANDBOX_NAMESPACE: &str = "openshell.ai/sandbox-namespace"; /// Container/pod label carrying the sandbox workspace. pub const LABEL_SANDBOX_WORKSPACE: &str = "openshell.ai/sandbox-workspace"; +/// Label selector that matches all OpenShell-managed resources which carry a +/// sandbox ID label. Used by list and watch operations to exclude foreign +/// resources from the same namespace. +pub fn openshell_sandbox_label_selector() -> String { + format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE},{LABEL_SANDBOX_ID}") +} + // --------------------------------------------------------------------------- /// Path to the sandbox supervisor binary inside the container image. @@ -75,6 +82,293 @@ pub const TLS_KEY_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.key"; /// Container-side mount path for the per-sandbox JWT token. pub const SANDBOX_TOKEN_MOUNT_PATH: &str = "/etc/openshell/auth/sandbox.jwt"; +/// Container-side mount path for the corporate upstream-proxy credentials. +/// +/// The file holds the `user:pass` userinfo used to build the +/// `Proxy-Authorization` header. It is delivered through a root-only secret +/// mount so the credential never appears in container environment/metadata. +pub const UPSTREAM_PROXY_AUTH_MOUNT_PATH: &str = "/etc/openshell/auth/upstream-proxy"; + +/// A validated corporate upstream-proxy address. +/// +/// Produced by [`parse_upstream_proxy_url`], which is the single source of +/// truth for what counts as a valid upstream proxy URL. Compute drivers use +/// it to reject bad operator config at sandbox-create time, and the +/// in-container supervisor applies the same rules to its driver-supplied +/// arguments so a value one side accepts is never rejected (or silently +/// ignored) by the other. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UpstreamProxyAddr { + /// Proxy hostname, IPv4, or IPv6 address (IPv6 without brackets). + pub host: String, + /// Proxy TCP port (always explicit in the accepted URL grammar). + pub port: u16, +} + +/// Why an upstream proxy URL was rejected by [`parse_upstream_proxy_url`]. +/// +/// Kept as a typed error so each consumer (driver config validation, +/// supervisor startup) can phrase the message for its own surface while +/// enforcing identical semantics. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum UpstreamProxyUrlError { + /// The value is empty or whitespace-only. + #[error("proxy URL is empty")] + Empty, + /// The value does not parse as a URL. + #[error("not a valid proxy URL: {0}")] + Invalid(url::ParseError), + /// The value has no `scheme://` prefix. Bare `host[:port]` forms are + /// rejected so the accepted grammar matches the documented + /// `http://host:port` contract exactly. + #[error("proxy URL must include an explicit scheme, e.g. http://proxy.corp.com:3128")] + MissingScheme, + /// The URL uses a scheme other than `http` (TLS and SOCKS proxies are + /// not supported by the sandbox supervisor). + #[error( + "unsupported proxy scheme '{0}': only http:// forward proxies are \ + supported by the sandbox supervisor" + )] + UnsupportedScheme(String), + /// The URL has no explicit port. Corporate proxies rarely listen on the + /// scheme default (80), so a forgotten port is rejected instead of + /// silently dialing port 80. + #[error("proxy URL must include an explicit proxy port, e.g. http://proxy.corp.com:3128")] + MissingPort, + /// The URL specifies port `0`, which is not a connectable TCP port. It + /// would pass startup validation but fail every proxied dial, so it is + /// rejected up front. + #[error("proxy URL port must not be 0")] + ZeroPort, + /// The URL embeds `user:pass@` credentials, which would leak into config + /// and container metadata. Credentials must come from the proxy auth file. + #[error("proxy URL must not embed credentials; supply them via the proxy auth file")] + InlineCredentials, + /// The URL has no host component. + #[error("proxy URL is missing a proxy host")] + MissingHost, + /// The URL carries a path, query, or fragment. A forward proxy is + /// addressed by `host:port` only, so extra components indicate a + /// misconfiguration (e.g. a pasted endpoint URL) and are rejected instead + /// of being silently discarded. + #[error("proxy URL must not contain a {0}; use scheme://host:port only")] + UnexpectedComponent(&'static str), +} + +/// Parse and validate a corporate upstream-proxy URL. +/// +/// The accepted grammar is exactly `http://host:port`: the scheme and the +/// port must both be explicit, only `http://` proxies are accepted, and +/// inline userinfo is rejected. The URL must address the proxy only: a path +/// (other than a bare trailing `/`), query, or fragment is rejected rather +/// than silently discarded. +/// +/// # Errors +/// +/// Returns an [`UpstreamProxyUrlError`] describing the first rule the value +/// violates. +pub fn parse_upstream_proxy_url(raw: &str) -> Result { + let trimmed = raw.trim(); + if trimmed.is_empty() { + return Err(UpstreamProxyUrlError::Empty); + } + if !trimmed.contains("://") { + return Err(UpstreamProxyUrlError::MissingScheme); + } + let parsed = url::Url::parse(trimmed).map_err(UpstreamProxyUrlError::Invalid)?; + + if !parsed.scheme().eq_ignore_ascii_case("http") { + return Err(UpstreamProxyUrlError::UnsupportedScheme( + parsed.scheme().to_string(), + )); + } + if !parsed.username().is_empty() || parsed.password().is_some() { + return Err(UpstreamProxyUrlError::InlineCredentials); + } + let host = match parsed.host() { + // `Host::Ipv6` renders without brackets, which is what socket + // connect APIs expect. + Some(url::Host::Ipv6(ip)) => ip.to_string(), + Some(host) => host.to_string(), + None => return Err(UpstreamProxyUrlError::MissingHost), + }; + if host.is_empty() { + return Err(UpstreamProxyUrlError::MissingHost); + } + // The `url` crate normalizes an absent path to "/" for http URLs, so a + // bare trailing slash is indistinguishable from no path and is accepted. + if !matches!(parsed.path(), "" | "/") { + return Err(UpstreamProxyUrlError::UnexpectedComponent("path")); + } + if parsed.query().is_some() { + return Err(UpstreamProxyUrlError::UnexpectedComponent("query")); + } + if parsed.fragment().is_some() { + return Err(UpstreamProxyUrlError::UnexpectedComponent("fragment")); + } + if !authority_has_explicit_port(trimmed) { + return Err(UpstreamProxyUrlError::MissingPort); + } + // Explicit-port presence was verified above; `port()` is `None` only + // when the URL spells out the scheme default (`:80`), which the url crate + // normalizes away. + let port = parsed.port().unwrap_or(80); + if port == 0 { + return Err(UpstreamProxyUrlError::ZeroPort); + } + Ok(UpstreamProxyAddr { host, port }) +} + +/// Return `true` when the raw URL's authority carries an explicit `:port`. +/// +/// The `url` crate normalizes a scheme-default port (`:80` for http) to +/// `None`, making it indistinguishable from an absent port in the parsed +/// form, so the raw authority must be inspected instead. +fn authority_has_explicit_port(raw: &str) -> bool { + let after_scheme = raw.split_once("://").map_or(raw, |(_, rest)| rest); + let authority_end = after_scheme + .find(['/', '?', '#']) + .unwrap_or(after_scheme.len()); + let authority = &after_scheme[..authority_end]; + // Userinfo is rejected by the caller, but strip it anyway so this check + // never misreads a `user:pass@` colon as a port. + let host_port = authority.rsplit_once('@').map_or(authority, |(_, hp)| hp); + host_port.rfind(']').map_or_else( + || { + host_port + .rsplit_once(':') + .is_some_and(|(_, port)| !port.is_empty()) + }, + // Bracketed IPv6 literal: a port can only follow the bracket, and a + // bare trailing `]:` is no more explicit than no port at all. + |end| { + host_port[end + 1..] + .strip_prefix(':') + .is_some_and(|port| !port.is_empty()) + }, + ) +} + +/// Why an upstream proxy credential was rejected by +/// [`parse_upstream_proxy_credential`]. +/// +/// Variants carry no payload so an error can never leak credential content +/// into logs or user-facing messages. +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +pub enum UpstreamProxyCredentialError { + /// The credential is empty or whitespace-only. + #[error("credential is empty")] + Empty, + /// The credential contains control characters (CR, LF, NUL, tab, ...) + /// that could inject additional HTTP headers. + #[error("credential contains control characters")] + ControlCharacters, + /// The credential has no `:` separating user from password. + #[error("credential must use the user:pass form (missing ':')")] + MissingSeparator, + /// The credential has an empty user before the `:` separator. + #[error("credential must use the user:pass form (empty user)")] + EmptyUser, +} + +/// Validate a corporate upstream-proxy credential read from the proxy auth +/// file, returning the trimmed `user:pass` value. +/// +/// Single source of truth for what counts as a valid proxy credential: the +/// compute driver applies it at sandbox-create time (before staging the +/// secret) and the in-container supervisor applies it again before building +/// the `Proxy-Authorization: Basic` header, so a credential one side accepts +/// is never rejected by the other. +/// +/// Surrounding whitespace (including the conventional trailing newline) is +/// trimmed. The user part must be non-empty; the password may be empty and +/// may itself contain `:` (per RFC 7617 the first `:` is the separator). +/// +/// # Errors +/// +/// Returns an [`UpstreamProxyCredentialError`] describing the first rule the +/// value violates. Errors never contain the credential itself. +pub fn parse_upstream_proxy_credential(raw: &str) -> Result<&str, UpstreamProxyCredentialError> { + let credential = raw.trim(); + if credential.is_empty() { + return Err(UpstreamProxyCredentialError::Empty); + } + if credential.contains(|c: char| c.is_control()) { + return Err(UpstreamProxyCredentialError::ControlCharacters); + } + match credential.split_once(':') { + None => Err(UpstreamProxyCredentialError::MissingSeparator), + Some(("", _)) => Err(UpstreamProxyCredentialError::EmptyUser), + Some(_) => Ok(credential), + } +} + +/// Hard upper bound on the size of a proxy-auth credential file. +/// +/// A `user:pass` credential is tiny; this cap only exists to stop a hostile +/// or misconfigured path (a huge file, or a special file such as +/// `/dev/zero`) from exhausting memory during a bounded read. +pub const MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES: u64 = 4096; + +/// Read a proxy-auth credential file with a hard size bound. +/// +/// Rejects non-regular files (e.g. `/dev/zero`, directories, FIFOs) and +/// files larger than [`MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES`], and reads at +/// most that many bytes, so a hostile or misconfigured path cannot exhaust +/// gateway or supervisor memory. Returns the raw contents; callers pass the +/// result to [`parse_upstream_proxy_credential`]. +/// +/// Shared by the compute driver (at sandbox-create time) and the in-container +/// supervisor so both enforce the same bound. This is a blocking read; async +/// callers should wrap it (e.g. `tokio::task::spawn_blocking`). +/// +/// # Errors +/// +/// Returns a descriptive error (never containing file contents) when the path +/// cannot be opened or stat'd, is not a regular file, or exceeds the size +/// bound. +pub fn read_upstream_proxy_credential_file(path: &str) -> Result { + use std::io::Read as _; + + // On Unix, open non-blocking so a FIFO with no writer does not hang the + // open() call indefinitely; the regular-file check below then rejects it. + // O_NONBLOCK has no effect on the subsequent read of a regular file. + #[cfg(unix)] + let open_result = { + use std::os::unix::fs::OpenOptionsExt as _; + std::fs::OpenOptions::new() + .read(true) + .custom_flags(nix::libc::O_NONBLOCK) + .open(path) + }; + #[cfg(not(unix))] + let open_result = std::fs::File::open(path); + + let file = open_result.map_err(|e| format!("failed to open proxy auth file '{path}': {e}"))?; + let metadata = file + .metadata() + .map_err(|e| format!("failed to stat proxy auth file '{path}': {e}"))?; + if !metadata.is_file() { + return Err(format!("proxy auth file '{path}' is not a regular file")); + } + if metadata.len() > MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES { + return Err(format!( + "proxy auth file '{path}' exceeds the {MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES}-byte limit" + )); + } + // Bound the read even if the file grows between stat and read. + let mut buf = String::new(); + file.take(MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES + 1) + .read_to_string(&mut buf) + .map_err(|e| format!("failed to read proxy auth file '{path}': {e}"))?; + if buf.len() as u64 > MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES { + return Err(format!( + "proxy auth file '{path}' exceeds the {MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES}-byte limit" + )); + } + Ok(buf) +} + /// Return the XDG state path for a driver's sandbox JWT token file. /// /// The resulting path is `$XDG_STATE_HOME/openshell/[/]//sandbox.jwt`. @@ -163,3 +457,226 @@ pub fn supervisor_image_tag(image: &str) -> Option<&str> { pub fn supervisor_image_should_refresh(image: &str) -> bool { matches!(supervisor_image_tag(image), Some("dev" | "latest")) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn upstream_proxy_url_accepts_http_with_port() { + let addr = parse_upstream_proxy_url("http://proxy.corp.com:8080").unwrap(); + assert_eq!(addr.host, "proxy.corp.com"); + assert_eq!(addr.port, 8080); + } + + #[test] + fn upstream_proxy_url_rejects_missing_scheme() { + for url in [ + "proxy.corp.com", + "proxy.corp.com:3128", + "user:pass@proxy.corp.com:8080", + ] { + assert_eq!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::MissingScheme), + "{url}" + ); + } + } + + #[test] + fn upstream_proxy_url_rejects_missing_port() { + for url in [ + "http://proxy.corp.com", + "http://proxy.corp.com/", + "http://proxy.corp.com:", + "http://[fd00::1]", + "http://[fd00::1]:", + ] { + assert_eq!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::MissingPort), + "{url}" + ); + } + // An explicit scheme-default port is accepted even though the url + // crate normalizes it away in the parsed form. + let addr = parse_upstream_proxy_url("http://proxy.corp.com:80").unwrap(); + assert_eq!(addr.port, 80); + } + + #[test] + fn upstream_proxy_url_rejects_zero_port() { + // Port 0 parses as an explicit port but is not connectable; reject it + // up front instead of failing every proxied dial later. + for url in ["http://proxy.corp.com:0", "http://[fd00::1]:0"] { + assert_eq!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::ZeroPort), + "{url}" + ); + } + } + + #[test] + fn upstream_proxy_url_ipv6_host_is_bracket_free() { + let addr = parse_upstream_proxy_url("http://[fd00::1]:8080").unwrap(); + assert_eq!(addr.host, "fd00::1"); + assert_eq!(addr.port, 8080); + } + + #[test] + fn upstream_proxy_url_rejects_tls_and_socks_schemes() { + for url in ["https://proxy:443", "socks5://proxy:1080"] { + assert!(matches!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::UnsupportedScheme(_)) + )); + } + } + + #[test] + fn upstream_proxy_url_rejects_inline_credentials() { + for url in ["http://user:pass@proxy:8080", "http://user@proxy:8080"] { + assert_eq!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::InlineCredentials) + ); + } + } + + #[test] + fn upstream_proxy_url_rejects_empty_and_invalid() { + assert_eq!( + parse_upstream_proxy_url(" "), + Err(UpstreamProxyUrlError::Empty) + ); + assert!(matches!( + parse_upstream_proxy_url("http://proxy:notaport"), + Err(UpstreamProxyUrlError::Invalid(_)) + )); + assert!(parse_upstream_proxy_url("http://").is_err()); + } + + #[test] + fn upstream_proxy_url_rejects_path_query_and_fragment() { + for (url, component) in [ + ("http://proxy.corp.com:8080/some/path", "path"), + ("http://proxy.corp.com:8080?x=1", "query"), + ("http://proxy.corp.com:8080/?x=1", "query"), + ("http://proxy.corp.com:8080#frag", "fragment"), + ] { + assert_eq!( + parse_upstream_proxy_url(url), + Err(UpstreamProxyUrlError::UnexpectedComponent(component)), + "{url}" + ); + } + // A bare trailing slash is URL normalization, not a real path. + let addr = parse_upstream_proxy_url("http://proxy.corp.com:8080/").unwrap(); + assert_eq!(addr.host, "proxy.corp.com"); + assert_eq!(addr.port, 8080); + } + + #[test] + fn upstream_proxy_credential_accepts_user_pass_and_trims() { + assert_eq!( + parse_upstream_proxy_credential("user:pass\n"), + Ok("user:pass") + ); + // The password may be empty and may contain further colons. + assert_eq!(parse_upstream_proxy_credential("user:"), Ok("user:")); + assert_eq!( + parse_upstream_proxy_credential("user:p@:ss"), + Ok("user:p@:ss") + ); + } + + #[test] + fn upstream_proxy_credential_rejects_empty() { + for raw in ["", " ", "\n"] { + assert_eq!( + parse_upstream_proxy_credential(raw), + Err(UpstreamProxyCredentialError::Empty) + ); + } + } + + #[test] + fn upstream_proxy_credential_rejects_control_characters() { + for raw in ["user:pa\r\nss", "user:pa\0ss", "user:pa\tss"] { + assert_eq!( + parse_upstream_proxy_credential(raw), + Err(UpstreamProxyCredentialError::ControlCharacters) + ); + } + } + + #[test] + fn upstream_proxy_credential_rejects_malformed_user_pass_form() { + assert_eq!( + parse_upstream_proxy_credential("userpass"), + Err(UpstreamProxyCredentialError::MissingSeparator) + ); + assert_eq!( + parse_upstream_proxy_credential(":pass"), + Err(UpstreamProxyCredentialError::EmptyUser) + ); + } + + #[test] + fn credential_file_reads_within_the_size_bound() { + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), "user:pass\n").unwrap(); + let raw = read_upstream_proxy_credential_file(file.path().to_str().unwrap()).unwrap(); + assert_eq!(parse_upstream_proxy_credential(&raw), Ok("user:pass")); + } + + #[test] + fn credential_file_rejects_oversized_files() { + let file = tempfile::NamedTempFile::new().unwrap(); + let huge = vec![b'a'; usize::try_from(MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES + 1).unwrap()]; + std::fs::write(file.path(), &huge).unwrap(); + let err = read_upstream_proxy_credential_file(file.path().to_str().unwrap()).unwrap_err(); + assert!(err.contains("limit"), "{err}"); + } + + #[test] + fn credential_file_rejects_non_regular_files() { + // A directory is a non-regular path; /dev/zero would be rejected the + // same way (not a regular file) without risking an unbounded read. + let dir = tempfile::tempdir().unwrap(); + let err = read_upstream_proxy_credential_file(dir.path().to_str().unwrap()).unwrap_err(); + assert!(err.contains("regular file"), "{err}"); + + if std::path::Path::new("/dev/zero").exists() { + let err = read_upstream_proxy_credential_file("/dev/zero").unwrap_err(); + assert!(err.contains("regular file"), "{err}"); + } + } + + #[test] + fn credential_file_missing_path_is_an_error() { + let err = read_upstream_proxy_credential_file("/nonexistent/proxy-auth").unwrap_err(); + assert!(err.contains("open proxy auth file"), "{err}"); + } + + #[cfg(unix)] + #[test] + fn credential_file_rejects_fifo_without_hanging() { + // A FIFO with no writer would block a blocking open() forever. The + // reader opens non-blocking and rejects the non-regular file, so it + // must return promptly even though nothing ever opens the write end. + let dir = tempfile::tempdir().unwrap(); + let fifo = dir.path().join("proxy-auth-fifo"); + nix::unistd::mkfifo(&fifo, nix::sys::stat::Mode::S_IRUSR).unwrap(); + + let start = std::time::Instant::now(); + let err = read_upstream_proxy_credential_file(fifo.to_str().unwrap()).unwrap_err(); + assert!(err.contains("regular file"), "{err}"); + assert!( + start.elapsed() < std::time::Duration::from_secs(5), + "reading a FIFO must not block" + ); + } +} diff --git a/crates/openshell-core/src/net.rs b/crates/openshell-core/src/net.rs index 59bfdfc5d5..3f14a397b8 100644 --- a/crates/openshell-core/src/net.rs +++ b/crates/openshell-core/src/net.rs @@ -10,6 +10,7 @@ //! - The mechanistic mapper for proposal filtering //! - The gateway server for defense-in-depth validation on approval +use ipnet::{IpNet, Ipv4Net, Ipv6Net}; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; /// Check if a hostname is a known cloud metadata hostname that resolves to an @@ -80,9 +81,9 @@ pub fn is_always_blocked_ip(ip: IpAddr) -> bool { /// /// Used at policy load time and server-side approval to reject entries that /// would be silently blocked at runtime by [`is_always_blocked_ip`]. -pub fn is_always_blocked_net(net: ipnet::IpNet) -> bool { +pub fn is_always_blocked_net(net: IpNet) -> bool { match net { - ipnet::IpNet::V4(v4net) => { + IpNet::V4(v4net) => { let network = v4net.network(); let broadcast = v4net.broadcast(); @@ -107,7 +108,7 @@ pub fn is_always_blocked_net(net: ipnet::IpNet) -> bool { false } - ipnet::IpNet::V6(v6net) => { + IpNet::V6(v6net) => { // For IPv6, check the network address itself and representative // addresses within the range. let network = v6net.network(); @@ -155,6 +156,43 @@ pub fn is_always_blocked_net(net: ipnet::IpNet) -> bool { } } +const RFC1918_10_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(10, 0, 0, 0), 8); +const RFC1918_172_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(172, 16, 0, 0), 12); +const RFC1918_192_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(192, 168, 0, 0), 16); +const CGNAT_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(100, 64, 0, 0), 10); +const IETF_PROTOCOL_ASSIGNMENTS_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(192, 0, 0, 0), 24); +const TEST_NET_1: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(192, 0, 2, 0), 24); +const BENCHMARKING_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(198, 18, 0, 0), 15); +const TEST_NET_2: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(198, 51, 100, 0), 24); +const TEST_NET_3: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::new(203, 0, 113, 0), 24); +const LIMITED_BROADCAST_NET: Ipv4Net = Ipv4Net::new_assert(Ipv4Addr::BROADCAST, 32); +const IPV6_ULA_NET: Ipv6Net = Ipv6Net::new_assert(Ipv6Addr::new(0xfc00, 0, 0, 0, 0, 0, 0, 0), 7); + +const NON_HARD_INTERNAL_V4_NETS: [Ipv4Net; 10] = [ + RFC1918_10_NET, + RFC1918_172_NET, + RFC1918_192_NET, + CGNAT_NET, + IETF_PROTOCOL_ASSIGNMENTS_NET, + TEST_NET_1, + BENCHMARKING_NET, + TEST_NET_2, + TEST_NET_3, + LIMITED_BROADCAST_NET, +]; + +fn ipv4_nets_intersect(left: Ipv4Net, right: Ipv4Net) -> bool { + left.network() <= right.broadcast() && right.network() <= left.broadcast() +} + +fn ipv6_nets_intersect(left: Ipv6Net, right: Ipv6Net) -> bool { + left.network() <= right.broadcast() && right.network() <= left.broadcast() +} + +fn ipv4_net_to_mapped_ipv6(net: Ipv4Net) -> Ipv6Net { + Ipv6Net::new_assert(net.network().to_ipv6_mapped(), 96_u8 + net.prefix_len()) +} + /// Check if an IP address is internal (loopback, private RFC 1918, link-local, /// or unspecified). /// @@ -188,6 +226,26 @@ pub fn is_internal_ip(ip: IpAddr) -> bool { } } +/// Check if a CIDR network intersects any address range classified by +/// [`is_internal_ip`]. +pub fn is_internal_net(net: IpNet) -> bool { + if is_always_blocked_net(net) { + return true; + } + + match net { + IpNet::V4(net) => NON_HARD_INTERNAL_V4_NETS + .iter() + .any(|internal| ipv4_nets_intersect(net, *internal)), + IpNet::V6(net) => { + ipv6_nets_intersect(net, IPV6_ULA_NET) + || NON_HARD_INTERNAL_V4_NETS + .iter() + .any(|internal| ipv6_nets_intersect(net, ipv4_net_to_mapped_ipv6(*internal))) + } + } +} + /// IPv4 internal address check covering RFC 1918, CGNAT (RFC 6598), and other /// special-use ranges that should never be reachable from sandbox egress. fn is_internal_v4(v4: Ipv4Addr) -> bool { @@ -382,45 +440,45 @@ mod tests { #[test] fn test_always_blocked_net_loopback_v4() { - let net: ipnet::IpNet = "127.0.0.0/8".parse().unwrap(); + let net: IpNet = "127.0.0.0/8".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_link_local_v4() { - let net: ipnet::IpNet = "169.254.0.0/16".parse().unwrap(); + let net: IpNet = "169.254.0.0/16".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_unspecified_v4() { - let net: ipnet::IpNet = "0.0.0.0/32".parse().unwrap(); + let net: IpNet = "0.0.0.0/32".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_loopback_v6() { - let net: ipnet::IpNet = "::1/128".parse().unwrap(); + let net: IpNet = "::1/128".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_link_local_v6() { - let net: ipnet::IpNet = "fe80::/10".parse().unwrap(); + let net: IpNet = "fe80::/10".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_ipv4_mapped_v6_loopback() { - let net: ipnet::IpNet = "::ffff:127.0.0.1/128".parse().unwrap(); + let net: IpNet = "::ffff:127.0.0.1/128".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_allows_rfc1918() { - let net10: ipnet::IpNet = "10.0.0.0/8".parse().unwrap(); - let net172: ipnet::IpNet = "172.16.0.0/12".parse().unwrap(); - let net192: ipnet::IpNet = "192.168.0.0/16".parse().unwrap(); + let net10: IpNet = "10.0.0.0/8".parse().unwrap(); + let net172: IpNet = "172.16.0.0/12".parse().unwrap(); + let net192: IpNet = "192.168.0.0/16".parse().unwrap(); assert!(!is_always_blocked_net(net10)); assert!(!is_always_blocked_net(net172)); assert!(!is_always_blocked_net(net192)); @@ -428,44 +486,44 @@ mod tests { #[test] fn test_always_blocked_net_allows_public() { - let net: ipnet::IpNet = "8.8.8.0/24".parse().unwrap(); + let net: IpNet = "8.8.8.0/24".parse().unwrap(); assert!(!is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_single_ip_loopback() { - let net: ipnet::IpNet = "127.0.0.1/32".parse().unwrap(); + let net: IpNet = "127.0.0.1/32".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_single_ip_metadata() { - let net: ipnet::IpNet = "169.254.169.254/32".parse().unwrap(); + let net: IpNet = "169.254.169.254/32".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_broad_cidr_containing_blocked() { // 0.0.0.0/0 contains everything including unspecified, loopback, link-local - let net: ipnet::IpNet = "0.0.0.0/0".parse().unwrap(); + let net: IpNet = "0.0.0.0/0".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_v6_broad_containing_loopback() { - let net: ipnet::IpNet = "::/0".parse().unwrap(); + let net: IpNet = "::/0".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_v6_ipv4_mapped_loopback_single() { - let net: ipnet::IpNet = "::ffff:127.0.0.1/128".parse().unwrap(); + let net: IpNet = "::ffff:127.0.0.1/128".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_v6_ipv4_mapped_link_local_single() { - let net: ipnet::IpNet = "::ffff:169.254.0.1/128".parse().unwrap(); + let net: IpNet = "::ffff:169.254.0.1/128".parse().unwrap(); assert!(is_always_blocked_net(net)); } @@ -474,7 +532,7 @@ mod tests { // ::ffff:168.0.0.0/103 has a public network address (168.0.0.0) but // the range covers 168.0.0.0–169.255.255.255, which includes the // link-local block 169.254.0.0/16. - let net: ipnet::IpNet = "::ffff:168.0.0.0/103".parse().unwrap(); + let net: IpNet = "::ffff:168.0.0.0/103".parse().unwrap(); assert!(is_always_blocked_net(net)); } @@ -482,17 +540,74 @@ mod tests { fn test_always_blocked_net_v6_ipv4_mapped_broad_spans_loopback() { // ::ffff:64.0.0.0/98 has a public network address (64.0.0.0) but the // range covers 64.0.0.0–127.255.255.255, which includes loopback. - let net: ipnet::IpNet = "::ffff:64.0.0.0/98".parse().unwrap(); + let net: IpNet = "::ffff:64.0.0.0/98".parse().unwrap(); assert!(is_always_blocked_net(net)); } #[test] fn test_always_blocked_net_v6_ipv4_mapped_allows_public() { // ::ffff:8.8.8.8/128 is a public address — should not be blocked. - let net: ipnet::IpNet = "::ffff:8.8.8.8/128".parse().unwrap(); + let net: IpNet = "::ffff:8.8.8.8/128".parse().unwrap(); assert!(!is_always_blocked_net(net)); } + // -- is_internal_net -- + + #[test] + fn test_internal_net_rfc1918_contained_and_supernet() { + for cidr in ["10.1.0.0/16", "8.0.0.0/5"] { + assert!(is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + + #[test] + fn test_internal_net_cgnat_contained_and_supernet() { + for cidr in ["100.64.0.0/10", "100.0.0.0/9"] { + assert!(is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + + #[test] + fn test_internal_net_ipv4_special_use_ranges() { + for cidr in [ + "192.0.0.0/24", + "192.0.2.0/24", + "198.18.0.0/15", + "198.51.100.0/24", + "203.0.113.0/24", + "255.255.255.255/32", + ] { + assert!(is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + + #[test] + fn test_internal_net_ipv6_ula() { + for cidr in ["fc00::/7", "fd00::/8"] { + assert!(is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + + #[test] + fn test_internal_net_ipv4_mapped_cgnat_supernet() { + let net: IpNet = "::ffff:100.0.0.0/105".parse().unwrap(); + assert!(is_internal_net(net)); + } + + #[test] + fn test_internal_net_always_blocked() { + for cidr in ["127.0.0.0/8", "fe80::/10"] { + assert!(is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + + #[test] + fn test_internal_net_allows_large_public_ranges() { + for cidr in ["8.0.0.0/8", "::ffff:8.0.0.0/104"] { + assert!(!is_internal_net(cidr.parse().unwrap()), "{cidr}"); + } + } + // -- is_internal_ip -- #[test] diff --git a/crates/openshell-core/src/proposals.rs b/crates/openshell-core/src/proposals.rs index fcb6b110ce..53586629ca 100644 --- a/crates/openshell-core/src/proposals.rs +++ b/crates/openshell-core/src/proposals.rs @@ -1,83 +1,50 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Process-wide flag controlling agent-driven policy proposals. +//! Shared state controlling agent-driven policy proposals. //! //! Initialised once during sandbox start from the `agent_policy_proposals_enabled` -//! setting and updated by the policy poll loop when the setting changes. Read -//! by the `policy.local` route handler and by the skills installer to gate the -//! agent-controlled mutation surface. Tests use [`test_helpers::ProposalsFlagGuard`] -//! to flip the flag through a serialized guard. +//! setting and updated by the policy poll loop or authoritative sidecar control +//! when the setting changes. Read by the `policy.local` route handler and by +//! the skills installer to gate the agent-controlled mutation surface. use std::sync::Arc; -use std::sync::OnceLock; use std::sync::atomic::{AtomicBool, Ordering}; -/// Process-wide handle to the agent-proposals flag. +/// Shared handle for the agent-driven policy proposal surface. /// -/// Set once by `run_sandbox()` during start; subsequent attempts to set it are -/// ignored. The contained `AtomicBool` is updated by the policy poll loop. -pub static AGENT_PROPOSALS_ENABLED: OnceLock> = OnceLock::new(); - -/// Read the current value of the agent proposals feature flag. -/// -/// Returns `false` if the flag has not been initialized (e.g. during unit -/// tests), matching the documented default for the setting. -pub fn agent_proposals_enabled() -> bool { - AGENT_PROPOSALS_ENABLED - .get() - .is_some_and(|flag| flag.load(Ordering::Relaxed)) +/// Clones point at the same atomic value, so the sandbox orchestrator can pass +/// this into the process and network supervisors and then update it from the +/// settings poll loop or sidecar control. +#[derive(Clone, Debug)] +pub struct AgentProposals { + enabled: Arc, } -/// Test-only helpers shared across crates' test modules. -#[cfg(any(test, feature = "test-helpers"))] -pub mod test_helpers { - use std::sync::Arc; - use std::sync::LazyLock; - use std::sync::atomic::{AtomicBool, Ordering}; - use tokio::sync::MutexGuard; - - static PROPOSALS_FLAG_LOCK: LazyLock> = - LazyLock::new(|| tokio::sync::Mutex::new(())); - - /// Guard for tests that toggle the process-wide flag. - /// - /// Acquires a process-wide async mutex, swaps in the requested value, and - /// restores the previous value on drop. Hold the guard for the duration of - /// any code that reads `agent_proposals_enabled()`. - pub struct ProposalsFlagGuard { - prev: bool, - flag: Arc, - _lock: MutexGuard<'static, ()>, +impl AgentProposals { + #[must_use] + pub fn new(enabled: bool) -> Self { + Self { + enabled: Arc::new(AtomicBool::new(enabled)), + } } - impl ProposalsFlagGuard { - pub async fn set(enabled: bool) -> Self { - let lock = PROPOSALS_FLAG_LOCK.lock().await; - Self::with_lock(enabled, lock) - } + #[must_use] + pub fn enabled(&self) -> bool { + self.enabled.load(Ordering::Relaxed) + } - pub fn set_blocking(enabled: bool) -> Self { - let lock = PROPOSALS_FLAG_LOCK.blocking_lock(); - Self::with_lock(enabled, lock) - } + pub fn set_enabled(&self, enabled: bool) { + self.enabled.store(enabled, Ordering::Relaxed); + } - fn with_lock(enabled: bool, lock: MutexGuard<'static, ()>) -> Self { - let flag = super::AGENT_PROPOSALS_ENABLED - .get_or_init(|| Arc::new(AtomicBool::new(false))) - .clone(); - let prev = flag.swap(enabled, Ordering::Relaxed); - Self { - prev, - flag, - _lock: lock, - } - } + pub fn swap_enabled(&self, enabled: bool) -> bool { + self.enabled.swap(enabled, Ordering::Relaxed) } +} - impl Drop for ProposalsFlagGuard { - fn drop(&mut self) { - self.flag.store(self.prev, Ordering::Relaxed); - } +impl Default for AgentProposals { + fn default() -> Self { + Self::new(false) } } diff --git a/crates/openshell-core/src/sandbox_env.rs b/crates/openshell-core/src/sandbox_env.rs index f066580636..f15ce34bb1 100644 --- a/crates/openshell-core/src/sandbox_env.rs +++ b/crates/openshell-core/src/sandbox_env.rs @@ -114,3 +114,8 @@ pub const SANDBOX_UID: &str = "OPENSHELL_SANDBOX_UID"; /// Used alongside UID for PVC init container `chown` operations and when the /// supervisor drops privileges to a group other than the UID's primary group. pub const SANDBOX_GID: &str = "OPENSHELL_SANDBOX_GID"; + +// The corporate upstream-proxy configuration deliberately has no reserved +// environment variables: it travels on the supervisor's argv +// (`--upstream-proxy` and friends), which a sandbox image cannot forge the +// way it could bake `ENV` values. diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 5dba71adcc..c784f10db9 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -21,7 +21,7 @@ use kube::{Client, Error as KubeError}; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, - LABEL_SANDBOX_WORKSPACE, SUPERVISOR_IMAGE_BINARY_PATH, + LABEL_SANDBOX_WORKSPACE, SUPERVISOR_IMAGE_BINARY_PATH, openshell_sandbox_label_selector, }; use openshell_core::gpu::{driver_gpu_requirements, effective_driver_gpu_count}; use openshell_core::progress::{ @@ -744,10 +744,9 @@ impl KubernetesComputeDriver { .await?; match tokio::time::timeout( KUBE_API_TIMEOUT, - agent_sandbox_api.api.list( - &ListParams::default() - .labels(&format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}")), - ), + agent_sandbox_api + .api + .list(&ListParams::default().labels(&openshell_sandbox_label_selector())), ) .await { @@ -755,8 +754,16 @@ impl KubernetesComputeDriver { let mut sandboxes: Vec = list .items .into_iter() - .filter_map(|obj| sandbox_from_object(&self.config.namespace, obj).ok()) - .map(|(_, s)| s) + .filter_map(|obj| { + let name = obj.metadata.name.clone().unwrap_or_default(); + match sandbox_from_object(&self.config.namespace, obj) { + Ok((_, s)) => Some(s), + Err(err) => { + warn!(object_name = %name, error = %err, "skipping unrecognized Sandbox in list"); + None + } + } + }) .collect(); sandboxes.sort_by(|left, right| { left.name @@ -1034,8 +1041,7 @@ impl KubernetesComputeDriver { .supported_agent_sandbox_api(self.watch_client.clone()) .await?; let event_api: Api = Api::namespaced(self.watch_client.clone(), &namespace); - let watcher_config = watcher::Config::default() - .labels(&format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}")); + let watcher_config = watcher::Config::default().labels(&openshell_sandbox_label_selector()); let mut sandbox_stream = watcher::watcher(agent_sandbox_api.api, watcher_config).boxed(); let mut event_stream = watcher::watcher(event_api, watcher::Config::default()).boxed(); let (tx, rx) = mpsc::channel(256); @@ -1132,7 +1138,8 @@ impl KubernetesComputeDriver { let _ = tx.send(Err(KubernetesDriverError::Message(err.to_string()))).await; break; } - } + }, + () = tx.closed() => break, } } }); diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index c6a619f3d1..d068dfbc8f 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -331,7 +331,7 @@ Podman resources after out-of-band container removal or label drift. | Environment Variable | CLI Flag | Default | Description | |---|---|---|---| -| `OPENSHELL_PODMAN_SOCKET` | `--podman-socket` | `$XDG_RUNTIME_DIR/podman/podman.sock` on Linux, `$HOME/.local/share/containers/podman/machine/podman.sock` on macOS | Podman API Unix socket path. | +| `OPENSHELL_PODMAN_SOCKET` | `--podman-socket` | Probes known local Podman API sockets and uses the first responsive socket. Fails to start if none respond. | Podman API Unix socket path. | | `OPENSHELL_SANDBOX_IMAGE` | `--sandbox-image` | From gateway config | Default OCI image for sandboxes. | | `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY` | `--sandbox-image-pull-policy` | `missing` | Pull policy: `always`, `missing`, `never`, or `newer`. | | `OPENSHELL_GRPC_ENDPOINT` | `--grpc-endpoint` | Auto-detected via `host.containers.internal` | Gateway gRPC endpoint for sandbox callbacks. | @@ -345,6 +345,38 @@ Podman resources after out-of-band container removal or label drift. | `OPENSHELL_PODMAN_TLS_CA` | `--podman-tls-ca` | unset | Host path to the CA certificate mounted for sandbox mTLS. | | `OPENSHELL_PODMAN_TLS_CERT` | `--podman-tls-cert` | unset | Host path to the client certificate mounted for sandbox mTLS. | | `OPENSHELL_PODMAN_TLS_KEY` | `--podman-tls-key` | unset | Host path to the client private key mounted for sandbox mTLS. | +| `OPENSHELL_SANDBOX_HTTPS_PROXY` | `--sandbox-https-proxy` | unset | Corporate forward proxy URL for the supervisor's upstream TLS dials, chained with HTTP CONNECT. Only credential-free `http://host:port` URLs are supported (scheme and port required). Plain-HTTP requests always dial directly. | +| `OPENSHELL_SANDBOX_NO_PROXY` | `--sandbox-no-proxy` | unset | Comma-separated `NO_PROXY` list (hostnames, domain suffixes, IPs, CIDRs, each with an optional `:port` qualifier) dialed directly instead of through the corporate proxy. IP/CIDR entries also match hostnames through their validated DNS resolution. | +| `OPENSHELL_SANDBOX_PROXY_AUTH_FILE` | `--sandbox-proxy-auth-file` | unset | Path to a file containing the proxy credentials as `user:pass`. Staged as a root-only Podman secret so credentials never appear in config or container metadata. Requires the insecure-auth acknowledgement below. | +| `OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE` | `--sandbox-proxy-auth-allow-insecure` | unset | Explicit acknowledgement (`true`) that the credential is sent as cleartext Basic auth over the plain-TCP connection to the `http://` proxy. Required when the auth file is set; rejected when it is not. | +| `OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME` | `--sandbox-proxy-connect-by-hostname` | unset | Send the destination hostname in CONNECT requests instead of a validated IP. Last resort for proxies whose ACLs filter on hostnames: the proxy then resolves the name itself, so sandbox SSRF/`allowed_ips` validation no longer binds the connection. | + +Through the gateway, the same settings are the `https_proxy`, `no_proxy`, +`proxy_auth_file`, `proxy_auth_allow_insecure`, and +`proxy_connect_by_hostname` keys under `[openshell.drivers.podman]`; see +`docs/reference/gateway-config.mdx`. + +This is an operator-owned egress boundary: the driver passes the settings on +the supervisor's command line, so sandbox and template environment — and any +`ENV` baked into the sandbox image — cannot override them, and the +conventional `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` variables a sandbox +controls do not steer it. Credentials must be supplied through +`proxy_auth_file`; an inline `user:pass@` in the URL is rejected at startup. + +Basic auth over an `http://` proxy is cleartext on the wire: anyone on the +network path between the sandbox host and the proxy can recover the +credential. Setting `proxy_auth_file` therefore requires +`proxy_auth_allow_insecure = true`; both the driver and the in-container +supervisor reject credentials without that explicit acknowledgement. + +CONNECT requests target a validated resolved IP by default, so the proxy +performs no DNS resolution and the tunnel stays bound to the address that +passed the sandbox's SSRF and `allowed_ips` checks; the hostname still +travels inside the tunnel (TLS SNI, application `Host`). In split-horizon +networks, point the gateway host at the corporate resolver. Set +`proxy_connect_by_hostname = true` only when the proxy's ACLs filter on +hostnames and reject IP CONNECT targets — it re-opens proxy-side DNS +resolution, making the proxy's ACLs the effective egress control. ## Rootless-Specific Adaptations diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index def8e5f3dc..ba193e8e1c 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -69,10 +69,9 @@ impl FromStr for ImagePullPolicy { #[derive(Clone, serde::Serialize, serde::Deserialize)] #[serde(default, deny_unknown_fields)] pub struct PodmanComputeConfig { - /// Path to the Podman API Unix socket. - /// Default: `$XDG_RUNTIME_DIR/podman/podman.sock` (Linux), - /// `$HOME/.local/share/containers/podman/machine/podman.sock` (macOS). - pub socket_path: PathBuf, + /// Podman API Unix socket. When unset, use the socket selected by + /// gateway auto-detection. + pub socket_path: Option, /// Default OCI image for sandboxes. pub default_image: String, /// Image pull policy for sandbox images. @@ -134,6 +133,57 @@ pub struct PodmanComputeConfig { /// Set to `0` to disable health checks entirely. /// Defaults to [`DEFAULT_HEALTH_CHECK_INTERVAL_SECS`] (10 seconds). pub health_check_interval_secs: u64, + /// Corporate forward proxy URL passed to the in-container supervisor + /// (e.g. `http://proxy.corp.com:8080`). + /// + /// The supervisor chains policy-approved TLS tunnels through this proxy + /// with HTTP CONNECT instead of dialing upstream destinations directly. + /// Only `http://` proxy URLs in explicit `http://host:port` form (scheme + /// and port required) are supported. This is an operator-owned egress + /// boundary delivered on the supervisor's command line, so + /// sandbox/template environment cannot override it, and the conventional + /// `HTTPS_PROXY` variables are not used. + pub https_proxy: Option, + /// Comma-separated `NO_PROXY` list passed alongside the proxy URL (e.g. + /// `*.svc.cluster.local,10.0.0.0/8`). Destinations matching an entry are + /// dialed directly instead of through the corporate proxy. Entries take + /// an optional `:port` qualifier that limits them to that destination + /// port, and IP/CIDR entries also match hostnames through their + /// validated DNS resolution. + pub no_proxy: Option, + /// Path (on the gateway host) to a file containing the corporate proxy + /// credentials as `user:pass`. + /// + /// Credentials must be supplied through this file, never embedded in the + /// proxy URL: an inline `user:pass@` in `https_proxy` is + /// rejected at startup because it would leak into `gateway.toml` and + /// container metadata. The gateway reads this file at sandbox-create time + /// and delivers it to the supervisor through a root-only secret mount. + pub proxy_auth_file: Option, + /// Explicit acknowledgement that proxy credentials are sent in cleartext. + /// + /// `Proxy-Authorization: Basic` is base64, not encryption, and the + /// connection to an `http://` corporate proxy is plain TCP, so anyone on + /// the network path between the sandbox host and the proxy can recover + /// the credential. Setting `proxy_auth_file` therefore requires + /// `proxy_auth_allow_insecure = true`; without it the configuration is + /// rejected at startup. Set it only when the path to the proxy is a + /// trusted network segment. + pub proxy_auth_allow_insecure: Option, + /// Send the destination *hostname* in CONNECT requests to the corporate + /// proxy instead of a validated IP. + /// + /// By default the supervisor CONNECTs to an address that already passed + /// SSRF and `allowed_ips` validation, so the proxy performs no DNS + /// resolution and the tunnel stays bound to the validated answer. Set + /// this to `true` only when the proxy's ACLs filter on hostnames and + /// reject IP CONNECT targets: the proxy then resolves the name itself, + /// so a name that resolves differently at the proxy (split-horizon DNS, + /// rebinding) can reach destinations the sandbox policy never approved, + /// and the proxy's own ACLs become the effective egress control. Prefer + /// pointing the gateway host at the corporate resolver so validated-IP + /// CONNECT works in split-horizon networks. + pub proxy_connect_by_hostname: Option, } pub const DEFAULT_HEALTH_CHECK_INTERVAL_SECS: u64 = 10; @@ -189,6 +239,94 @@ impl PodmanComputeConfig { Ok(()) } + /// Validate optional corporate proxy configuration. + /// + /// Shares validation semantics with the in-container supervisor through + /// [`openshell_core::driver_utils::parse_upstream_proxy_url`], so a value + /// accepted here can never be rejected by the supervisor at sandbox + /// startup (or vice versa). The supervisor only supports `http://` + /// forward proxies, so other schemes are rejected at config time instead + /// of failing inside every sandbox. Credentials must be supplied through + /// `proxy_auth_file`; an inline `user:pass@` in the URL is rejected + /// because it would otherwise be stored in `gateway.toml` and exposed in + /// container metadata. + pub fn validate_proxy_config(&self) -> Result<(), crate::client::PodmanApiError> { + use openshell_core::driver_utils::{UpstreamProxyUrlError, parse_upstream_proxy_url}; + if let Some(url) = &self.https_proxy { + parse_upstream_proxy_url(url).map_err(|err| { + crate::client::PodmanApiError::InvalidInput(match err { + UpstreamProxyUrlError::Empty => { + "https_proxy must not be empty when set".to_string() + } + UpstreamProxyUrlError::InlineCredentials => { + "https_proxy must not embed credentials in the URL; supply them via \ + proxy_auth_file so they are not stored in config or container metadata" + .to_string() + } + err => format!("https_proxy {err}"), + }) + })?; + } + + // The supervisor treats a present-but-empty driver-supplied argument + // as a fatal misconfiguration, so never accept (and later pass) one. + if let Some(list) = self.no_proxy.as_deref() { + if list.trim().is_empty() { + return Err(crate::client::PodmanApiError::InvalidInput( + "no_proxy must not be empty when set; omit it instead".to_string(), + )); + } + // A bypass list only makes sense relative to a proxy boundary. An + // operator who set one believed proxying was in effect, so accepting + // it while all egress dials directly would hide a fail-open state. + if self.https_proxy.is_none() { + return Err(crate::client::PodmanApiError::InvalidInput( + "no_proxy is set but no https_proxy is configured".to_string(), + )); + } + } + + if let Some(path) = self.proxy_auth_file.as_deref() { + if path.trim().is_empty() { + return Err(crate::client::PodmanApiError::InvalidInput( + "proxy_auth_file must not be empty when set".to_string(), + )); + } + if self.https_proxy.is_none() { + return Err(crate::client::PodmanApiError::InvalidInput( + "proxy_auth_file is set but no https_proxy is configured".to_string(), + )); + } + // Basic auth over the plain-TCP proxy connection is readable by + // anyone on the network path; sending it requires an explicit + // operator acknowledgement rather than being an implicit side + // effect of configuring credentials. + if self.proxy_auth_allow_insecure != Some(true) { + return Err(crate::client::PodmanApiError::InvalidInput( + "proxy_auth_file sends the credential as cleartext Basic auth over the \ + plain-TCP connection to the http:// proxy; set proxy_auth_allow_insecure \ + = true to accept that exposure, or remove proxy_auth_file" + .to_string(), + )); + } + } else if self.proxy_auth_allow_insecure.is_some() { + // The acknowledgement without credentials means the operator + // believed an auth file was configured; surface the mismatch. + return Err(crate::client::PodmanApiError::InvalidInput( + "proxy_auth_allow_insecure is set but no proxy_auth_file is configured".to_string(), + )); + } + + // The CONNECT-target mode only means something relative to a proxy + // boundary the operator believed was in effect. + if self.proxy_connect_by_hostname.is_some() && self.https_proxy.is_none() { + return Err(crate::client::PodmanApiError::InvalidInput( + "proxy_connect_by_hostname is set but no https_proxy is configured".to_string(), + )); + } + Ok(()) + } + /// Validate optional host gateway override. pub fn validate_host_gateway_ip(&self) -> Result<(), crate::client::PodmanApiError> { let trimmed = self.host_gateway_ip.trim(); @@ -215,38 +353,12 @@ impl PodmanComputeConfig { String::new() } } - - /// Resolve the default socket path from the environment. - /// - /// - **macOS**: `$HOME/.local/share/containers/podman/machine/podman.sock` - /// (the symlink created by `podman machine` pointing to the VM API socket). - /// - **Linux**: `$XDG_RUNTIME_DIR/podman/podman.sock` when set (by - /// `pam_systemd`/logind), otherwise `/run/user/{uid}/podman/podman.sock` - /// using the real UID via `getuid()`. - #[must_use] - pub fn default_socket_path() -> PathBuf { - #[cfg(target_os = "macos")] - { - let home = std::env::var("HOME").expect("HOME must be set on macOS"); - PathBuf::from(home).join(".local/share/containers/podman/machine/podman.sock") - } - #[cfg(target_os = "linux")] - { - std::env::var("XDG_RUNTIME_DIR").map_or_else( - |_| { - let uid = rustix::process::getuid().as_raw(); - PathBuf::from(format!("/run/user/{uid}/podman/podman.sock")) - }, - |xdg| PathBuf::from(xdg).join("podman/podman.sock"), - ) - } - } } impl Default for PodmanComputeConfig { fn default() -> Self { Self { - socket_path: Self::default_socket_path(), + socket_path: None, default_image: openshell_core::image::default_sandbox_image(), image_pull_policy: ImagePullPolicy::default(), grpc_endpoint: String::new(), @@ -262,6 +374,11 @@ impl Default for PodmanComputeConfig { sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT, enable_bind_mounts: false, health_check_interval_secs: DEFAULT_HEALTH_CHECK_INTERVAL_SECS, + https_proxy: None, + no_proxy: None, + proxy_auth_file: None, + proxy_auth_allow_insecure: None, + proxy_connect_by_hostname: None, } } } @@ -288,6 +405,12 @@ impl std::fmt::Debug for PodmanComputeConfig { "health_check_interval_secs", &self.health_check_interval_secs, ) + // Proxy URLs may embed credentials in userinfo; log presence only. + .field("https_proxy", &self.https_proxy.is_some()) + .field("no_proxy", &self.no_proxy) + .field("proxy_auth_file", &self.proxy_auth_file.is_some()) + .field("proxy_auth_allow_insecure", &self.proxy_auth_allow_insecure) + .field("proxy_connect_by_hostname", &self.proxy_connect_by_hostname) .finish() } } @@ -296,39 +419,6 @@ impl std::fmt::Debug for PodmanComputeConfig { mod tests { use super::*; - /// Serialises env-mutating tests so that parallel test threads cannot - /// observe each other's changes to `XDG_RUNTIME_DIR`. - static ENV_LOCK: std::sync::LazyLock> = - std::sync::LazyLock::new(|| std::sync::Mutex::new(())); - - #[test] - #[cfg(target_os = "linux")] - fn default_socket_path_respects_xdg_runtime_dir() { - let _guard = ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - temp_env::with_vars([("XDG_RUNTIME_DIR", Some("/tmp/test-xdg"))], || { - let path = PodmanComputeConfig::default_socket_path(); - assert_eq!(path, PathBuf::from("/tmp/test-xdg/podman/podman.sock")); - }); - } - - #[test] - #[cfg(target_os = "linux")] - fn default_socket_path_falls_back_to_uid() { - let _guard = ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - temp_env::with_vars([("XDG_RUNTIME_DIR", None::<&str>)], || { - let path = PodmanComputeConfig::default_socket_path(); - let uid = rustix::process::getuid().as_raw(); - assert_eq!( - path, - PathBuf::from(format!("/run/user/{uid}/podman/podman.sock")) - ); - }); - } - #[test] fn default_config_sets_health_check_interval() { let cfg = PodmanComputeConfig::default(); @@ -382,19 +472,208 @@ mod tests { assert!(err.to_string().contains("sandbox_pids_limit")); } + // ── Proxy config validation ─────────────────────────────────────── + #[test] - #[cfg(target_os = "macos")] - fn default_socket_path_uses_podman_machine_on_macos() { - let _guard = ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - temp_env::with_vars([("HOME", Some("/Users/testuser"))], || { - let path = PodmanComputeConfig::default_socket_path(); - assert_eq!( - path, - PathBuf::from("/Users/testuser/.local/share/containers/podman/machine/podman.sock") + fn validate_proxy_config_accepts_unset_and_http() { + assert!( + PodmanComputeConfig::default() + .validate_proxy_config() + .is_ok() + ); + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + no_proxy: Some("*.svc.cluster.local".to_string()), + ..PodmanComputeConfig::default() + }; + assert!(cfg.validate_proxy_config().is_ok()); + } + + #[test] + fn validate_proxy_config_rejects_non_http_schemes() { + for url in ["https://proxy:443", "socks5://proxy:1080"] { + let cfg = PodmanComputeConfig { + https_proxy: Some(url.to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!( + err.to_string().contains("unsupported proxy scheme"), + "{url}: {err}" ); - }); + } + } + + #[test] + fn validate_proxy_config_rejects_url_components() { + for url in [ + "http://proxy.corp.com:8080/path", + "http://proxy.corp.com:8080?x=1", + "http://proxy.corp.com:8080#frag", + ] { + let cfg = PodmanComputeConfig { + https_proxy: Some(url.to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!( + err.to_string().contains("scheme://host:port"), + "{url}: {err}" + ); + } + } + + #[test] + fn validate_proxy_config_rejects_zero_port() { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:0".to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("port must not be 0"), "{err}"); + } + + #[test] + fn validate_proxy_config_rejects_missing_scheme_or_port() { + // A scheme-less value (previously normalized to http://) and a + // port-less value (previously defaulted to 80) are both rejected so + // gateway.toml matches the documented http://host:port grammar. + for url in ["proxy.corp.com:8080", "http://proxy.corp.com"] { + let cfg = PodmanComputeConfig { + https_proxy: Some(url.to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("explicit"), "{url}: {err}"); + } + } + + #[test] + fn validate_proxy_config_rejects_empty_value() { + let cfg = PodmanComputeConfig { + https_proxy: Some(" ".to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("https_proxy"), "{err}"); + } + + #[test] + fn validate_proxy_config_rejects_empty_no_proxy() { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + no_proxy: Some(" ".to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("no_proxy"), "{err}"); + } + + #[test] + fn validate_proxy_config_rejects_no_proxy_without_proxy() { + let cfg = PodmanComputeConfig { + no_proxy: Some("*.svc.cluster.local".to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("no_proxy"), "{err}"); + } + + #[test] + fn validate_proxy_config_rejects_inline_credentials() { + for url in [ + "http://user:pass@proxy.corp.com:8080", + "http://user@proxy.corp.com:8080", + ] { + let cfg = PodmanComputeConfig { + https_proxy: Some(url.to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!( + err.to_string().contains("proxy_auth_file"), + "{url} should be rejected and point at proxy_auth_file: {err}" + ); + } + } + + #[test] + fn validate_proxy_config_accepts_auth_file_with_proxy_and_acknowledgement() { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + proxy_auth_file: Some("/etc/openshell/secrets/proxy-auth".to_string()), + proxy_auth_allow_insecure: Some(true), + ..PodmanComputeConfig::default() + }; + assert!(cfg.validate_proxy_config().is_ok()); + } + + #[test] + fn validate_proxy_config_rejects_auth_file_without_insecure_acknowledgement() { + // Basic auth over the plain-TCP proxy connection is readable on the + // network path; sending it must be an explicit operator decision. + for allow in [None, Some(false)] { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + proxy_auth_file: Some("/etc/openshell/secrets/proxy-auth".to_string()), + proxy_auth_allow_insecure: allow, + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!( + err.to_string().contains("proxy_auth_allow_insecure"), + "{allow:?}: {err}" + ); + assert!(err.to_string().contains("cleartext"), "{allow:?}: {err}"); + } + } + + #[test] + fn validate_proxy_config_accepts_connect_by_hostname_with_proxy() { + for by_hostname in [Some(true), Some(false)] { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + proxy_connect_by_hostname: by_hostname, + ..PodmanComputeConfig::default() + }; + assert!(cfg.validate_proxy_config().is_ok(), "{by_hostname:?}"); + } + } + + #[test] + fn validate_proxy_config_rejects_connect_by_hostname_without_proxy() { + let cfg = PodmanComputeConfig { + proxy_connect_by_hostname: Some(true), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("no https_proxy"), "{err}"); + } + + #[test] + fn validate_proxy_config_rejects_acknowledgement_without_auth_file() { + for allow in [Some(true), Some(false)] { + let cfg = PodmanComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + proxy_auth_allow_insecure: allow, + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!( + err.to_string().contains("no proxy_auth_file"), + "{allow:?}: {err}" + ); + } + } + + #[test] + fn validate_proxy_config_rejects_auth_file_without_proxy() { + let cfg = PodmanComputeConfig { + proxy_auth_file: Some("/etc/openshell/secrets/proxy-auth".to_string()), + ..PodmanComputeConfig::default() + }; + let err = cfg.validate_proxy_config().unwrap_err(); + assert!(err.to_string().contains("proxy_auth_file"), "{err}"); } // ── TLS config validation ───────────────────────────────────────── diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index d38a3b9eb1..e417358c6e 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -53,12 +53,15 @@ const VOLUME_PREFIX: &str = "openshell-sandbox-"; /// Secret name prefix for per-sandbox gateway JWTs. const TOKEN_SECRET_PREFIX: &str = "openshell-token-"; +const PROXY_AUTH_SECRET_PREFIX: &str = "openshell-proxy-auth-"; /// Container-side mount paths for client TLS materials and the sandbox token. const TLS_CA_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CA_MOUNT_PATH; const TLS_CERT_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CERT_MOUNT_PATH; const TLS_KEY_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_KEY_MOUNT_PATH; const SANDBOX_TOKEN_MOUNT_PATH: &str = openshell_core::driver_utils::SANDBOX_TOKEN_MOUNT_PATH; +const UPSTREAM_PROXY_AUTH_MOUNT_PATH: &str = + openshell_core::driver_utils::UPSTREAM_PROXY_AUTH_MOUNT_PATH; /// Directory inside sandbox containers where the supervisor binary is mounted. const SUPERVISOR_MOUNT_DIR: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_DIR; @@ -162,6 +165,12 @@ pub fn token_secret_name(sandbox_id: &str) -> String { format!("{TOKEN_SECRET_PREFIX}{sandbox_id}") } +/// Build the per-sandbox Podman secret name for the corporate proxy credentials. +#[must_use] +pub fn proxy_auth_secret_name(sandbox_id: &str) -> String { + format!("{PROXY_AUTH_SECRET_PREFIX}{sandbox_id}") +} + /// Truncate a container ID to 12 characters (standard short form). #[must_use] pub fn short_id(id: &str) -> String { @@ -349,6 +358,41 @@ pub fn resolve_image<'a>(sandbox: &'a DriverSandbox, config: &'a PodmanComputeCo /// User-supplied vars are inserted first so that the required driver /// vars always win -- preventing spec/template overrides of security- /// critical values like `OPENSHELL_ENDPOINT` or `OPENSHELL_SANDBOX_ID`. +/// Build the corporate upstream-proxy command-line arguments passed to the +/// supervisor. +/// +/// This operator-owned egress boundary travels on argv, which sandbox +/// spec/template environment and image `ENV` cannot influence. Credentials +/// are never on argv — only the root-only mount path is passed; the +/// supervisor reads the secret from the mount. +fn upstream_proxy_cli_args(config: &PodmanComputeConfig) -> Vec { + let mut args = Vec::new(); + if let Some(url) = &config.https_proxy { + args.push("--upstream-proxy".to_string()); + args.push(url.clone()); + } + if let Some(list) = &config.no_proxy { + args.push("--upstream-no-proxy".to_string()); + args.push(list.clone()); + } + if config.proxy_auth_file.is_some() { + args.push("--upstream-proxy-auth-file".to_string()); + args.push(UPSTREAM_PROXY_AUTH_MOUNT_PATH.to_string()); + } + // Config validation guarantees the acknowledgement is `true` whenever an + // auth file is configured; the supervisor independently refuses + // credentials without it. + if config.proxy_auth_allow_insecure == Some(true) { + args.push("--upstream-proxy-auth-allow-insecure".to_string()); + } + // Absent means the default validated-IP CONNECT binding; only the + // explicit hostname opt-in is passed through. + if config.proxy_connect_by_hostname == Some(true) { + args.push("--upstream-proxy-connect-by-hostname".to_string()); + } + args +} + fn build_env( sandbox: &DriverSandbox, config: &PodmanComputeConfig, @@ -386,6 +430,12 @@ fn build_env( } // 2. Required driver vars (highest priority -- always overwrite). + + // The operator's corporate egress proxy settings are not environment + // variables: they travel on the supervisor's argv (see + // `upstream_proxy_cli_args`), which sandbox spec/template environment + // and image ENV cannot influence. + env.insert( openshell_core::sandbox_env::SANDBOX.into(), sandbox.name.clone(), @@ -900,7 +950,10 @@ pub fn build_container_spec_with_token_and_gpu_devices( // Without this, the container would run the entrypoint binary with // the supervisor path as an argument instead of executing it directly. entrypoint: vec![SUPERVISOR_BINARY_PATH.into()], - command: vec![], + // Operator-owned corporate proxy flags. The workload command is not + // part of argv (the supervisor takes it from the reserved command + // env var), so these flags are the whole command list. + command: upstream_proxy_cli_args(config), // Force the supervisor to run as root (UID 0). Sandbox images may // set a non-root USER directive (e.g. `USER sandbox`), but the // supervisor needs root to create network namespaces, set up the @@ -994,15 +1047,32 @@ pub fn build_container_spec_with_token_and_gpu_devices( }, resource_limits, secret_env: BTreeMap::new(), - secrets: token_secret_name.map_or_else(Vec::new, |source| { - vec![SecretMount { - source: source.to_string(), - target: SANDBOX_TOKEN_MOUNT_PATH.into(), - uid: 0, - gid: 0, - mode: 0o400, - }] - }), + secrets: { + let mut secrets = Vec::new(); + if let Some(source) = token_secret_name { + secrets.push(SecretMount { + source: source.to_string(), + target: SANDBOX_TOKEN_MOUNT_PATH.into(), + uid: 0, + gid: 0, + mode: 0o400, + }); + } + // Corporate proxy credentials, when configured, are mounted as a + // root-only secret. The driver creates a matching Podman secret + // (see `create_sandbox_proxy_auth_secret`) named deterministically + // from the sandbox id, so no name needs threading through here. + if config.proxy_auth_file.is_some() { + secrets.push(SecretMount { + source: proxy_auth_secret_name(&sandbox.id), + target: UPSTREAM_PROXY_AUTH_MOUNT_PATH.into(), + uid: 0, + gid: 0, + mode: 0o400, + }); + } + secrets + }, stop_timeout: config.stop_timeout_secs, // Inject stable host aliases into /etc/hosts so sandbox containers can // reach services on the host. `host.openshell.internal` is the driver- @@ -1650,6 +1720,125 @@ mod tests { ); } + /// Extract the container spec's supervisor argv (`command`) as strings. + fn spec_command(spec: &Value) -> Vec { + spec["command"] + .as_array() + .expect("command should be an array") + .iter() + .map(|v| { + v.as_str() + .expect("command arg should be a string") + .to_string() + }) + .collect() + } + + #[test] + fn container_spec_passes_operator_proxy_on_supervisor_argv() { + let sandbox = test_sandbox("test-id", "test-name"); + let mut config = test_config(); + config.https_proxy = Some("http://proxy.corp.com:8080".to_string()); + config.no_proxy = Some("*.svc.cluster.local,10.0.0.0/8".to_string()); + + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + + // Config travels on argv (the image cannot forge process arguments), + // as flag/value pairs. + let idx = command + .iter() + .position(|a| a == "--upstream-proxy") + .expect("proxy URL flag present"); + assert_eq!( + command.get(idx + 1).map(String::as_str), + Some("http://proxy.corp.com:8080") + ); + let idx = command + .iter() + .position(|a| a == "--upstream-no-proxy") + .expect("no_proxy flag present"); + assert_eq!( + command.get(idx + 1).map(String::as_str), + Some("*.svc.cluster.local,10.0.0.0/8") + ); + + // The proxy settings are argv-only; nothing about them lands in the + // environment, and the conventional proxy variables (which belong to + // the sandbox creator) are not touched by operator config. + let env_map = spec["env"].as_object().expect("env should be an object"); + for key in ["HTTPS_PROXY", "https_proxy", "HTTP_PROXY", "http_proxy"] { + assert!( + !env_map.contains_key(key), + "{key} must not be populated from operator proxy config" + ); + } + } + + #[test] + fn container_spec_omits_proxy_argv_when_unconfigured() { + let sandbox = test_sandbox("test-id", "test-name"); + let spec = build_container_spec(&sandbox, &test_config()); + let command = spec_command(&spec); + + assert!( + !command.iter().any(|a| a.starts_with("--upstream-proxy")), + "no proxy flags without operator proxy config: {command:?}" + ); + } + + #[test] + fn container_spec_sandbox_env_cannot_influence_proxy_argv() { + use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; + + // A sandbox creator tries to steer the egress boundary through spec + // and template environment (image-baked ENV behaves the same at the + // runtime layer). The supervisor takes proxy config only from the + // argv the driver builds out of operator config, so none of it has + // any effect. + let mut sandbox = test_sandbox("test-id", "test-name"); + sandbox.spec = Some(DriverSandboxSpec { + environment: std::collections::HashMap::from([ + ( + "HTTPS_PROXY".to_string(), + "http://attacker:9999".to_string(), + ), + ("NO_PROXY".to_string(), "*".to_string()), + ]), + template: Some(DriverSandboxTemplate { + environment: std::collections::HashMap::from([( + "NO_PROXY".to_string(), + "*".to_string(), + )]), + ..Default::default() + }), + ..Default::default() + }); + let mut config = test_config(); + config.https_proxy = Some("http://proxy.corp.com:8080".to_string()); + + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + + // Only the operator's proxy is delivered, and only on argv. + let idx = command + .iter() + .position(|a| a == "--upstream-proxy") + .expect("operator proxy flag present"); + assert_eq!( + command.get(idx + 1).map(String::as_str), + Some("http://proxy.corp.com:8080") + ); + assert!( + !command.iter().any(|a| a == "--upstream-no-proxy"), + "sandbox environment must not add a NO_PROXY bypass: {command:?}" + ); + assert!( + !command.iter().any(|a| a.contains("attacker")), + "attacker proxy must not reach argv: {command:?}" + ); + } + #[test] fn container_spec_required_labels_cannot_be_overridden() { use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; @@ -1789,7 +1978,7 @@ mod tests { fn test_config() -> PodmanComputeConfig { PodmanComputeConfig { - socket_path: std::path::PathBuf::from("/tmp/test.sock"), + socket_path: Some(std::path::PathBuf::from("/tmp/test.sock")), default_image: "test-image:latest".to_string(), grpc_endpoint: "http://localhost:50051".to_string(), host_gateway_ip: String::new(), @@ -2390,6 +2579,104 @@ mod tests { ); } + #[test] + fn container_spec_proxy_auth_file_mounts_secret_and_sets_path_only() { + let sandbox = test_sandbox("proxy-id", "proxy-name"); + let mut config = test_config(); + config.https_proxy = Some("http://proxy.corp.com:8080".to_string()); + config.proxy_auth_file = Some("/etc/openshell/secrets/proxy-auth".to_string()); + config.proxy_auth_allow_insecure = Some(true); + + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + + // The supervisor gets only the mount *path* on argv, never the + // credential itself. + let idx = command + .iter() + .position(|a| a == "--upstream-proxy-auth-file") + .expect("auth-file flag present"); + assert_eq!( + command.get(idx + 1).map(String::as_str), + Some(UPSTREAM_PROXY_AUTH_MOUNT_PATH) + ); + // The cleartext-credential acknowledgement travels with the auth + // file so the supervisor's fail-closed pairing check passes. + assert!( + command + .iter() + .any(|a| a == "--upstream-proxy-auth-allow-insecure"), + "acknowledgement flag present: {command:?}" + ); + // The raw credential path from config never appears anywhere in the + // spec (only the fixed mount path does). + assert!( + !command + .iter() + .any(|a| a == "/etc/openshell/secrets/proxy-auth"), + "host-side credential path must not reach the container: {command:?}" + ); + + let secrets = spec["secrets"] + .as_array() + .expect("secrets should be an array"); + assert!( + secrets.iter().any(|secret| { + secret["source"].as_str() == Some(proxy_auth_secret_name(&sandbox.id).as_str()) + && secret["target"].as_str() == Some(UPSTREAM_PROXY_AUTH_MOUNT_PATH) + && secret["mode"].as_u64() == Some(0o400) + }), + "proxy credentials must be delivered through a root-only secret mount" + ); + } + + #[test] + fn container_spec_omits_proxy_auth_mount_when_unconfigured() { + let sandbox = test_sandbox("proxy-id", "proxy-name"); + let mut config = test_config(); + config.https_proxy = Some("http://proxy.corp.com:8080".to_string()); + + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + assert!( + !command.iter().any(|a| a == "--upstream-proxy-auth-file"), + "auth-file flag must be absent when no proxy_auth_file is configured: {command:?}" + ); + assert!( + !command + .iter() + .any(|a| a == "--upstream-proxy-auth-allow-insecure"), + "acknowledgement flag must be absent when no proxy_auth_file is configured: {command:?}" + ); + } + + #[test] + fn container_spec_connect_by_hostname_passed_only_on_opt_in() { + let sandbox = test_sandbox("proxy-id", "proxy-name"); + let mut config = test_config(); + config.https_proxy = Some("http://proxy.corp.com:8080".to_string()); + + // Default: no flag, the supervisor uses validated-IP CONNECT binding. + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + assert!( + !command + .iter() + .any(|a| a == "--upstream-proxy-connect-by-hostname"), + "hostname CONNECT must be absent without the operator opt-in: {command:?}" + ); + + config.proxy_connect_by_hostname = Some(true); + let spec = build_container_spec(&sandbox, &config); + let command = spec_command(&spec); + assert!( + command + .iter() + .any(|a| a == "--upstream-proxy-connect-by-hostname"), + "hostname CONNECT flag present on opt-in: {command:?}" + ); + } + #[test] fn container_spec_omits_tls_without_config() { let sandbox = test_sandbox("notls-id", "notls-name"); diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index d59cbb9e1d..3878f59836 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -19,7 +19,7 @@ use openshell_core::gpu::{ use openshell_core::proto::compute::v1::{ DriverSandbox, GetCapabilitiesResponse, GpuResourceRequirements, }; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::Duration; use tracing::{debug, info, warn}; @@ -114,6 +114,57 @@ async fn cleanup_sandbox_token_secret(client: &PodmanClient, secret_name: &str) } } +/// Read the operator's proxy credentials file and stage it as a per-sandbox +/// Podman secret, so the credentials reach the supervisor through a root-only +/// mount rather than the container environment. +/// +/// Fails closed: when `proxy_auth_file` is configured but cannot be read or +/// does not hold a valid `user:pass` credential, the sandbox is not created. +/// Credential validation is shared with the in-container supervisor through +/// [`openshell_core::driver_utils::parse_upstream_proxy_credential`], so a +/// credential staged here can never be rejected at supervisor startup. +async fn create_sandbox_proxy_auth_secret( + client: &PodmanClient, + config: &PodmanComputeConfig, + sandbox: &DriverSandbox, +) -> Result, ComputeDriverError> { + let Some(path) = config.proxy_auth_file.as_deref() else { + return Ok(None); + }; + + // Bounded, blocking read shared with the supervisor: rejects non-regular + // files (e.g. /dev/zero) and caps the size so a hostile path cannot + // exhaust gateway memory. + let path_owned = path.to_string(); + let raw = tokio::task::spawn_blocking(move || { + openshell_core::driver_utils::read_upstream_proxy_credential_file(&path_owned) + }) + .await + .map_err(|e| ComputeDriverError::Message(format!("proxy_auth_file read task failed: {e}")))? + .map_err(ComputeDriverError::Message)?; + let credential = + openshell_core::driver_utils::parse_upstream_proxy_credential(&raw).map_err(|err| { + ComputeDriverError::InvalidArgument(format!("proxy_auth_file '{path}': {err}")) + })?; + + let secret_name = container::proxy_auth_secret_name(&sandbox.id); + client + .create_secret(&secret_name, format!("{credential}\n").as_bytes()) + .await + .map_err(ComputeDriverError::from)?; + Ok(Some(secret_name)) +} + +async fn cleanup_sandbox_proxy_auth_secret(client: &PodmanClient, secret_name: &str) { + if let Err(err) = client.remove_secret(secret_name).await { + warn!( + secret = %secret_name, + error = %err, + "Failed to remove Podman sandbox proxy-auth secret" + ); + } +} + fn local_podman_cdi_gpu_inventory_from(dev_root: &Path) -> CdiGpuInventory { let mut device_ids = std::fs::read_dir(dev_root) .ok() @@ -157,22 +208,48 @@ fn podman_gpu_selection_error(err: CdiGpuSelectionError) -> ComputeDriverError { ComputeDriverError::Precondition(err.to_string()) } +/// Resolve the socket to connect to: explicit configuration wins, otherwise +/// fall back to `detect`. Returns an error if neither resolves. +/// +/// Takes `detect` as a parameter (rather than calling +/// [`openshell_core::config::detect_podman_socket`] directly) so tests can +/// exercise the precedence deterministically, without touching real +/// environment variables or the filesystem. +fn resolve_socket_path( + configured: Option, + detect: impl FnOnce() -> Option, +) -> Result { + configured.or_else(detect).ok_or_else(|| { + PodmanApiError::InvalidInput( + "no responsive Podman API socket found; set OPENSHELL_PODMAN_SOCKET \ + or configure socket_path" + .to_string(), + ) + }) +} + impl PodmanComputeDriver { /// Create a new driver, verifying the Podman socket is reachable. pub async fn new(mut config: PodmanComputeConfig) -> Result { const MAX_PING_RETRIES: u32 = 5; const PING_RETRY_DELAY: Duration = Duration::from_secs(2); - if !config.socket_path.exists() { + let socket_path = resolve_socket_path( + config.socket_path.clone(), + openshell_core::config::detect_podman_socket, + )?; + config.socket_path = Some(socket_path.clone()); + + if !socket_path.exists() { if cfg!(target_os = "macos") { warn!( - path = %config.socket_path.display(), + path = %socket_path.display(), "Podman socket not found; is podman machine running? \ Try `podman machine start` or set OPENSHELL_PODMAN_SOCKET to override." ); } else { warn!( - path = %config.socket_path.display(), + path = %socket_path.display(), "Podman socket not found; is the Podman service running? \ Set OPENSHELL_PODMAN_SOCKET or XDG_RUNTIME_DIR to override." ); @@ -185,8 +262,9 @@ impl PodmanComputeDriver { config.validate_tls_config()?; config.validate_runtime_limits()?; config.validate_host_gateway_ip()?; + config.validate_proxy_config()?; - let client = PodmanClient::new(config.socket_path.clone()); + let client = PodmanClient::new(socket_path); // Verify connectivity, retrying briefly to tolerate transient socket // unavailability (e.g. podman.socket restarting after a package @@ -516,6 +594,29 @@ impl PodmanComputeDriver { return Err(e); } }; + let proxy_auth_secret_name = + match create_sandbox_proxy_auth_secret(&self.client, &self.config, sandbox).await { + Ok(name) => name, + Err(e) => { + let _ = self.client.remove_volume(&vol_name).await; + if let Some(secret) = token_secret_name.as_deref() { + cleanup_sandbox_token_secret(&self.client, secret).await; + } + return Err(e); + } + }; + + // Clean up the volume and both per-sandbox secrets on any failure past + // this point. + let cleanup_created = || async { + let _ = self.client.remove_volume(&vol_name).await; + if let Some(secret) = token_secret_name.as_deref() { + cleanup_sandbox_token_secret(&self.client, secret).await; + } + if let Some(secret) = proxy_auth_secret_name.as_deref() { + cleanup_sandbox_proxy_auth_secret(&self.client, secret).await; + } + }; // 3. Create container. let gpu_devices = match self.resolve_gpu_cdi_devices( @@ -525,10 +626,7 @@ impl PodmanComputeDriver { ) { Ok(devices) => devices, Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { - cleanup_sandbox_token_secret(&self.client, secret).await; - } + cleanup_created().await; return Err(e); } }; @@ -540,10 +638,7 @@ impl PodmanComputeDriver { ) { Ok(spec) => spec, Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { - cleanup_sandbox_token_secret(&self.client, secret).await; - } + cleanup_created().await; return Err(e); } }; @@ -554,17 +649,11 @@ impl PodmanComputeDriver { // sandbox's ID, not the conflicting container's ID (which // has the same name but a different ID), so it would be // orphaned otherwise. - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { - cleanup_sandbox_token_secret(&self.client, secret).await; - } + cleanup_created().await; return Err(ComputeDriverError::AlreadyExists); } Err(e) => { - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { - cleanup_sandbox_token_secret(&self.client, secret).await; - } + cleanup_created().await; return Err(ComputeDriverError::from(e)); } } @@ -577,10 +666,7 @@ impl PodmanComputeDriver { "Failed to start container; cleaning up" ); let _ = self.client.remove_container(&name).await; - let _ = self.client.remove_volume(&vol_name).await; - if let Some(secret) = token_secret_name.as_deref() { - cleanup_sandbox_token_secret(&self.client, secret).await; - } + cleanup_created().await; return Err(ComputeDriverError::from(e)); } @@ -636,6 +722,11 @@ impl PodmanComputeDriver { } cleanup_sandbox_token_secret(&self.client, &container::token_secret_name(sandbox_id)) .await; + cleanup_sandbox_proxy_auth_secret( + &self.client, + &container::proxy_auth_secret_name(sandbox_id), + ) + .await; return Ok(false); }; info!(sandbox_id = %sandbox_id, container = %container_id, "Deleting sandbox container"); @@ -663,6 +754,11 @@ impl PodmanComputeDriver { ); } cleanup_sandbox_token_secret(&self.client, &container::token_secret_name(sandbox_id)).await; + cleanup_sandbox_proxy_auth_secret( + &self.client, + &container::proxy_auth_secret_name(sandbox_id), + ) + .await; Ok(container_existed) } @@ -773,7 +869,7 @@ impl PodmanComputeDriver { gpu_inventory: CdiGpuInventory, allow_all_default_gpu: bool, ) -> Self { - let client = PodmanClient::new(config.socket_path.clone()); + let client = PodmanClient::new(config.socket_path.clone().unwrap_or_default()); let refresh_inventory = gpu_inventory.clone(); Self { client, @@ -847,7 +943,37 @@ mod tests { }; use std::collections::HashMap; use std::fs; - use std::path::PathBuf; + use std::path::{Path, PathBuf}; + + // ── socket resolution ─────────────────────────────────────────────── + // + // These test resolve_socket_path directly with an injected detector, so + // they are deterministic regardless of the host's real environment + // variables or whether a Podman socket happens to be running. + + #[test] + fn resolve_socket_path_prefers_explicit_configuration() { + let path = resolve_socket_path(Some(PathBuf::from("/explicit.sock")), || { + Some(PathBuf::from("/detected.sock")) + }) + .unwrap(); + + assert_eq!(path, PathBuf::from("/explicit.sock")); + } + + #[test] + fn resolve_socket_path_uses_detected_socket_when_unconfigured() { + let path = resolve_socket_path(None, || Some(PathBuf::from("/detected.sock"))).unwrap(); + + assert_eq!(path, PathBuf::from("/detected.sock")); + } + + #[test] + fn resolve_socket_path_errors_when_neither_source_resolves() { + let err = resolve_socket_path(None, || None).unwrap_err(); + + assert!(err.to_string().contains("no responsive Podman API socket")); + } fn cdi_devices_config(device_ids: &[&str]) -> prost_types::Struct { prost_types::Struct { @@ -1244,7 +1370,7 @@ mod tests { fn test_driver(socket_path: PathBuf) -> PodmanComputeDriver { let config = PodmanComputeConfig { - socket_path, + socket_path: Some(socket_path), stop_timeout_secs: 10, ..PodmanComputeConfig::default() }; @@ -1424,7 +1550,7 @@ mod tests { )], ); let config = PodmanComputeConfig { - socket_path: socket_path.clone(), + socket_path: Some(socket_path.clone()), enable_bind_mounts: true, ..PodmanComputeConfig::default() }; @@ -1477,6 +1603,156 @@ mod tests { let _ = fs::remove_file(socket_path); } + /// Write a valid `user:pass` credential to a unique path for proxy-auth + /// secret tests. Caller removes it. + fn write_proxy_auth_file(test_name: &str) -> PathBuf { + let path = crate::test_utils::unique_socket_path(test_name).with_extension("auth"); + fs::write(&path, "user:pass\n").expect("write proxy auth file"); + path + } + + fn proxy_auth_config(socket_path: PathBuf, auth_file: &Path) -> PodmanComputeConfig { + PodmanComputeConfig { + socket_path: Some(socket_path), + stop_timeout_secs: 10, + proxy_auth_file: Some(auth_file.to_string_lossy().into_owned()), + proxy_auth_allow_insecure: Some(true), + ..PodmanComputeConfig::default() + } + } + + fn plain_sandbox(id: &str, name: &str) -> DriverSandbox { + DriverSandbox { + id: id.to_string(), + name: name.to_string(), + namespace: String::new(), + workspace: String::new(), + spec: None, + status: None, + } + } + + fn secret_delete_request(sandbox_id: &str) -> String { + format!( + "DELETE {}", + api_path(&format!( + "/libpod/secrets/{}", + container::proxy_auth_secret_name(sandbox_id) + )) + ) + } + + #[tokio::test] + async fn create_sandbox_removes_proxy_auth_secret_on_container_create_failure() { + // A credential secret is staged before the container is created, so a + // container-create failure must remove it — no credential residue. + let sandbox_id = "sandbox-cc"; + let auth_file = write_proxy_auth_file("create-fail"); + let (socket_path, request_log, handle) = spawn_podman_stub( + "create-container-fail", + vec![ + StubResponse::new(StatusCode::OK, "{}"), // pull supervisor image + StubResponse::new(StatusCode::OK, "{}"), // pull sandbox image + StubResponse::new(StatusCode::CREATED, "{}"), // create volume + StubResponse::new(StatusCode::CREATED, "{}"), // create proxy-auth secret + StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, r#"{"message":"boom"}"#), // create container + StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove volume + StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove proxy-auth secret + ], + ); + let driver = test_driver_with_config(proxy_auth_config(socket_path.clone(), &auth_file)); + + driver + .create_sandbox(&plain_sandbox(sandbox_id, "demo")) + .await + .expect_err("container create should fail"); + + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert!( + requests.contains(&secret_delete_request(sandbox_id)), + "proxy-auth secret must be removed on container-create failure: {requests:?}" + ); + let _ = fs::remove_file(&auth_file); + let _ = fs::remove_file(socket_path); + } + + #[tokio::test] + async fn create_sandbox_removes_proxy_auth_secret_on_start_failure() { + // The container is created but fails to start; the staged credential + // secret must still be removed. + let sandbox_id = "sandbox-sf"; + let auth_file = write_proxy_auth_file("start-fail"); + let (socket_path, request_log, handle) = spawn_podman_stub( + "create-start-fail", + vec![ + StubResponse::new(StatusCode::OK, "{}"), // pull supervisor image + StubResponse::new(StatusCode::OK, "{}"), // pull sandbox image + StubResponse::new(StatusCode::CREATED, "{}"), // create volume + StubResponse::new(StatusCode::CREATED, "{}"), // create proxy-auth secret + StubResponse::new(StatusCode::CREATED, "{}"), // create container + StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, r#"{"message":"boom"}"#), // start container + StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove container + StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove volume + StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove proxy-auth secret + ], + ); + let driver = test_driver_with_config(proxy_auth_config(socket_path.clone(), &auth_file)); + + driver + .create_sandbox(&plain_sandbox(sandbox_id, "demo")) + .await + .expect_err("container start should fail"); + + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert!( + requests.contains(&secret_delete_request(sandbox_id)), + "proxy-auth secret must be removed on start failure: {requests:?}" + ); + let _ = fs::remove_file(&auth_file); + let _ = fs::remove_file(socket_path); + } + + #[tokio::test] + async fn delete_sandbox_removes_proxy_auth_secret() { + // Deleting a sandbox (here already gone out of band) must remove the + // per-sandbox proxy-auth secret so credentials never outlive it. + let sandbox_id = "sandbox-del"; + let (socket_path, request_log, handle) = spawn_podman_stub( + "delete-proxy-auth", + vec![ + StubResponse::new(StatusCode::OK, "[]"), // list_containers (not found) + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove volume + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove token secret + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove proxy-auth secret + ], + ); + let driver = test_driver(socket_path.clone()); + + driver + .delete_sandbox(sandbox_id) + .await + .expect("delete should succeed"); + + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert!( + requests.contains(&secret_delete_request(sandbox_id)), + "proxy-auth secret must be removed on delete: {requests:?}" + ); + let _ = fs::remove_file(socket_path); + } + #[tokio::test] async fn delete_sandbox_finds_container_by_label_and_removes() { let sandbox_id = "sandbox-request-id"; diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index 1f7cce89a4..8e68a91e72 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -170,7 +170,7 @@ mod tests { fn test_service(socket_path: PathBuf) -> ComputeDriverService { let config = PodmanComputeConfig { - socket_path, + socket_path: Some(socket_path), stop_timeout_secs: 10, ..PodmanComputeConfig::default() }; diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index c57aff4277..35191e4cda 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -103,6 +103,36 @@ struct Args { /// Host path to the client private key for sandbox mTLS. #[arg(long, env = "OPENSHELL_PODMAN_TLS_KEY")] podman_tls_key: Option, + + /// Corporate forward proxy URL for the supervisor's upstream TLS dials, + /// in explicit `http://host:port` form (scheme and port required). + /// Credentials must not be embedded in the URL; use + /// `--sandbox-proxy-auth-file` instead. + #[arg(long, env = "OPENSHELL_SANDBOX_HTTPS_PROXY")] + sandbox_https_proxy: Option, + + /// Comma-separated `NO_PROXY` list injected alongside the proxy URL. + #[arg(long, env = "OPENSHELL_SANDBOX_NO_PROXY")] + sandbox_no_proxy: Option, + + /// Path to a file containing the corporate proxy credentials as + /// `user:pass`. Delivered to the supervisor through a root-only secret + /// mount so the credentials never appear in config or container metadata. + #[arg(long, env = "OPENSHELL_SANDBOX_PROXY_AUTH_FILE")] + sandbox_proxy_auth_file: Option, + + /// Explicit acknowledgement (`true`) that the proxy credential is sent + /// as cleartext Basic auth over the plain-TCP connection to the http:// + /// proxy. Required when `--sandbox-proxy-auth-file` is set. + #[arg(long, env = "OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE")] + sandbox_proxy_auth_allow_insecure: Option, + + /// Send the destination hostname in CONNECT requests to the corporate + /// proxy instead of a validated IP. Only for proxies whose ACLs filter + /// on hostnames: the proxy then resolves the name itself, so sandbox + /// SSRF/`allowed_ips` validation no longer binds the connection. + #[arg(long, env = "OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME")] + sandbox_proxy_connect_by_hostname: Option, } #[tokio::main] @@ -114,12 +144,8 @@ async fn main() -> Result<()> { ) .init(); - let socket_path = args - .podman_socket - .unwrap_or_else(PodmanComputeConfig::default_socket_path); - let driver = PodmanComputeDriver::new(PodmanComputeConfig { - socket_path, + socket_path: args.podman_socket, default_image: args.sandbox_image.unwrap_or_default(), image_pull_policy: args.sandbox_image_pull_policy, grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), @@ -137,6 +163,11 @@ async fn main() -> Result<()> { guest_tls_cert: args.podman_tls_cert, guest_tls_key: args.podman_tls_key, sandbox_pids_limit: args.sandbox_pids_limit, + https_proxy: args.sandbox_https_proxy, + no_proxy: args.sandbox_no_proxy, + proxy_auth_file: args.sandbox_proxy_auth_file, + proxy_auth_allow_insecure: args.sandbox_proxy_auth_allow_insecure, + proxy_connect_by_hostname: args.sandbox_proxy_connect_by_hostname, ..PodmanComputeConfig::default() }) .await diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 35ef365a6d..7af0ddc389 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -3270,10 +3270,9 @@ async fn connect_local_container_engine() -> Option { return Some(docker); } - let podman_socket = podman_socket_path(); - if podman_socket.exists() - && let Ok(docker) = - Docker::connect_with_unix(podman_socket.to_str()?, 120, bollard::API_DEFAULT_VERSION) + let podman_socket = openshell_core::config::detect_podman_socket()?; + if let Ok(docker) = + Docker::connect_with_unix(podman_socket.to_str()?, 120, bollard::API_DEFAULT_VERSION) && docker.ping().await.is_ok() { info!( @@ -3286,25 +3285,6 @@ async fn connect_local_container_engine() -> Option { None } -/// Podman user socket path for the current platform. -fn podman_socket_path() -> PathBuf { - #[cfg(target_os = "macos")] - { - let home = std::env::var("HOME").unwrap_or_default(); - PathBuf::from(home).join(".local/share/containers/podman/machine/podman.sock") - } - #[cfg(target_os = "linux")] - { - std::env::var("XDG_RUNTIME_DIR").map_or_else( - |_| { - let uid = nix::unistd::getuid(); - PathBuf::from(format!("/run/user/{uid}/podman/podman.sock")) - }, - |xdg| PathBuf::from(xdg).join("podman/podman.sock"), - ) - } -} - fn is_openshell_local_build_image_ref(image_ref: &str) -> bool { image_ref.starts_with("openshell/sandbox-from:") } @@ -3558,6 +3538,8 @@ impl VmDriver { apply_registry_layer_blob(image_ref, rootfs, layer).await?; } + remove_registry_layer_staging(staging_dir).await?; + Ok(()) } @@ -3708,6 +3690,16 @@ async fn apply_registry_layer_blob( }) } +async fn remove_registry_layer_staging(staging_dir: &Path) -> Result<(), Status> { + let layers_dir = staging_dir.join("layers"); + tokio::fs::remove_dir_all(&layers_dir).await.map_err(|err| { + Status::internal(format!( + "remove registry layer staging dir '{}' failed: {err}", + layers_dir.display() + )) + }) +} + async fn download_registry_descriptor_blob_file( client: &OciClient, reference: &Reference, @@ -6635,6 +6627,29 @@ mod tests { ); } + #[tokio::test] + async fn remove_registry_layer_staging_preserves_merged_rootfs() { + let base = unique_temp_dir(); + let layers_dir = base.join("layers"); + let rootfs_dir = base.join("rootfs"); + fs::create_dir_all(&layers_dir).unwrap(); + fs::create_dir_all(&rootfs_dir).unwrap(); + fs::write(layers_dir.join("layer.blob"), b"compressed layer").unwrap(); + fs::write(rootfs_dir.join("merged.txt"), b"merged rootfs").unwrap(); + + remove_registry_layer_staging(&base) + .await + .expect("remove layer staging"); + + assert!(!layers_dir.exists()); + assert_eq!( + fs::read(rootfs_dir.join("merged.txt")).unwrap(), + b"merged rootfs" + ); + + let _ = fs::remove_dir_all(base); + } + #[test] fn sanitize_image_identity_rewrites_path_separators() { assert_eq!( diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs index 536b359e38..c71ebe6884 100644 --- a/crates/openshell-driver-vm/src/rootfs.rs +++ b/crates/openshell-driver-vm/src/rootfs.rs @@ -461,12 +461,19 @@ fn round_up_to_mib(bytes: u64) -> u64 { bytes.div_ceil(MIB) * MIB } +enum FormatterAttempt { + Succeeded, + Failed(String), + Unavailable(String), +} + fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), String> { - let mut last_error = None; - for tool in ["mke2fs", "mkfs.ext4"] { - for candidate in e2fs_tool_candidates(tool) { + let candidates = ["mke2fs", "mkfs.ext4"] + .into_iter() + .flat_map(e2fs_tool_candidates); + run_ext4_formatter_candidates(candidates, |candidate| { let label = candidate.display().to_string(); - let output = Command::new(&candidate) + let output = Command::new(candidate) .arg("-q") .arg("-F") .arg("-t") @@ -478,29 +485,51 @@ fn format_ext4_image_from_dir(source: &Path, image_path: &Path) -> Result<(), St .arg(image_path) .output(); match output { - Ok(output) if output.status.success() => return Ok(()), - Ok(output) => { - last_error = Some(format!( + Ok(output) if output.status.success() => FormatterAttempt::Succeeded, + Ok(output) => FormatterAttempt::Failed(format!( "{label} failed with status {}\nstdout: {}\nstderr: {}", output.status, String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) - )); - } + )), Err(err) if err.kind() == std::io::ErrorKind::NotFound => { - last_error = Some(format!("{label} not found")); - } - Err(err) => { - last_error = Some(format!("run {label}: {err}")); + FormatterAttempt::Unavailable(format!("{label} not found")) } + Err(err) => FormatterAttempt::Failed(format!("run {label}: {err}")), } + }) + .map_err(|details| { + format!( + "failed to create ext4 rootfs image from {}: {details}. Install e2fsprogs (mke2fs/mkfs.ext4) and retry", + source.display() + ) + }) +} + +fn run_ext4_formatter_candidates( + candidates: impl IntoIterator, + mut run: impl FnMut(&Path) -> FormatterAttempt, +) -> Result<(), String> { + let mut failures = Vec::new(); + let mut unavailable = Vec::new(); + + for candidate in candidates { + match run(&candidate) { + FormatterAttempt::Succeeded => return Ok(()), + FormatterAttempt::Failed(error) => failures.push(error), + FormatterAttempt::Unavailable(error) => unavailable.push(error), } } - Err(format!( - "failed to create ext4 rootfs image from {}: {}. Install e2fsprogs (mke2fs/mkfs.ext4) and retry", - source.display(), - last_error.unwrap_or_else(|| "no ext4 formatter found".to_string()) - )) + + if failures.is_empty() { + Err(if unavailable.is_empty() { + "no ext4 formatter candidates configured".to_string() + } else { + unavailable.join("\n") + }) + } else { + Err(failures.join("\n")) + } } fn ensure_rootfs_image_parent_dirs(image_path: &Path, guest_path: &str) { @@ -1131,6 +1160,61 @@ mod tests { assert_eq!(debugfs_quote_argument("/tmp/bad\npath"), None); } + #[test] + fn formatter_candidates_preserve_executed_failure_over_missing_fallback() { + let candidates = vec![PathBuf::from("mke2fs"), PathBuf::from("missing")]; + + let err = run_ext4_formatter_candidates(candidates, |candidate| { + if candidate == Path::new("mke2fs") { + FormatterAttempt::Failed( + "mke2fs failed with status 1\nstdout: formatter output\nstderr: no space left" + .to_string(), + ) + } else { + FormatterAttempt::Unavailable("missing not found".to_string()) + } + }) + .expect_err("formatter should fail"); + + assert!(err.contains("mke2fs failed with status 1")); + assert!(err.contains("no space left")); + assert!(!err.contains("missing not found")); + } + + #[test] + fn formatter_candidates_report_all_missing_tools() { + let candidates = vec![PathBuf::from("mke2fs"), PathBuf::from("mkfs.ext4")]; + + let err = run_ext4_formatter_candidates(candidates, |candidate| { + FormatterAttempt::Unavailable(format!("{} not found", candidate.display())) + }) + .expect_err("formatter should be unavailable"); + + assert!(err.contains("mke2fs not found")); + assert!(err.contains("mkfs.ext4 not found")); + } + + #[test] + fn formatter_candidates_accept_successful_fallback() { + let candidates = vec![PathBuf::from("first"), PathBuf::from("second")]; + let mut attempted = Vec::new(); + + run_ext4_formatter_candidates(candidates, |candidate| { + attempted.push(candidate.to_path_buf()); + if candidate == Path::new("second") { + FormatterAttempt::Succeeded + } else { + FormatterAttempt::Failed("first failed".to_string()) + } + }) + .expect("fallback should succeed"); + + assert_eq!( + attempted, + vec![PathBuf::from("first"), PathBuf::from("second")] + ); + } + fn unique_temp_dir() -> PathBuf { static COUNTER: AtomicU64 = AtomicU64::new(0); let nanos = SystemTime::now() diff --git a/crates/openshell-gateway-interceptors/src/runtime.rs b/crates/openshell-gateway-interceptors/src/runtime.rs index 04c9aba603..d510956aed 100644 --- a/crates/openshell-gateway-interceptors/src/runtime.rs +++ b/crates/openshell-gateway-interceptors/src/runtime.rs @@ -911,7 +911,6 @@ mod tests { .with_ansi(false) .without_time(); let subscriber = tracing_subscriber::registry().with(fmt_layer); - let dispatch = tracing::Dispatch::new(subscriber); let plan = BindingPlan { interceptor_name: "test".to_string(), binding_id: "binding".to_string(), @@ -940,7 +939,11 @@ mod tests { ..InterceptorResult::default() }; - tracing::dispatcher::with_default(&dispatch, || { + tracing::subscriber::with_default(subscriber, || { + // Other parallel tests may register this callsite while no subscriber + // is active. Refresh the process-wide cache after installing this + // thread-local subscriber so the event cannot remain disabled. + tracing::callsite::rebuild_interest_cache(); emit_evaluation_log(&plan, &result, "allow", 2); }); diff --git a/crates/openshell-policy/src/l7_validate.rs b/crates/openshell-policy/src/l7_validate.rs new file mode 100644 index 0000000000..18580fd2c1 --- /dev/null +++ b/crates/openshell-policy/src/l7_validate.rs @@ -0,0 +1,445 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared L7 endpoint semantic validation. +//! +//! Both profile lint (`openshell-providers`) and the runtime policy +//! validator (`openshell-supervisor-network`) call +//! [`validate_l7_endpoint_semantics`] to enforce the same constraints on +//! L7 endpoint field combinations, preventing drift between lint-time +//! and runtime checks. + +/// Known L7 inspection protocols. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum L7Protocol { + Rest, + Websocket, + Graphql, + Sql, + JsonRpc, + Mcp, +} + +impl L7Protocol { + /// Parse a protocol string into a known variant. + pub fn parse(s: &str) -> Option { + match s.to_ascii_lowercase().as_str() { + "rest" => Some(Self::Rest), + "websocket" => Some(Self::Websocket), + "graphql" => Some(Self::Graphql), + "sql" => Some(Self::Sql), + "json-rpc" => Some(Self::JsonRpc), + "mcp" => Some(Self::Mcp), + _ => None, + } + } + + /// Returns `true` for protocols in the JSON-RPC family (`json-rpc`, + /// `mcp`). + pub fn is_jsonrpc_family(self) -> bool { + matches!(self, Self::JsonRpc | Self::Mcp) + } +} + +/// Fields extracted from an endpoint definition needed for L7 semantic +/// validation. Both profile lint and the runtime validator construct this +/// from their own data representation. +#[allow(clippy::struct_excessive_bools)] +pub struct L7EndpointFields<'a> { + /// Protocol string as authored (e.g. `"rest"`, `"mcp"`). Empty + /// string means no L7 protocol was specified. + pub protocol: &'a str, + + /// Access preset string (e.g. `"read-only"`, `"full"`). Empty string + /// means no access preset. + pub access: &'a str, + + /// `true` when the endpoint has a non-empty rules list. + pub has_rules: bool, + + /// `true` when the endpoint has a non-empty `deny_rules` list. + pub has_deny_rules: bool, + + /// `true` when rules are present (non-empty) but would deny all + /// traffic because every entry lacks an allow clause. + pub rules_would_deny_all: bool, + + /// Value of `mcp.allow_all_known_mcp_methods` (defaults to `false`). + pub allow_all_known_mcp_methods: bool, +} + +/// Validate the semantic consistency of an L7 endpoint's field +/// combination. +/// +/// Returns a list of error message strings. An empty list means the +/// endpoint passes validation. Messages are bare — callers prepend +/// their own location context. +pub fn validate_l7_endpoint_semantics(ep: &L7EndpointFields<'_>) -> Vec { + let mut errors = Vec::new(); + let protocol = ep.protocol; + let l7_protocol = L7Protocol::parse(protocol); + let jsonrpc_family = l7_protocol.is_some_and(L7Protocol::is_jsonrpc_family); + let is_mcp = matches!(l7_protocol, Some(L7Protocol::Mcp)); + let is_jsonrpc = matches!(l7_protocol, Some(L7Protocol::JsonRpc)); + + // 1. Unknown protocol + if !protocol.is_empty() && l7_protocol.is_none() { + errors.push(format!( + "unknown protocol '{protocol}' (expected rest, websocket, graphql, sql, json-rpc, or mcp)" + )); + } + + // 2. rules + access mutually exclusive + if ep.has_rules && !ep.access.is_empty() { + errors.push("rules and access are mutually exclusive".to_string()); + } + + // 3. JSON-RPC family cannot use access presets + if jsonrpc_family && !ep.access.is_empty() { + if is_mcp { + errors.push(format!( + "protocol {protocol} does not support access presets; \ + use rules/deny_rules or set mcp.allow_all_known_mcp_methods: true \ + for an allow-all MCP policy" + )); + } else { + errors.push(format!( + "protocol {protocol} does not support access presets; \ + use explicit rules with allow.method such as \"*\"" + )); + } + } + + // 4. json-rpc requires explicit rules + if is_jsonrpc && !ep.has_rules && ep.access.is_empty() { + errors.push(format!( + "protocol {protocol} requires explicit rules with allow.method" + )); + } + + // 5. Non-MCP, non-JSON-RPC protocol requires rules or access (JSON-RPC's + // dedicated message is emitted by rule 4). + if !protocol.is_empty() && !is_mcp && !is_jsonrpc && !ep.has_rules && ep.access.is_empty() { + errors.push("protocol requires rules or access to define allowed traffic".to_string()); + } + + // 6. MCP requires rules when allow_all_known_mcp_methods is false + if is_mcp && !ep.has_rules && ep.access.is_empty() && !ep.allow_all_known_mcp_methods { + errors.push( + "protocol mcp requires rules when mcp.allow_all_known_mcp_methods is false".to_string(), + ); + } + + // 7. Rules would deny all traffic (MCP with allow_all_known_mcp_methods + // is excluded because normalization supplies method: "*"). + if ep.rules_would_deny_all && !(is_mcp && ep.allow_all_known_mcp_methods) { + errors.push( + "rules would deny all traffic (no allow clause found). \ + Use `access: full` or add allow clauses to rules." + .to_string(), + ); + } + + // 8. deny_rules require protocol + if ep.has_deny_rules && protocol.is_empty() { + errors.push("deny_rules require protocol (L7 inspection must be enabled)".to_string()); + } + + // 9. deny_rules require base allow set + if ep.has_deny_rules && !is_mcp && !ep.has_rules && ep.access.is_empty() { + errors.push("deny_rules require rules or access to define the base allow set".to_string()); + } + + errors +} + +#[cfg(test)] +mod tests { + use super::*; + + fn valid_rest_endpoint() -> L7EndpointFields<'static> { + L7EndpointFields { + protocol: "rest", + access: "read-only", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + } + } + + #[test] + fn valid_endpoint_produces_no_errors() { + let errors = validate_l7_endpoint_semantics(&valid_rest_endpoint()); + assert!(errors.is_empty(), "expected no errors, got: {errors:?}"); + } + + #[test] + fn rejects_unknown_protocol() { + let ep = L7EndpointFields { + protocol: "ftp", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.iter().any(|e| e.contains("unknown protocol"))); + } + + #[test] + fn rejects_rules_and_access_together() { + let ep = L7EndpointFields { + protocol: "rest", + access: "full", + has_rules: true, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.iter().any(|e| e.contains("mutually exclusive"))); + } + + #[test] + fn rejects_jsonrpc_with_access_presets() { + let ep = L7EndpointFields { + protocol: "json-rpc", + access: "full", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + errors + .iter() + .any(|e| e.contains("does not support access presets")) + ); + } + + #[test] + fn rejects_mcp_with_access_presets() { + let ep = L7EndpointFields { + protocol: "mcp", + access: "full", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + errors + .iter() + .any(|e| e.contains("allow_all_known_mcp_methods")) + ); + } + + #[test] + fn rejects_jsonrpc_without_rules() { + let ep = L7EndpointFields { + protocol: "json-rpc", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.iter().any(|e| e.contains("requires explicit rules"))); + } + + #[test] + fn rejects_protocol_without_rules_or_access() { + let ep = L7EndpointFields { + protocol: "rest", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + errors + .iter() + .any(|e| e.contains("protocol requires rules or access")) + ); + } + + #[test] + fn rejects_mcp_without_rules_when_allow_all_false() { + let ep = L7EndpointFields { + protocol: "mcp", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.iter().any(|e| e.contains("mcp requires rules when"))); + } + + #[test] + fn accepts_mcp_with_allow_all_true() { + let ep = L7EndpointFields { + protocol: "mcp", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: true, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.is_empty(), "expected no errors, got: {errors:?}"); + } + + #[test] + fn rejects_rules_that_deny_all() { + let ep = L7EndpointFields { + protocol: "rest", + access: "", + has_rules: true, + has_deny_rules: false, + rules_would_deny_all: true, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.iter().any(|e| e.contains("would deny all traffic"))); + } + + #[test] + fn mcp_allow_all_with_empty_allow_not_rejected() { + let ep = L7EndpointFields { + protocol: "mcp", + access: "", + has_rules: true, + has_deny_rules: false, + rules_would_deny_all: true, + allow_all_known_mcp_methods: true, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + !errors.iter().any(|e| e.contains("would deny all traffic")), + "MCP with allow_all_known_mcp_methods should not reject empty allow rules: {errors:?}" + ); + } + + #[test] + fn rejects_deny_rules_without_protocol() { + let ep = L7EndpointFields { + protocol: "", + access: "", + has_rules: false, + has_deny_rules: true, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + errors + .iter() + .any(|e| e.contains("deny_rules require protocol")) + ); + } + + #[test] + fn rejects_deny_rules_without_allow_base() { + let ep = L7EndpointFields { + protocol: "rest", + access: "", + has_rules: false, + has_deny_rules: true, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!( + errors + .iter() + .any(|e| e.contains("deny_rules require rules or access")) + ); + } + + #[test] + fn no_protocol_no_errors() { + let ep = L7EndpointFields { + protocol: "", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert!(errors.is_empty(), "expected no errors, got: {errors:?}"); + } + + #[test] + fn l7_protocol_parse_known_variants() { + assert_eq!(L7Protocol::parse("rest"), Some(L7Protocol::Rest)); + assert_eq!(L7Protocol::parse("websocket"), Some(L7Protocol::Websocket)); + assert_eq!(L7Protocol::parse("graphql"), Some(L7Protocol::Graphql)); + assert_eq!(L7Protocol::parse("sql"), Some(L7Protocol::Sql)); + assert_eq!(L7Protocol::parse("json-rpc"), Some(L7Protocol::JsonRpc)); + assert_eq!(L7Protocol::parse("mcp"), Some(L7Protocol::Mcp)); + assert_eq!(L7Protocol::parse("unknown"), None); + assert_eq!(L7Protocol::parse(""), None); + assert_eq!(L7Protocol::parse("REST"), Some(L7Protocol::Rest)); + assert_eq!(L7Protocol::parse("Mcp"), Some(L7Protocol::Mcp)); + assert_eq!(L7Protocol::parse("JSON-RPC"), Some(L7Protocol::JsonRpc)); + } + + #[test] + fn jsonrpc_with_access_emits_single_diagnostic() { + let ep = L7EndpointFields { + protocol: "json-rpc", + access: "full", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert_eq!( + errors, + vec![ + "protocol json-rpc does not support access presets; \ + use explicit rules with allow.method such as \"*\"" + ], + "should emit only the access-preset error, not the redundant missing-rules error" + ); + } + + #[test] + fn jsonrpc_without_rules_or_access_emits_single_diagnostic() { + let ep = L7EndpointFields { + protocol: "json-rpc", + access: "", + has_rules: false, + has_deny_rules: false, + rules_would_deny_all: false, + allow_all_known_mcp_methods: false, + }; + let errors = validate_l7_endpoint_semantics(&ep); + assert_eq!( + errors, + vec!["protocol json-rpc requires explicit rules with allow.method"], + "should emit only the missing-rules error" + ); + } + + #[test] + fn l7_protocol_jsonrpc_family() { + assert!(L7Protocol::JsonRpc.is_jsonrpc_family()); + assert!(L7Protocol::Mcp.is_jsonrpc_family()); + assert!(!L7Protocol::Rest.is_jsonrpc_family()); + assert!(!L7Protocol::Websocket.is_jsonrpc_family()); + assert!(!L7Protocol::Graphql.is_jsonrpc_family()); + assert!(!L7Protocol::Sql.is_jsonrpc_family()); + } +} diff --git a/crates/openshell-policy/src/lib.rs b/crates/openshell-policy/src/lib.rs index 3c72b19b32..ea838b3b74 100644 --- a/crates/openshell-policy/src/lib.rs +++ b/crates/openshell-policy/src/lib.rs @@ -10,6 +10,7 @@ //! these types, ensuring round-trip fidelity. mod compose; +mod l7_validate; mod merge; mod middleware; @@ -29,6 +30,7 @@ pub use compose::{ PROVIDER_RULE_NAME_PREFIX, ProviderPolicyLayer, compose_effective_policy, is_provider_rule_name, provider_rule_name, strip_provider_rule_names, }; +pub use l7_validate::{L7EndpointFields, L7Protocol, validate_l7_endpoint_semantics}; pub use merge::{ PolicyMergeError, PolicyMergeOp, PolicyMergeResult, PolicyMergeWarning, generated_rule_name, merge_policy, policy_covers_rule, diff --git a/crates/openshell-prover/src/lib.rs b/crates/openshell-prover/src/lib.rs index 2267052048..0fb8757577 100644 --- a/crates/openshell-prover/src/lib.rs +++ b/crates/openshell-prover/src/lib.rs @@ -26,11 +26,11 @@ use policy::parse_policy; use queries::run_all_queries; use report::{render_compact, render_report}; -/// Run the prover end-to-end and return an exit code. +/// Run the prover end-to-end and return a result containing an exit code. /// -/// - `0` — pass (no critical/high findings, or all accepted) -/// - `1` — fail (critical or high findings present) -/// - `2` — input error +/// - `Ok(0)` — pass (no findings, or all accepted) +/// - `Ok(1)` — fail (one or more unaccepted findings present) +/// - `Err(_)` — input or registry loading error /// /// Binary and API capability registries are embedded at compile time. /// Pass `registry_dir` to override with a custom filesystem registry. diff --git a/crates/openshell-providers/Cargo.toml b/crates/openshell-providers/Cargo.toml index 9b294d7b71..abf6a6f11a 100644 --- a/crates/openshell-providers/Cargo.toml +++ b/crates/openshell-providers/Cargo.toml @@ -13,6 +13,7 @@ repository.workspace = true [dependencies] glob = { workspace = true } openshell-core = { path = "../openshell-core", default-features = false } +openshell-policy = { path = "../openshell-policy" } serde = { workspace = true } serde_json = { workspace = true } serde_yml = { workspace = true } diff --git a/crates/openshell-providers/src/profiles.rs b/crates/openshell-providers/src/profiles.rs index ab2535cf63..2ba071db16 100644 --- a/crates/openshell-providers/src/profiles.rs +++ b/crates/openshell-providers/src/profiles.rs @@ -13,6 +13,7 @@ use openshell_core::proto::{ ProviderProfileCredential, ProviderProfileDiscovery, }; use openshell_core::secrets::uses_reserved_revision_namespace; +use openshell_policy::{L7EndpointFields, validate_l7_endpoint_semantics}; use serde::ser::SerializeStruct; use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; use std::collections::{HashMap, HashSet}; @@ -51,6 +52,12 @@ pub enum ProfileError { InvalidEndpoint { id: String, host: String, port: u32 }, #[error("provider profile '{id}' has duplicate credential env var '{env_var}'")] DuplicateCredentialEnvVar { id: String, env_var: String }, + #[error("provider profile '{id}' validation error: {field}: {message}")] + ValidationError { + id: String, + field: String, + message: String, + }, } #[derive(Debug, Clone, PartialEq, Eq)] @@ -200,14 +207,14 @@ pub struct EndpointProfile { pub access: String, #[serde(default, skip_serializing_if = "String::is_empty")] pub enforcement: String, - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub rules: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rules: Option>, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub allowed_ips: Vec, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub ports: Vec, - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub deny_rules: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub deny_rules: Option>, #[serde(default, skip_serializing_if = "is_false")] pub allow_encoded_slash: bool, #[serde(default, skip_serializing_if = "is_false")] @@ -264,6 +271,25 @@ pub struct L7AllowProfile { pub operation_name: String, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub fields: Vec, + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub params: HashMap, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool: Option, +} + +impl L7AllowProfile { + fn has_tool_selector(&self) -> bool { + self.params.contains_key("name") || self.tool.is_some() + } + + fn is_effectively_empty(&self) -> bool { + self.method.is_empty() + && self.path.is_empty() + && self.command.is_empty() + && self.operation_type.is_empty() + && self.operation_name.is_empty() + && !self.has_tool_selector() + } } #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] @@ -282,9 +308,19 @@ pub struct L7DenyRuleProfile { pub operation_name: String, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub fields: Vec, + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub params: HashMap, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool: Option, } -#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +impl L7DenyRuleProfile { + fn has_tool_selector(&self) -> bool { + self.params.contains_key("name") || self.tool.is_some() + } +} + +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] pub struct L7QueryMatcherProfile { #[serde(default, skip_serializing_if = "String::is_empty")] pub glob: String, @@ -292,6 +328,32 @@ pub struct L7QueryMatcherProfile { pub any: Vec, } +impl<'de> Deserialize<'de> for L7QueryMatcherProfile { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + #[derive(Deserialize)] + #[serde(untagged)] + enum StringOrObject { + Scalar(String), + Object { + #[serde(default)] + glob: String, + #[serde(default)] + any: Vec, + }, + } + match StringOrObject::deserialize(deserializer)? { + StringOrObject::Scalar(s) => Ok(Self { + glob: s, + any: vec![], + }), + StringOrObject::Object { glob, any } => Ok(Self { glob, any }), + } + } +} + #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] pub struct GraphqlOperationProfile { #[serde(default, skip_serializing_if = "String::is_empty")] @@ -532,6 +594,63 @@ impl ProviderTypeProfile { binaries: self.binaries.iter().map(binary_to_proto).collect(), } } + + pub fn validate_before_lowering(&self, source: &str) -> Vec { + let mut diagnostics = Vec::new(); + for (index, endpoint) in self.endpoints.iter().enumerate() { + if endpoint.rules.as_ref().is_some_and(Vec::is_empty) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + &self.id, + format!("endpoints[{index}]"), + "rules list cannot be empty (would deny all traffic). \ + Use `access: full` or remove rules.", + )); + } + if endpoint.deny_rules.as_ref().is_some_and(Vec::is_empty) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + &self.id, + format!("endpoints[{index}]"), + "deny_rules list cannot be empty (would have no effect). \ + Remove it if no denials are needed.", + )); + } + + if endpoint.protocol != "mcp" { + continue; + } + if let Some(rules) = &endpoint.rules { + for (rule_idx, rule) in rules.iter().enumerate() { + if rule + .allow + .as_ref() + .is_some_and(|a| a.tool.is_some() && a.params.contains_key("name")) + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + &self.id, + format!("endpoints[{index}].rules[{rule_idx}].allow"), + "MCP rules must use either tool or params.name, not both", + )); + } + } + } + if let Some(deny_rules) = &endpoint.deny_rules { + for (deny_idx, deny_rule) in deny_rules.iter().enumerate() { + if deny_rule.tool.is_some() && deny_rule.params.contains_key("name") { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + &self.id, + format!("endpoints[{index}].deny_rules[{deny_idx}]"), + "MCP rules must use either tool or params.name, not both", + )); + } + } + } + } + diagnostics + } } #[allow(clippy::trivially_copy_pass_by_ref)] @@ -936,10 +1055,22 @@ fn endpoint_to_proto(endpoint: &EndpointProfile) -> NetworkEndpoint { tls: endpoint.tls.clone(), enforcement: endpoint.enforcement.clone(), access: endpoint.access.clone(), - rules: endpoint.rules.iter().map(rule_to_proto).collect(), + rules: endpoint + .rules + .as_deref() + .unwrap_or_default() + .iter() + .map(rule_to_proto) + .collect(), allowed_ips: endpoint.allowed_ips.clone(), ports: endpoint.ports.clone(), - deny_rules: endpoint.deny_rules.iter().map(deny_rule_to_proto).collect(), + deny_rules: endpoint + .deny_rules + .as_deref() + .unwrap_or_default() + .iter() + .map(deny_rule_to_proto) + .collect(), allow_encoded_slash: endpoint.allow_encoded_slash, websocket_credential_rewrite: endpoint.websocket_credential_rewrite, request_body_credential_rewrite: endpoint.request_body_credential_rewrite, @@ -968,14 +1099,24 @@ fn endpoint_from_proto(endpoint: &NetworkEndpoint) -> EndpointProfile { tls: endpoint.tls.clone(), access: endpoint.access.clone(), enforcement: endpoint.enforcement.clone(), - rules: endpoint.rules.iter().map(rule_from_proto).collect(), + rules: if endpoint.rules.is_empty() { + None + } else { + Some(endpoint.rules.iter().map(rule_from_proto).collect()) + }, allowed_ips: endpoint.allowed_ips.clone(), ports: endpoint.ports.clone(), - deny_rules: endpoint - .deny_rules - .iter() - .map(deny_rule_from_proto) - .collect(), + deny_rules: if endpoint.deny_rules.is_empty() { + None + } else { + Some( + endpoint + .deny_rules + .iter() + .map(deny_rule_from_proto) + .collect(), + ) + }, allow_encoded_slash: endpoint.allow_encoded_slash, websocket_credential_rewrite: endpoint.websocket_credential_rewrite, request_body_credential_rewrite: endpoint.request_body_credential_rewrite, @@ -1035,7 +1176,65 @@ fn rule_from_proto(rule: &L7Rule) -> L7RuleProfile { } } +fn glob_uses_wildcard(s: &str) -> bool { + s.contains('*') || s.contains('?') || s.contains('[') || s.contains('{') +} + +fn matcher_uses_glob_wildcard(matcher: &L7QueryMatcherProfile) -> bool { + glob_uses_wildcard(&matcher.glob) || matcher.any.iter().any(|s| glob_uses_wildcard(s)) +} + +fn validate_mcp_matcher( + diagnostics: &mut Vec, + source: &str, + profile_id: &str, + field: &str, + matcher: &L7QueryMatcherProfile, + strict_tool_names: bool, +) { + if !matcher.glob.is_empty() && !matcher.any.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + field.to_string(), + "matcher cannot specify both glob and any", + )); + } + if matcher.glob.is_empty() && matcher.any.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + field.to_string(), + "matcher must specify glob or any", + )); + } + if !strict_tool_names && matcher_uses_glob_wildcard(matcher) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + field.to_string(), + "wildcard tool-name matchers require mcp.strict_tool_names to remain enabled", + )); + } +} + +fn method_matcher_matches_tools_call(method: &str) -> bool { + method == "tools/call" + || method == "*" + || glob::Pattern::new(method).is_ok_and(|pattern| pattern.matches("tools/call")) +} + fn allow_to_proto(allow: &L7AllowProfile) -> L7Allow { + let mut params: HashMap = allow + .params + .iter() + .map(|(name, matcher)| (name.clone(), query_matcher_to_proto(matcher))) + .collect(); + if let Some(tool) = &allow.tool { + params + .entry("name".to_string()) + .or_insert_with(|| query_matcher_to_proto(tool)); + } L7Allow { method: allow.method.clone(), path: allow.path.clone(), @@ -1048,7 +1247,7 @@ fn allow_to_proto(allow: &L7AllowProfile) -> L7Allow { operation_type: allow.operation_type.clone(), operation_name: allow.operation_name.clone(), fields: allow.fields.clone(), - params: HashMap::new(), + params, } } @@ -1065,10 +1264,26 @@ fn allow_from_proto(allow: &L7Allow) -> L7AllowProfile { operation_type: allow.operation_type.clone(), operation_name: allow.operation_name.clone(), fields: allow.fields.clone(), + params: allow + .params + .iter() + .map(|(name, matcher)| (name.clone(), query_matcher_from_proto(matcher))) + .collect(), + tool: None, } } fn deny_rule_to_proto(rule: &L7DenyRuleProfile) -> L7DenyRule { + let mut params: HashMap = rule + .params + .iter() + .map(|(name, matcher)| (name.clone(), query_matcher_to_proto(matcher))) + .collect(); + if let Some(tool) = &rule.tool { + params + .entry("name".to_string()) + .or_insert_with(|| query_matcher_to_proto(tool)); + } L7DenyRule { method: rule.method.clone(), path: rule.path.clone(), @@ -1081,7 +1296,7 @@ fn deny_rule_to_proto(rule: &L7DenyRuleProfile) -> L7DenyRule { operation_type: rule.operation_type.clone(), operation_name: rule.operation_name.clone(), fields: rule.fields.clone(), - params: HashMap::new(), + params, } } @@ -1098,6 +1313,12 @@ fn deny_rule_from_proto(rule: &L7DenyRule) -> L7DenyRuleProfile { operation_type: rule.operation_type.clone(), operation_name: rule.operation_name.clone(), fields: rule.fields.clone(), + params: rule + .params + .iter() + .map(|(name, matcher)| (name.clone(), query_matcher_from_proto(matcher))) + .collect(), + tool: None, } } @@ -1210,6 +1431,11 @@ fn validate_profiles(profiles: &[ProviderTypeProfile]) -> Result<(), ProfileErro port: endpoint.port, }); } + return Err(ProfileError::ValidationError { + id: diagnostic.profile_id.clone(), + field: diagnostic.field.clone(), + message: diagnostic.message.clone(), + }); } Ok(()) @@ -1624,6 +1850,342 @@ pub fn validate_profile_set( format!("invalid endpoint '{}:{}'", endpoint.host, endpoint.port), )); } + + if endpoint.rules.as_ref().is_some_and(Vec::is_empty) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}]"), + "rules list cannot be empty (would deny all traffic). Use `access: full` or remove rules.", + )); + } + + if endpoint.deny_rules.as_ref().is_some_and(Vec::is_empty) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}]"), + "deny_rules list cannot be empty (would have no effect). Remove it if no denials are needed.", + )); + } + + let l7_fields = L7EndpointFields { + protocol: &endpoint.protocol, + access: &endpoint.access, + has_rules: endpoint.rules.as_ref().is_some_and(|r| !r.is_empty()), + has_deny_rules: endpoint.deny_rules.as_ref().is_some_and(|r| !r.is_empty()), + rules_would_deny_all: endpoint.rules.as_ref().is_some_and(|r| { + !r.is_empty() + && r.iter().all(|rule| { + rule.allow + .as_ref() + .is_none_or(L7AllowProfile::is_effectively_empty) + }) + }), + allow_all_known_mcp_methods: endpoint + .mcp + .as_ref() + .and_then(|opts| opts.allow_all_known_mcp_methods) + .unwrap_or(false), + }; + for msg in validate_l7_endpoint_semantics(&l7_fields) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}]"), + msg, + )); + } + + if endpoint.protocol == "mcp" { + let strict_tool_names = endpoint + .mcp + .as_ref() + .and_then(|opts| opts.strict_tool_names) + .unwrap_or(true); + + if let Some(rules) = &endpoint.rules { + for (rule_idx, rule) in rules.iter().enumerate() { + if let Some(allow) = &rule.allow { + let allow_loc = format!("endpoints[{index}].rules[{rule_idx}].allow"); + if allow.tool.is_some() && allow.params.contains_key("name") { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &allow_loc, + "MCP rules must use either tool or params.name, not both", + )); + } + if allow.has_tool_selector() + && !allow.method.is_empty() + && allow.method != "tools/call" + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &allow_loc, + "method must be tools/call when an MCP rule uses \ + tool or params.name", + )); + } + if allow.method.is_empty() && !l7_fields.allow_all_known_mcp_methods { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &allow_loc, + "method is required when \ + mcp.allow_all_known_mcp_methods is false", + )); + } + if glob_uses_wildcard(&allow.method) + && !allow.method.starts_with("tools/") + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("{allow_loc}.method"), + "MCP method globs are only valid for the tools/ \ + method family; omit method to use the endpoint \ + method profile", + )); + } + if !allow.path.is_empty() || !allow.query.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &allow_loc, + "mcp L7 rules must use method/tool, not path/query", + )); + } + for key in allow.params.keys() { + if key != "name" { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("{allow_loc}.params.{key}"), + "MCP tool argument matching is not supported yet", + )); + } + } + if let Some(tool) = &allow.tool { + validate_mcp_matcher( + &mut diagnostics, + source, + profile_id, + &format!("{allow_loc}.tool"), + tool, + strict_tool_names, + ); + } + if let Some(params_name) = allow.params.get("name") { + validate_mcp_matcher( + &mut diagnostics, + source, + profile_id, + &format!("{allow_loc}.params.name"), + params_name, + strict_tool_names, + ); + } + } + } + } + + if let Some(deny_rules) = &endpoint.deny_rules { + for (deny_idx, deny_rule) in deny_rules.iter().enumerate() { + let deny_loc = format!("endpoints[{index}].deny_rules[{deny_idx}]"); + if deny_rule.tool.is_some() && deny_rule.params.contains_key("name") { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &deny_loc, + "MCP rules must use either tool or params.name, not both", + )); + } + if deny_rule.has_tool_selector() + && !deny_rule.method.is_empty() + && deny_rule.method != "tools/call" + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &deny_loc, + "method must be tools/call when an MCP rule uses \ + tool or params.name", + )); + } + if deny_rule.method.is_empty() && !l7_fields.allow_all_known_mcp_methods { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &deny_loc, + "method is required when \ + mcp.allow_all_known_mcp_methods is false", + )); + } + if glob_uses_wildcard(&deny_rule.method) + && !deny_rule.method.starts_with("tools/") + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("{deny_loc}.method"), + "MCP method globs are only valid for the tools/ \ + method family; omit method to use the endpoint \ + method profile", + )); + } + if !deny_rule.path.is_empty() || !deny_rule.query.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + &deny_loc, + "mcp L7 rules must use method/tool, not path/query", + )); + } + for key in deny_rule.params.keys() { + if key != "name" { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("{deny_loc}.params.{key}"), + "MCP tool argument matching is not supported yet", + )); + } + } + if let Some(tool) = &deny_rule.tool { + validate_mcp_matcher( + &mut diagnostics, + source, + profile_id, + &format!("{deny_loc}.tool"), + tool, + strict_tool_names, + ); + } + if let Some(params_name) = deny_rule.params.get("name") { + validate_mcp_matcher( + &mut diagnostics, + source, + profile_id, + &format!("{deny_loc}.params.name"), + params_name, + strict_tool_names, + ); + } + } + } + + let has_tool_allow_selectors = endpoint.rules.as_ref().is_some_and(|r| { + r.iter().any(|rule| { + rule.allow + .as_ref() + .is_some_and(L7AllowProfile::has_tool_selector) + }) + }); + + if has_tool_allow_selectors { + if let Some(rules) = &endpoint.rules { + for (rule_idx, rule) in rules.iter().enumerate() { + if let Some(allow) = &rule.allow { + if !allow.has_tool_selector() { + let method = allow.method.as_str(); + if method_matcher_matches_tools_call(method) + || (method.is_empty() + && l7_fields.allow_all_known_mcp_methods) + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].rules[{rule_idx}].allow"), + "method matcher allows every tool call and conflicts \ + with MCP tool allow rules; add tool or params.name \ + to narrow tools/call, or remove the tool allow rules", + )); + } + } + } else if l7_fields.allow_all_known_mcp_methods { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].rules[{rule_idx}].allow"), + "method matcher allows every tool call and conflicts \ + with MCP tool allow rules; add tool or params.name \ + to narrow tools/call, or remove the tool allow rules", + )); + } + } + } + + if let Some(deny_rules) = &endpoint.deny_rules { + for (deny_idx, deny_rule) in deny_rules.iter().enumerate() { + if !deny_rule.has_tool_selector() + && method_matcher_matches_tools_call(&deny_rule.method) + { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].deny_rules[{deny_idx}]"), + "method matcher denies every tool call and conflicts \ + with MCP tool allow rules; add tool or params.name to \ + deny specific tools, or remove the tool allow rules", + )); + } + } + } + } + } else { + if endpoint.mcp.is_some() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}]"), + "mcp options are only valid for protocol mcp", + )); + } + if let Some(rules) = &endpoint.rules { + for (rule_idx, rule) in rules.iter().enumerate() { + if let Some(allow) = &rule.allow { + if allow.tool.is_some() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].rules[{rule_idx}].allow.tool"), + "MCP tool matching is only valid for protocol mcp", + )); + } + if !allow.params.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].rules[{rule_idx}].allow.params"), + "params matching is only valid for protocol mcp", + )); + } + } + } + } + if let Some(deny_rules) = &endpoint.deny_rules { + for (deny_idx, deny_rule) in deny_rules.iter().enumerate() { + if deny_rule.tool.is_some() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].deny_rules[{deny_idx}].tool"), + "MCP tool matching is only valid for protocol mcp", + )); + } + if !deny_rule.params.is_empty() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + format!("endpoints[{index}].deny_rules[{deny_idx}].params"), + "params matching is only valid for protocol mcp", + )); + } + } + } + } } for (index, binary) in profile.binaries.iter().enumerate() { @@ -2003,9 +2565,9 @@ mod tests { use openshell_core::proto::ProviderProfileCategory; use super::{ - DiscoveryProfile, ProfileError, ProviderTypeProfile, builtin_profiles, - normalize_profile_id, parse_profile_catalog_yamls, parse_profile_json, parse_profile_yaml, - profile_to_json, profile_to_yaml, validate_profile_set, + DiscoveryProfile, L7AllowProfile, L7QueryMatcherProfile, ProfileError, ProviderTypeProfile, + builtin_profiles, normalize_profile_id, parse_profile_catalog_yamls, parse_profile_json, + parse_profile_yaml, profile_to_json, profile_to_yaml, validate_profile_set, }; fn builtin_profile(id: &str) -> &'static ProviderTypeProfile { @@ -2377,6 +2939,53 @@ binaries: assert!(exported.contains("strict_tool_names: false")); } + #[test] + fn mcp_allow_params_round_trips_through_proto() { + let profile = parse_profile_yaml( + r" +id: mcp-params +display_name: MCP Params +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let rules = profile.endpoints[0].rules.as_ref().expect("rules present"); + assert_eq!(rules.len(), 1); + let allow = rules[0].allow.as_ref().expect("allow present"); + assert!( + allow.params.contains_key("name"), + "params.name should be set" + ); + + let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); + let rt_rules = from_proto.endpoints[0] + .rules + .as_ref() + .expect("rules survive proto round-trip"); + assert_eq!(rt_rules.len(), 1); + let rt_allow = rt_rules[0] + .allow + .as_ref() + .expect("allow survives proto round-trip"); + assert!( + rt_allow.params.contains_key("name"), + "params.name should survive proto round-trip" + ); + } + #[test] fn profile_refresh_metadata_round_trips_through_proto_and_yaml() { let profile = parse_profile_yaml( @@ -2838,12 +3447,23 @@ id: advanced display_name: Advanced category: other endpoints: + - host: graphql.example.com + port: 443 + protocol: graphql + access: read-only + persisted_queries: allow_registered + graphql_persisted_queries: + hash-a: + operation_type: query + operation_name: Viewer + fields: [viewer] + graphql_max_body_bytes: 131072 + path: /graphql - host: api.example.com ports: [443, 8443] protocol: rest tls: terminate enforcement: enforce - access: read-only rules: - allow: method: GET @@ -2856,14 +3476,6 @@ endpoints: - method: POST path: /admin/** allow_encoded_slash: true - persisted_queries: allow_registered - graphql_persisted_queries: - hash-a: - operation_type: query - operation_name: Viewer - fields: [viewer] - graphql_max_body_bytes: 131072 - path: /graphql binaries: - path: /usr/bin/custom harness: true @@ -2877,39 +3489,44 @@ binaries: ); let proto = profile.to_proto(); - let endpoint = proto.endpoints.first().expect("endpoint should exist"); - assert_eq!(endpoint.port, 0); - assert_eq!(endpoint.ports, vec![443, 8443]); - assert_eq!(endpoint.tls, "terminate"); - assert_eq!(endpoint.allowed_ips, vec!["10.0.0.0/24"]); - assert!(endpoint.allow_encoded_slash); - assert_eq!(endpoint.persisted_queries, "allow_registered"); - assert_eq!(endpoint.graphql_max_body_bytes, 131_072); - assert_eq!(endpoint.path, "/graphql"); + + let graphql_ep = &proto.endpoints[0]; + assert_eq!(graphql_ep.access, "read-only"); + assert_eq!(graphql_ep.persisted_queries, "allow_registered"); + assert_eq!(graphql_ep.graphql_max_body_bytes, 131_072); + assert_eq!(graphql_ep.path, "/graphql"); + assert_eq!( + graphql_ep + .graphql_persisted_queries + .get("hash-a") + .map(|operation| operation.operation_name.as_str()), + Some("Viewer") + ); + + let rest_ep = &proto.endpoints[1]; + assert_eq!(rest_ep.port, 0); + assert_eq!(rest_ep.ports, vec![443, 8443]); + assert_eq!(rest_ep.tls, "terminate"); + assert_eq!(rest_ep.allowed_ips, vec!["10.0.0.0/24"]); + assert!(rest_ep.allow_encoded_slash); assert_eq!( - endpoint + rest_ep .rules .first() .and_then(|rule| rule.allow.as_ref()) .map(|allow| allow.method.as_str()), Some("GET") ); - assert_eq!(endpoint.deny_rules[0].method, "POST"); - assert_eq!( - endpoint - .graphql_persisted_queries - .get("hash-a") - .map(|operation| operation.operation_name.as_str()), - Some("Viewer") - ); + assert_eq!(rest_ep.deny_rules[0].method, "POST"); assert!(proto.binaries[0].harness); let reparsed = parse_profile_yaml(&profile_to_yaml(&profile).expect("serialize YAML")) .expect("serialized profile should parse"); let reprotoo = reparsed.to_proto(); - assert_eq!(reprotoo.endpoints[0].rules.len(), 1); - assert_eq!(reprotoo.endpoints[0].deny_rules.len(), 1); - assert_eq!(reprotoo.endpoints[0].ports, vec![443, 8443]); + assert_eq!(reprotoo.endpoints[0].access, "read-only"); + assert_eq!(reprotoo.endpoints[1].rules.len(), 1); + assert_eq!(reprotoo.endpoints[1].deny_rules.len(), 1); + assert_eq!(reprotoo.endpoints[1].ports, vec![443, 8443]); assert!(reprotoo.binaries[0].harness); } @@ -3089,22 +3706,47 @@ endpoints: } #[test] - fn aws_sts_strategy_serde_roundtrip() { - use openshell_core::proto::ProviderCredentialRefreshStrategy; - assert_eq!( - super::provider_refresh_strategy_from_yaml("aws_sts_assume_role"), - Some(ProviderCredentialRefreshStrategy::AwsStsAssumeRole) - ); - assert_eq!( - super::provider_refresh_strategy_to_yaml( - ProviderCredentialRefreshStrategy::AwsStsAssumeRole - ), - "aws_sts_assume_role" - ); - } - - #[test] - fn aws_profile_parses_correctly() { + fn parse_profile_catalog_yamls_rejects_l7_validation_errors() { + let err = parse_profile_catalog_yamls(&[r" +id: bad-l7 +display_name: Bad L7 +endpoints: + - host: api.example.com + port: 443 + protocol: rest + access: read-write + rules: + - allow: + method: GET + path: /v1/** +binaries: + - /usr/bin/example-agent +"]) + .unwrap_err(); + + assert!( + matches!(err, ProfileError::ValidationError { ref id, ref message, .. } if id == "bad-l7" && message.contains("mutually exclusive")), + "expected L7 validation error for access+rules, got: {err}" + ); + } + + #[test] + fn aws_sts_strategy_serde_roundtrip() { + use openshell_core::proto::ProviderCredentialRefreshStrategy; + assert_eq!( + super::provider_refresh_strategy_from_yaml("aws_sts_assume_role"), + Some(ProviderCredentialRefreshStrategy::AwsStsAssumeRole) + ); + assert_eq!( + super::provider_refresh_strategy_to_yaml( + ProviderCredentialRefreshStrategy::AwsStsAssumeRole + ), + "aws_sts_assume_role" + ); + } + + #[test] + fn aws_profile_parses_correctly() { let aws = builtin_profile("aws"); assert_eq!(aws.display_name, "AWS"); assert_eq!(aws.credentials.len(), 3); @@ -3511,4 +4153,1333 @@ credentials: "unexpected diagnostics: {diagnostics:?}" ); } + + // -- L7 endpoint semantic validation (shared with runtime) ---------------- + + #[test] + fn validate_rejects_protocol_without_rules_or_access() { + let profile = parse_profile_yaml( + r" +id: opencode-openrouter +display_name: OpenCode (OpenRouter) +credentials: + - name: api_key + env_vars: [OPENROUTER_API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: openrouter.ai + port: 443 + protocol: rest + enforcement: enforce +binaries: + - /usr/bin/opencode +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("protocol requires rules or access")), + "expected lint to reject protocol without rules or access, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_accepts_protocol_with_access() { + let profile = parse_profile_yaml( + r" +id: valid-rest +display_name: Valid REST +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: rest + access: read-write +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + let errors: Vec<_> = diagnostics + .iter() + .filter(|d| d.severity == "error") + .collect(); + assert!(errors.is_empty(), "unexpected errors: {errors:?}"); + } + + #[test] + fn validate_rejects_unknown_protocol() { + let profile = parse_profile_yaml( + r" +id: bad-protocol +display_name: Bad Protocol +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: ftp +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("unknown protocol")), + "expected lint to reject unknown protocol, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_rejects_rules_and_access_together() { + let profile = parse_profile_yaml( + r" +id: both-rules-access +display_name: Both +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: rest + access: full + rules: + - allow: + method: GET + path: /api/** +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("mutually exclusive")), + "expected lint to reject rules + access, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_rejects_empty_rules_list() { + let profile = parse_profile_yaml( + r" +id: empty-rules +display_name: Empty Rules +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: rest + access: full + rules: [] +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("rules list cannot be empty")), + "expected lint to reject empty rules list, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_rejects_empty_allow_object_as_deny_all() { + let profile = parse_profile_yaml( + r" +id: empty-allow +display_name: Empty Allow +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: rest + rules: + - allow: {} +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("would deny all traffic")), + "expected lint to reject empty allow object as deny-all, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_rejects_empty_deny_rules_list() { + let profile = parse_profile_yaml( + r" +id: empty-deny-rules +display_name: Empty Deny Rules +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + protocol: rest + access: full + deny_rules: [] +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("deny_rules list cannot be empty")), + "expected lint to reject empty deny_rules list, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_params_selector_not_classified_as_deny_all() { + let profile = parse_profile_yaml( + r" +id: mcp-params +display_name: MCP Params +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: read_* +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + !diagnostics + .iter() + .any(|d| d.message.contains("would deny all traffic")), + "MCP rule with params.name selector should not be classified as deny-all: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_empty_allow_with_allow_all_conflicts_with_tool_rules() { + let profile = parse_profile_yaml( + r" +id: mcp-wildcard +display_name: MCP Wildcard +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: read_* + - allow: {} +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("allows every tool call") + && d.message.contains("conflicts with MCP tool allow rules")), + "empty allow with allow_all should conflict with tool-specific rules: {diagnostics:?}" + ); + } + + #[test] + fn validate_unsupported_params_rejected() { + let profile = parse_profile_yaml( + r" +id: unsupported-params +display_name: Unsupported Params +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + arguments: + glob: '*' +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("not supported yet")), + "unsupported params key should be rejected: {diagnostics:?}" + ); + assert!( + !diagnostics + .iter() + .any(|d| d.message.contains("would deny all traffic")), + "MCP with allow_all_known_mcp_methods should not emit deny-all for empty method: {diagnostics:?}" + ); + } + + #[test] + fn mcp_deny_rule_params_round_trips_through_proto() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-params +display_name: MCP Deny Params +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + deny_rules: + - method: tools/call + params: + name: + glob: dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let deny_rules = profile.endpoints[0] + .deny_rules + .as_ref() + .expect("deny_rules present"); + assert!( + deny_rules[0].params.contains_key("name"), + "params.name should be set" + ); + + let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); + let rt_deny = from_proto.endpoints[0] + .deny_rules + .as_ref() + .expect("deny_rules survive proto round-trip"); + assert!( + rt_deny[0].params.contains_key("name"), + "deny rule params.name should survive proto round-trip" + ); + } + + #[test] + fn mcp_tool_allow_round_trips_through_proto() { + let profile = parse_profile_yaml( + r" +id: mcp-tool +display_name: MCP Tool +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + tool: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let rules = profile.endpoints[0].rules.as_ref().expect("rules present"); + let allow = rules[0].allow.as_ref().expect("allow present"); + assert!(allow.tool.is_some(), "tool should be parsed"); + assert_eq!(allow.tool.as_ref().unwrap().glob, "read_*"); + + let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); + let rt_allow = from_proto.endpoints[0] + .rules + .as_ref() + .expect("rules survive")[0] + .allow + .as_ref() + .expect("allow survives"); + assert!( + rt_allow.params.contains_key("name"), + "tool should be lowered to params.name in proto: {rt_allow:?}" + ); + assert_eq!(rt_allow.params["name"].glob, "read_*"); + assert!( + rt_allow.tool.is_none(), + "tool should be None after round-trip" + ); + } + + #[test] + fn mcp_tool_deny_round_trips_through_proto() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-tool +display_name: MCP Deny Tool +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + deny_rules: + - method: 'tools/call' + tool: + glob: dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let deny_rules = profile.endpoints[0] + .deny_rules + .as_ref() + .expect("deny_rules present"); + assert!(deny_rules[0].tool.is_some(), "tool should be parsed"); + + let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); + let rt_deny = from_proto.endpoints[0] + .deny_rules + .as_ref() + .expect("deny_rules survive")[0] + .clone(); + assert!( + rt_deny.params.contains_key("name"), + "deny tool should be lowered to params.name in proto" + ); + assert_eq!(rt_deny.params["name"].glob, "dangerous_*"); + } + + #[test] + fn validate_mcp_tool_and_params_name_rejected() { + let profile = parse_profile_yaml( + r" +id: mcp-both +display_name: MCP Both +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + tool: + glob: read_* + params: + name: + glob: write_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("must use either tool or params.name, not both")), + "expected mutual exclusivity error, got: {diagnostics:?}" + ); + } + + #[test] + fn mcp_tool_prevents_classify_as_deny_all() { + let allow = L7AllowProfile { + method: String::new(), + path: String::new(), + command: String::new(), + query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: vec![], + params: HashMap::new(), + tool: Some(L7QueryMatcherProfile { + glob: "read_*".to_string(), + any: vec![], + }), + }; + assert!( + !allow.is_effectively_empty(), + "allow with tool selector should not be classified as deny-all" + ); + } + + #[test] + fn validate_mcp_explicit_broad_method_without_allow_all() { + let profile = parse_profile_yaml( + r" +id: mcp-broad +display_name: MCP Broad +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + method: 'tools/call' + params: + name: + glob: read_* + - allow: + method: 'tools/call' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("allows every tool call and conflicts")), + "explicit broad method should be detected even without allow_all_known_mcp_methods: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_glob_broad_method_detected() { + let profile = parse_profile_yaml( + r" +id: mcp-glob +display_name: MCP Glob +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: read_* + - allow: + method: 'tools/*' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("allows every tool call and conflicts")), + "glob method tools/* should be detected: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_deny_rule_broad_method_conflict() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-broad +display_name: MCP Deny Broad +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: read_* + deny_rules: + - method: 'tools/call' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("denies every tool call and conflicts")), + "deny rule with broad method should be detected: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_deny_rule_with_tool_selector_ok() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-ok +display_name: MCP Deny OK +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + params: + name: + glob: '*' + deny_rules: + - method: 'tools/call' + tool: + glob: dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + !diagnostics + .iter() + .any(|d| d.message.contains("denies every tool call")), + "deny rule with tool selector should not trigger conflict: {diagnostics:?}" + ); + } + + #[test] + fn mcp_tool_scalar_matcher_round_trips_through_proto() { + let profile = parse_profile_yaml( + r" +id: mcp-scalar-tool +display_name: MCP Scalar Tool +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + method: tools/call + tool: search_web +binaries: + - /usr/bin/example-agent +", + ) + .expect("scalar tool matcher should parse"); + + let rules = profile.endpoints[0].rules.as_ref().expect("rules present"); + let allow = rules[0].allow.as_ref().expect("allow present"); + assert_eq!( + allow.tool.as_ref().unwrap().glob, + "search_web", + "scalar string should deserialize to glob field" + ); + + let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); + let rt_allow = from_proto.endpoints[0] + .rules + .as_ref() + .expect("rules survive")[0] + .allow + .as_ref() + .expect("allow survives"); + assert_eq!( + rt_allow.params["name"].glob, "search_web", + "scalar tool should lower to params.name in proto" + ); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + let errors: Vec<_> = diagnostics + .iter() + .filter(|d| d.severity == "error") + .collect(); + assert!( + errors.is_empty(), + "scalar tool matcher should produce no validation errors: {errors:?}" + ); + } + + #[test] + fn validate_mcp_deny_tool_and_params_name_rejected() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-both +display_name: MCP Deny Both +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + deny_rules: + - method: tools/call + tool: + glob: dangerous_* + params: + name: + glob: also_dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("either tool or params.name")), + "deny rule with both tool and params.name should be rejected: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_method_must_be_tools_call_with_tool_selector() { + let profile = parse_profile_yaml( + r" +id: mcp-bad-method +display_name: MCP Bad Method +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + method: resources/read + tool: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("method must be tools/call")), + "tool selector with wrong method should be rejected: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_deny_method_must_be_tools_call_with_tool_selector() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-bad-method +display_name: MCP Deny Bad Method +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + deny_rules: + - method: resources/read + tool: + glob: dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("method must be tools/call")), + "deny rule with tool selector and wrong method should be rejected: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_method_required_without_allow_all() { + let profile = parse_profile_yaml( + r" +id: mcp-no-method +display_name: MCP No Method +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + tool: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("method is required")), + "missing method without allow_all should be rejected: {diagnostics:?}" + ); + } + + #[test] + fn validate_before_lowering_catches_tool_params_conflict() { + let profile = parse_profile_yaml( + r" +id: mcp-both +display_name: MCP Both +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + method: tools/call + tool: + glob: read_* + params: + name: + glob: write_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = profile.validate_before_lowering("profile.yaml"); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("either tool or params.name")), + "pre-lowering validation should catch tool + params.name conflict: {diagnostics:?}" + ); + } + + #[test] + fn validate_before_lowering_ok_with_tool_only() { + let profile = parse_profile_yaml( + r" +id: mcp-tool-only +display_name: MCP Tool Only +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + rules: + - allow: + method: tools/call + tool: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = profile.validate_before_lowering("profile.yaml"); + assert!( + diagnostics.is_empty(), + "tool-only rule should pass pre-lowering validation: {diagnostics:?}" + ); + } + + #[test] + fn validate_before_lowering_deny_catches_conflict() { + let profile = parse_profile_yaml( + r" +id: mcp-deny-both +display_name: MCP Deny Both +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + deny_rules: + - method: tools/call + tool: + glob: dangerous_* + params: + name: + glob: also_dangerous_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = profile.validate_before_lowering("profile.yaml"); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("either tool or params.name")), + "deny rule with both should be caught before lowering: {diagnostics:?}" + ); + } + + #[test] + fn validate_rejects_deny_rules_without_protocol() { + let profile = parse_profile_yaml( + r" +id: deny-no-protocol +display_name: Deny No Protocol +credentials: + - name: api_key + env_vars: [API_KEY] + auth_style: bearer + header_name: authorization +discovery: + credentials: [api_key] +endpoints: + - host: api.example.com + port: 443 + deny_rules: + - method: POST +binaries: + - /usr/bin/app +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("deny_rules require protocol")), + "expected lint to reject deny_rules without protocol, got: {diagnostics:?}" + ); + } + + #[test] + fn validate_before_lowering_catches_empty_rules() { + let profile = parse_profile_yaml( + r" +id: empty-rules +display_name: Empty Rules +endpoints: + - host: api.example.com + port: 443 + protocol: mcp + rules: [] +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = profile.validate_before_lowering("profile.yaml"); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("rules list cannot be empty")), + "pre-lowering should catch empty rules list: {diagnostics:?}" + ); + } + + #[test] + fn validate_before_lowering_catches_empty_deny_rules() { + let profile = parse_profile_yaml( + r" +id: empty-deny-rules +display_name: Empty Deny Rules +endpoints: + - host: api.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/call + deny_rules: [] +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = profile.validate_before_lowering("profile.yaml"); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("deny_rules list cannot be empty")), + "pre-lowering should catch empty deny_rules list: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_unsupported_params_key() { + let profile = parse_profile_yaml( + r" +id: bad-params +display_name: Bad Params +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/call + params: + arguments: + glob: '*' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("MCP tool argument matching is not supported yet")), + "expected unsupported params key error: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_matcher_glob_and_any_rejected() { + let profile = parse_profile_yaml( + r" +id: glob-and-any +display_name: Glob And Any +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/call + tool: + glob: read_* + any: + - write_data +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("cannot specify both glob and any")), + "expected glob+any mutual exclusion error: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_matcher_empty_rejected() { + let yaml = r" +id: empty-matcher +display_name: Empty Matcher +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/call + tool: '' +binaries: + - /usr/bin/example-agent +"; + let profile = parse_profile_yaml(yaml).expect("profile should parse"); + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("matcher must specify glob or any")), + "expected empty matcher error: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_wildcard_without_strict_tool_names() { + let profile = parse_profile_yaml( + r" +id: wildcard-no-strict +display_name: Wildcard No Strict +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + strict_tool_names: false + rules: + - allow: + method: tools/call + tool: + glob: '*' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("wildcard tool-name matchers require mcp.strict_tool_names")), + "expected wildcard policy error: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_wildcard_with_strict_tool_names_ok() { + let profile = parse_profile_yaml( + r" +id: wildcard-strict +display_name: Wildcard Strict +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + strict_tool_names: true + rules: + - allow: + method: tools/call + tool: + glob: '*' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + !diagnostics.iter().any(|d| d + .message + .contains("wildcard tool-name matchers require mcp.strict_tool_names")), + "wildcard with strict_tool_names enabled should be OK: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_method_glob_restricted_to_tools() { + let profile = parse_profile_yaml( + r" +id: bad-method-glob +display_name: Bad Method Glob +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: 'resources/*' +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("MCP method globs are only valid for the tools/ method family")), + "expected method glob restriction error: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_path_rejected() { + let profile = parse_profile_yaml( + r" +id: mcp-path +display_name: MCP Path +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/call + path: /api/tools +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics + .iter() + .any(|d| d.message.contains("must use method/tool, not path/query")), + "expected path rejection for MCP: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_tool_on_non_mcp_rejected() { + let profile = parse_profile_yaml( + r" +id: rest-tool +display_name: REST Tool +endpoints: + - host: api.example.com + port: 443 + protocol: rest + rules: + - allow: + method: GET + tool: + glob: read_* +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("MCP tool matching is only valid for protocol mcp")), + "expected tool-on-non-mcp rejection: {diagnostics:?}" + ); + } + + #[test] + fn validate_mcp_options_on_non_mcp_rejected() { + let profile = parse_profile_yaml( + r" +id: rest-mcp-opts +display_name: REST MCP Options +endpoints: + - host: api.example.com + port: 443 + protocol: rest + mcp: + strict_tool_names: true + rules: + - allow: + method: GET +binaries: + - /usr/bin/example-agent +", + ) + .expect("profile should parse"); + + let diagnostics = validate_profile_set(&[("profile.yaml".to_string(), profile)]); + assert!( + diagnostics.iter().any(|d| d + .message + .contains("mcp options are only valid for protocol mcp")), + "expected mcp-options-on-non-mcp rejection: {diagnostics:?}" + ); + } } diff --git a/crates/openshell-sandbox/Cargo.toml b/crates/openshell-sandbox/Cargo.toml index f9c23cf60f..6a51635b14 100644 --- a/crates/openshell-sandbox/Cargo.toml +++ b/crates/openshell-sandbox/Cargo.toml @@ -60,7 +60,6 @@ default = ["telemetry"] telemetry = ["openshell-core/telemetry"] [dev-dependencies] -openshell-core = { path = "../openshell-core", features = ["test-helpers"] } tempfile = "3" temp-env = "0.3" tokio-tungstenite = { workspace = true } diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index 347e1509b6..3a5c7b70ae 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -52,17 +52,9 @@ use openshell_ocsf::{ /// `run_sandbox()` startup via `openshell_ocsf::ctx::set_ctx`. pub(crate) use openshell_ocsf::ctx::ctx as ocsf_ctx; -/// Process-wide flag for the agent-driven policy proposal surface. -/// Set once during `run_sandbox()` startup and updated by the settings poll -/// loop when `agent_policy_proposals_enabled` changes. Read by the -/// `policy.local` route handler and the L7 deny body's `next_steps` builder -/// to gate the agent-controlled mutation surface. Exposed `pub(crate)` so -/// unit tests in sibling modules can flip the flag through a serialized -/// guard (see `policy_local::tests::ProposalsFlagGuard`). -pub(crate) use openshell_core::proposals::AGENT_PROPOSALS_ENABLED; - use openshell_core::denial::DenialEvent; use openshell_core::policy::{NetworkMode, NetworkPolicy, ProxyPolicy, SandboxPolicy}; +use openshell_core::proposals::AgentProposals; use openshell_core::provider_credentials::ProviderCredentialState; use openshell_supervisor_network::opa::OpaEngine; use openshell_supervisor_process::process::ProcessEnforcementMode; @@ -106,6 +98,7 @@ pub async fn run_sandbox( ocsf_enabled: Arc, network_enabled: bool, process_enabled: bool, + upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, ) -> Result { let (program, args) = command .split_first() @@ -160,27 +153,34 @@ pub async fn run_sandbox( // Load policy and initialize OPA engine let openshell_endpoint_for_proxy = openshell_endpoint.clone(); let sandbox_name_for_agg = sandbox.clone(); - let (mut policy, opa_engine, retained_proto, middleware_registry_status, loaded_policy_origin) = - if let Some(bootstrap) = sidecar_bootstrap.as_ref() { - let (policy, opa_engine, retained_proto, loaded_policy_origin) = - load_policy_from_sidecar_bootstrap(bootstrap)?; - ( - policy, - opa_engine, - retained_proto, - MiddlewareRegistryStatus::Synchronized, - loaded_policy_origin, - ) - } else { - load_policy( - sandbox_id.clone(), - sandbox, - openshell_endpoint.clone(), - policy_rules, - policy_data, - ) - .await? - }; + let ( + mut policy, + opa_engine, + retained_proto, + middleware_registry_status, + loaded_policy_origin, + initial_agent_proposals_enabled, + ) = if let Some(bootstrap) = sidecar_bootstrap.as_ref() { + let (policy, opa_engine, retained_proto, loaded_policy_origin) = + load_policy_from_sidecar_bootstrap(bootstrap)?; + ( + policy, + opa_engine, + retained_proto, + MiddlewareRegistryStatus::Synchronized, + loaded_policy_origin, + bootstrap.agent_proposals_enabled, + ) + } else { + load_policy( + sandbox_id.clone(), + sandbox, + openshell_endpoint.clone(), + policy_rules, + policy_data, + ) + .await? + }; // Override the policy's process identity with the driver-resolved UID/GID // from the pod environment. The policy defaults to the name "sandbox" which @@ -290,25 +290,23 @@ pub async fn run_sandbox( let provider_env = provider_credentials.child_env_with_gcp_resolved(); (provider_credentials, provider_env) }; + + // Shared agent-proposals feature flag. Seed from the same initial settings + // snapshot that produced the policy so networking and process setup agree + // before the poll loop starts reconciling later changes. + let agent_proposals = AgentProposals::new(initial_agent_proposals_enabled); + let process_control_writer = process_control_connection .as_ref() .map(|connection| connection.writer.clone()); let mut process_control_closed = None; if let Some(connection) = process_control_connection { process_control_closed = Some(connection.closed); - spawn_sidecar_control_update_watcher(connection.updates, provider_credentials.clone()); - } - - // Initialize the agent-proposals feature flag. Default false until the - // initial settings fetch (or the poll loop) tells us otherwise. The flag - // gates the skill install, the policy.local route handler, and the L7 - // deny body's `next_steps` field — see `agent_proposals_enabled()`. - let proposals_enabled = Arc::new(std::sync::atomic::AtomicBool::new(false)); - if AGENT_PROPOSALS_ENABLED - .set(proposals_enabled.clone()) - .is_err() - { - debug!("agent proposals flag already initialized, keeping existing"); + spawn_sidecar_control_update_watcher( + connection.updates, + provider_credentials.clone(), + agent_proposals.clone(), + ); } // Shared PID: set after process spawn so the proxy can look up @@ -389,7 +387,9 @@ pub async fn run_sandbox( inference_routes.as_deref(), denial_tx, activity_tx, + agent_proposals.clone(), workspace_rx.clone(), + &upstream_proxy_args, ) .await?, ) @@ -422,6 +422,7 @@ pub async fn run_sandbox( policy_proto: proto.clone(), provider_env_revision: provider_credentials.snapshot().revision, provider_child_env: provider_env.clone(), + agent_proposals_enabled: agent_proposals.enabled(), proxy_ca_cert_path: ca_paths.as_ref().map(|paths| paths.0.clone()), proxy_ca_bundle_path: ca_paths.as_ref().map(|paths| paths.1.clone()), }, @@ -591,6 +592,7 @@ pub async fn run_sandbox( ocsf_enabled: poll_ocsf_enabled, provider_credentials: poll_provider_credentials, policy_local_ctx: poll_policy_local, + agent_proposals: agent_proposals.clone(), middleware_registry_status, sidecar_control_publisher: sidecar_control_publisher.clone(), workspace_tx, @@ -710,6 +712,7 @@ pub async fn run_sandbox( provider_credentials, provider_env, ca_file_paths, + agent_proposals.clone(), #[cfg(target_os = "linux")] netns.as_ref(), #[cfg(target_os = "linux")] @@ -870,11 +873,12 @@ fn load_policy_from_sidecar_bootstrap( fn spawn_sidecar_control_update_watcher( mut updates: tokio::sync::mpsc::UnboundedReceiver, provider_credentials: ProviderCredentialState, + agent_proposals: AgentProposals, ) -> tokio::task::JoinHandle<()> { tokio::spawn(async move { while let Some(update) = updates.recv().await { match update { - sidecar_control::ControlUpdate::ProviderEnvUpdated { + sidecar_control::ControlUpdate::ProviderEnv { revision, provider_child_env, } => { @@ -895,7 +899,7 @@ fn spawn_sidecar_control_update_watcher( .build() ); } - sidecar_control::ControlUpdate::PolicyUpdated { + sidecar_control::ControlUpdate::Policy { policy_proto, policy_hash, config_revision, @@ -907,6 +911,19 @@ fn spawn_sidecar_control_update_watcher( "Received sidecar policy update for process supervisor" ); } + sidecar_control::ControlUpdate::AgentProposals { + enabled, + config_revision, + } => { + apply_agent_proposals_enabled( + &agent_proposals, + enabled, + "sidecar control", + Some(config_revision), + None, + skills::install_static_skills, + ); + } } } }) @@ -1831,6 +1848,7 @@ async fn load_policy( Option, MiddlewareRegistryStatus, LoadedPolicyOrigin, + bool, )> { // File mode: load OPA engine from rego rules + YAML data (dev override) if let (Some(policy_file), Some(data_file)) = (&policy_rules, &policy_data) { @@ -1879,6 +1897,7 @@ async fn load_policy( None, MiddlewareRegistryStatus::Synchronized, LoadedPolicyOrigin::LocalOverride, + false, )); } @@ -2060,6 +2079,7 @@ async fn load_policy( LoadedPolicyOrigin::Gateway { revision: loaded_policy_revision, }, + agent_proposals_enabled_from_settings(&snapshot.settings), )); } @@ -2504,6 +2524,7 @@ struct PolicyPollLoopContext { ocsf_enabled: Arc, provider_credentials: ProviderCredentialState, policy_local_ctx: Option>, + agent_proposals: AgentProposals, middleware_registry_status: MiddlewareRegistryStatus, sidecar_control_publisher: Option, workspace_tx: tokio::sync::watch::Sender, @@ -2623,6 +2644,14 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> { match initial_poll_disposition(&ctx.loaded_policy_origin, &result) { InitialPollDisposition::Acknowledge(candidate) => { apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "initial settings poll", + Some(candidate.config_revision), + ctx.sidecar_control_publisher.as_ref(), + skills::install_static_skills, + ); current_config_revision = candidate.config_revision; current_policy_hash.clone_from(&candidate.policy_hash); current_middleware_services = result.supervisor_middleware_services; @@ -2639,6 +2668,14 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> { InitialPollDisposition::Reconcile => pending_result = Some(result), InitialPollDisposition::TrackOnly => { apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "initial settings poll", + Some(result.config_revision), + ctx.sidecar_control_publisher.as_ref(), + skills::install_static_skills, + ); current_config_revision = result.config_revision; current_policy_hash = result.policy_hash.clone(); current_middleware_services = result.supervisor_middleware_services; @@ -2913,34 +2950,16 @@ async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> { // off picks up the surface without a recreate. We never uninstall on // a true→false transition: stale skill content on disk is harmless // because route_request and agent_next_steps both gate on the live - // atomic, so the agent that reads the skill will see 404s and an + // shared flag, so the agent that reads the skill will see 404s and an // empty `next_steps` array regardless. - if let Some(flag) = AGENT_PROPOSALS_ENABLED.get() { - let new_proposals = extract_bool_setting( - &result.settings, - openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY, - ) - .unwrap_or(false); - let prev_proposals = flag.swap(new_proposals, Ordering::Relaxed); - if new_proposals != prev_proposals { - info!( - agent_policy_proposals_enabled = new_proposals, - "agent-driven policy proposals toggled" - ); - if new_proposals && !prev_proposals { - match skills::install_static_skills() { - Ok(installed) => info!( - path = %installed.policy_advisor.display(), - "Installed sandbox agent skill on toggle-on" - ), - Err(error) => warn!( - error = %error, - "Failed to install sandbox agent skill on toggle-on" - ), - } - } - } - } + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "settings poll", + Some(result.config_revision), + ctx.sidecar_control_publisher.as_ref(), + skills::install_static_skills, + ); current_config_revision = result.config_revision; if !reloads_gateway_policy { @@ -2979,6 +2998,52 @@ fn extract_bool_setting( }) } +fn agent_proposals_enabled_from_settings( + settings: &std::collections::HashMap, +) -> bool { + extract_bool_setting( + settings, + openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY, + ) + .unwrap_or(false) +} + +fn apply_agent_proposals_enabled( + agent_proposals: &AgentProposals, + enabled: bool, + source: &'static str, + config_revision: Option, + sidecar_control_publisher: Option<&sidecar_control::Publisher>, + install_static_skills: impl FnOnce() -> Result, +) { + let previously_enabled = agent_proposals.swap_enabled(enabled); + if enabled == previously_enabled { + return; + } + + info!( + agent_policy_proposals_enabled = enabled, + source, config_revision, "agent-driven policy proposals toggled" + ); + + if let (Some(publisher), Some(config_revision)) = (sidecar_control_publisher, config_revision) { + publisher.publish_agent_proposals(enabled, config_revision); + } + + if enabled && !previously_enabled { + match install_static_skills() { + Ok(installed) => info!( + path = %installed.policy_advisor.display(), + "Installed sandbox agent skill on toggle-on" + ), + Err(error) => warn!( + error = %error, + "Failed to install sandbox agent skill on toggle-on" + ), + } + } +} + /// Log individual setting changes between two snapshots. fn log_setting_changes( old: &std::collections::HashMap, @@ -3060,7 +3125,7 @@ mod tests { use openshell_core::policy::{ FilesystemPolicy, LandlockPolicy, NetworkMode, NetworkPolicy, ProcessPolicy, ProxyPolicy, }; - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; fn proxy_policy(http_addr: Option) -> SandboxPolicy { SandboxPolicy { @@ -3132,9 +3197,13 @@ mod tests { 1, std::collections::HashMap::from([("TOKEN".to_string(), "old".to_string())]), ); - let handle = spawn_sidecar_control_update_watcher(rx, provider_credentials.clone()); + let handle = spawn_sidecar_control_update_watcher( + rx, + provider_credentials.clone(), + AgentProposals::default(), + ); - tx.send(sidecar_control::ControlUpdate::ProviderEnvUpdated { + tx.send(sidecar_control::ControlUpdate::ProviderEnv { revision: 2, provider_child_env: std::collections::HashMap::from([( "TOKEN".to_string(), @@ -3160,7 +3229,7 @@ mod tests { Some("new") ); - tx.send(sidecar_control::ControlUpdate::ProviderEnvUpdated { + tx.send(sidecar_control::ControlUpdate::ProviderEnv { revision: 1, provider_child_env: std::collections::HashMap::from([( "TOKEN".to_string(), @@ -3180,6 +3249,72 @@ mod tests { handle.abort(); } + #[tokio::test] + async fn sidecar_control_agent_proposals_update_flips_shared_state() { + let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); + let provider_credentials = + ProviderCredentialState::from_child_env_snapshot(0, std::collections::HashMap::new()); + let agent_proposals = AgentProposals::new(true); + let handle = + spawn_sidecar_control_update_watcher(rx, provider_credentials, agent_proposals.clone()); + + tx.send(sidecar_control::ControlUpdate::AgentProposals { + enabled: false, + config_revision: 5, + }) + .unwrap(); + + timeout(Duration::from_secs(1), async { + loop { + if !agent_proposals.enabled() { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .unwrap(); + handle.abort(); + } + + #[test] + fn apply_agent_proposals_enabled_installs_only_on_false_to_true() { + let agent_proposals = AgentProposals::default(); + let installs = AtomicUsize::new(0); + + apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(1), None, || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert!(agent_proposals.enabled()); + assert_eq!(installs.load(Ordering::Relaxed), 1); + + apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(2), None, || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert_eq!(installs.load(Ordering::Relaxed), 1); + + apply_agent_proposals_enabled(&agent_proposals, false, "test", Some(3), None, || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert!(!agent_proposals.enabled()); + assert_eq!(installs.load(Ordering::Relaxed), 1); + } + #[test] fn apply_ocsf_json_setting_enables_from_initial_settings_snapshot() { let enabled = AtomicBool::new(false); @@ -3201,6 +3336,24 @@ mod tests { assert!(!enabled.load(Ordering::Relaxed)); } + #[test] + fn agent_proposals_setting_enables_from_initial_settings_snapshot() { + let mut settings = std::collections::HashMap::new(); + settings.insert( + openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), + effective_bool(true), + ); + + assert!(agent_proposals_enabled_from_settings(&settings)); + } + + #[test] + fn agent_proposals_setting_defaults_false_when_unset() { + let settings = std::collections::HashMap::new(); + + assert!(!agent_proposals_enabled_from_settings(&settings)); + } + // ---- Policy disk discovery tests ---- #[test] diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 71f881f68a..62ae37b5a1 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -101,6 +101,9 @@ impl std::str::FromStr for Mode { } /// `OpenShell` Sandbox - process isolation and monitoring. +// CLI flags are naturally boolean switches; grouping them into structs would +// only obscure the clap definition. +#[allow(clippy::struct_excessive_bools)] #[derive(Parser, Debug)] #[command(name = "openshell-sandbox")] #[command(version = openshell_core::VERSION)] @@ -200,6 +203,32 @@ struct Args { /// Shared TLS work directory between the network init container and sidecar. #[arg(long, env = "OPENSHELL_PROXY_TLS_DIR", default_value = SIDECAR_TLS_DIR)] sidecar_tls_dir: String, + + // Corporate upstream proxy. Operator-owned egress boundary: accepted + // only as command-line arguments (no `env =`), because the driver + // controls the supervisor's argv while a sandbox image could bake + // matching `ENV` values. + /// Corporate forward proxy URL (`http://host:port`) for upstream TLS egress. + #[arg(long)] + upstream_proxy: Option, + + /// Comma-separated `NO_PROXY` list for the corporate proxy. + #[arg(long)] + upstream_no_proxy: Option, + + /// Path to the root-only file holding corporate proxy credentials (`user:pass`). + #[arg(long)] + upstream_proxy_auth_file: Option, + + /// Acknowledge that proxy credentials travel as cleartext Basic auth over + /// the plain-TCP connection to the `http://` proxy. + #[arg(long)] + upstream_proxy_auth_allow_insecure: bool, + + /// Send the destination hostname in CONNECT instead of a validated IP + /// (for proxies whose ACLs filter on hostnames). + #[arg(long)] + upstream_proxy_connect_by_hostname: bool, } /// Copy the running executable to `dest`, creating parent directories as @@ -581,6 +610,14 @@ fn main() -> Result<()> { // is not yet initialized at this point (run_sandbox hasn't been called). // The shorthand layer will render it in fallback format. + let upstream_proxy_args = openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs { + https_proxy: args.upstream_proxy, + no_proxy: args.upstream_no_proxy, + proxy_auth_file: args.upstream_proxy_auth_file, + proxy_auth_allow_insecure: args.upstream_proxy_auth_allow_insecure, + proxy_connect_by_hostname: args.upstream_proxy_connect_by_hostname, + }; + run_sandbox( command, args.workdir, @@ -598,6 +635,7 @@ fn main() -> Result<()> { ocsf_enabled, args.mode.network, args.mode.process, + upstream_proxy_args, ) .await })?; diff --git a/crates/openshell-sandbox/src/sidecar_control.rs b/crates/openshell-sandbox/src/sidecar_control.rs index c98dbc33bf..8cc2854e41 100644 --- a/crates/openshell-sandbox/src/sidecar_control.rs +++ b/crates/openshell-sandbox/src/sidecar_control.rs @@ -25,6 +25,7 @@ pub struct BootstrapData { pub policy_proto: openshell_core::proto::SandboxPolicy, pub provider_env_revision: u64, pub provider_child_env: HashMap, + pub agent_proposals_enabled: bool, pub proxy_ca_cert_path: Option, pub proxy_ca_bundle_path: Option, } @@ -44,15 +45,19 @@ pub struct ExpectedPeer { #[derive(Debug, Clone)] pub enum ControlUpdate { - ProviderEnvUpdated { + ProviderEnv { revision: u64, provider_child_env: HashMap, }, - PolicyUpdated { + Policy { policy_proto: Box, policy_hash: String, config_revision: u64, }, + AgentProposals { + enabled: bool, + config_revision: u64, + }, } #[derive(Clone)] @@ -95,6 +100,21 @@ impl Publisher { config_revision, }); } + + pub fn publish_agent_proposals(&self, enabled: bool, config_revision: u64) { + { + let mut state = self.state.write().expect("sidecar control state poisoned"); + if state.agent_proposals_enabled == enabled { + return; + } + state.agent_proposals_enabled = enabled; + } + + let _ = self.updates.send(WireServerMessage::AgentProposalsUpdated { + enabled, + config_revision, + }); + } } pub struct ServerHandle { @@ -145,6 +165,7 @@ enum WireServerMessage { policy_proto: Vec, provider_env_revision: u64, provider_child_env: HashMap, + agent_proposals_enabled: bool, proxy_ca_cert_path: Option, proxy_ca_bundle_path: Option, }, @@ -157,6 +178,10 @@ enum WireServerMessage { policy_hash: String, config_revision: u64, }, + AgentProposalsUpdated { + enabled: bool, + config_revision: u64, + }, } impl BootstrapData { @@ -166,6 +191,7 @@ impl BootstrapData { policy_proto: self.policy_proto.encode_to_vec(), provider_env_revision: self.provider_env_revision, provider_child_env: self.provider_child_env.clone(), + agent_proposals_enabled: self.agent_proposals_enabled, proxy_ca_cert_path: self .proxy_ca_cert_path .as_ref() @@ -186,6 +212,7 @@ impl TryFrom for BootstrapData { policy_proto, provider_env_revision, provider_child_env, + agent_proposals_enabled, proxy_ca_cert_path, proxy_ca_bundle_path, } = message @@ -203,6 +230,7 @@ impl TryFrom for BootstrapData { policy_proto, provider_env_revision, provider_child_env, + agent_proposals_enabled, proxy_ca_cert_path: proxy_ca_cert_path.map(PathBuf::from), proxy_ca_bundle_path: proxy_ca_bundle_path.map(PathBuf::from), }) @@ -217,7 +245,7 @@ impl TryFrom for ControlUpdate { WireServerMessage::ProviderEnvUpdated { revision, provider_child_env, - } => Ok(Self::ProviderEnvUpdated { + } => Ok(Self::ProviderEnv { revision, provider_child_env, }), @@ -230,12 +258,19 @@ impl TryFrom for ControlUpdate { openshell_core::proto::SandboxPolicy::decode(policy_proto.as_slice()) .into_diagnostic() .wrap_err("failed to decode sidecar policy update")?; - Ok(Self::PolicyUpdated { + Ok(Self::Policy { policy_proto: Box::new(policy_proto), policy_hash, config_revision, }) } + WireServerMessage::AgentProposalsUpdated { + enabled, + config_revision, + } => Ok(Self::AgentProposals { + enabled, + config_revision, + }), WireServerMessage::BootstrapResponse { .. } => Err(miette::miette!( "unexpected sidecar bootstrap response after initial handshake" )), @@ -407,13 +442,16 @@ async fn handle_connection( } } + // Subscribe before taking the bootstrap snapshot so an update can neither + // be missed between the snapshot and the live update stream nor omitted + // from the snapshot itself. + let mut update_rx = updates.subscribe(); let bootstrap = { let state = state.read().expect("sidecar control state poisoned"); state.to_wire() }; write_json_line(&mut writer, &bootstrap).await?; - let mut update_rx = updates.subscribe(); loop { tokio::select! { line = lines.next_line() => { @@ -583,6 +621,7 @@ mod tests { }, provider_env_revision: 3, provider_child_env: env.clone(), + agent_proposals_enabled: true, proxy_ca_cert_path: Some(PathBuf::from("/tmp/ca.pem")), proxy_ca_bundle_path: Some(PathBuf::from("/tmp/bundle.pem")), }; @@ -595,6 +634,7 @@ mod tests { assert_eq!(received.policy_proto.version, 7); assert_eq!(received.provider_env_revision, 3); assert_eq!(received.provider_child_env, env); + assert!(received.agent_proposals_enabled); assert_eq!( received.proxy_ca_cert_path, Some(PathBuf::from("/tmp/ca.pem")) @@ -605,6 +645,46 @@ mod tests { ); } + #[tokio::test] + async fn agent_proposals_update_is_delivered_to_process_client() { + let dir = tempfile::tempdir().unwrap(); + let socket = dir.path().join("control.sock"); + let server = spawn_server( + &socket, + BootstrapData { + policy_proto: SandboxPolicy::default(), + provider_env_revision: 0, + provider_child_env: HashMap::new(), + agent_proposals_enabled: false, + proxy_ca_cert_path: None, + proxy_ca_bundle_path: None, + }, + current_peer(), + ) + .unwrap(); + let publisher = server.publisher(); + let (_bootstrap, mut connection) = connect_process_client(&socket, Duration::from_secs(1)) + .await + .unwrap(); + + publisher.publish_agent_proposals(true, 9); + + let update = tokio::time::timeout(Duration::from_secs(1), connection.updates.recv()) + .await + .unwrap() + .unwrap(); + match update { + ControlUpdate::AgentProposals { + enabled, + config_revision, + } => { + assert!(enabled); + assert_eq!(config_revision, 9); + } + other => panic!("unexpected sidecar update: {other:?}"), + } + } + #[tokio::test] async fn entrypoint_started_is_delivered_to_server() { let dir = tempfile::tempdir().unwrap(); @@ -615,6 +695,7 @@ mod tests { policy_proto: SandboxPolicy::default(), provider_env_revision: 0, provider_child_env: HashMap::new(), + agent_proposals_enabled: false, proxy_ca_cert_path: None, proxy_ca_bundle_path: None, }, @@ -655,6 +736,7 @@ mod tests { policy_proto: SandboxPolicy::default(), provider_env_revision: 0, provider_child_env: HashMap::new(), + agent_proposals_enabled: false, proxy_ca_cert_path: None, proxy_ca_bundle_path: None, }, @@ -688,6 +770,7 @@ mod tests { policy_proto: SandboxPolicy::default(), provider_env_revision: 0, provider_child_env: HashMap::new(), + agent_proposals_enabled: false, proxy_ca_cert_path: None, proxy_ca_bundle_path: None, }, @@ -716,6 +799,7 @@ mod tests { policy_proto: SandboxPolicy::default(), provider_env_revision: 0, provider_child_env: HashMap::new(), + agent_proposals_enabled: false, proxy_ca_cert_path: None, proxy_ca_bundle_path: None, }, @@ -745,6 +829,7 @@ mod tests { policy_proto: SandboxPolicy::default(), provider_env_revision: 0, provider_child_env: HashMap::new(), + agent_proposals_enabled: false, proxy_ca_cert_path: None, proxy_ca_bundle_path: None, }, diff --git a/crates/openshell-sdk/Cargo.toml b/crates/openshell-sdk/Cargo.toml index 364cfd8d56..a1016baec0 100644 --- a/crates/openshell-sdk/Cargo.toml +++ b/crates/openshell-sdk/Cargo.toml @@ -30,8 +30,10 @@ tower = { workspace = true } tracing = { workspace = true } [dev-dependencies] +serde_json = { workspace = true } tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } tokio-stream = { workspace = true } +wiremock = "0.6" [lints] workspace = true diff --git a/crates/openshell-sdk/README.md b/crates/openshell-sdk/README.md index c7039fe9cb..93afc5359e 100644 --- a/crates/openshell-sdk/README.md +++ b/crates/openshell-sdk/README.md @@ -26,7 +26,9 @@ client bound to the current token. When a refresher is wired, use `force_refresh` to recover after a raw RPC returns `Unauthenticated`. The SDK consumes a `Refresh` trait that the caller implements; it does not run -the OIDC browser flow itself. +the OIDC browser flow itself. Its non-interactive refresh-token exchange also +accepts scopes that identity providers may require to select the API resource +for the refreshed access token. ## Transport modes diff --git a/crates/openshell-sdk/src/config.rs b/crates/openshell-sdk/src/config.rs index 94cfc28c0b..27a7411734 100644 --- a/crates/openshell-sdk/src/config.rs +++ b/crates/openshell-sdk/src/config.rs @@ -4,8 +4,9 @@ //! Public input types for the SDK: how callers describe a gateway and the //! credentials used to talk to it. //! -//! The CLI keeps its own filesystem-aware `TlsOptions` for plumbing; it -//! converts to a `ClientConfig` at the moment of dialing the gateway. +//! The CLI keeps its own filesystem-aware `TlsOptions` for transport plumbing +//! and reuses the SDK's OIDC refresh primitive. Other consumers describe the +//! complete transport with [`ClientConfig`]. use crate::refresh::Refresh; use std::sync::Arc; diff --git a/crates/openshell-sdk/src/oidc.rs b/crates/openshell-sdk/src/oidc.rs index 3f9366ad5e..3f59c25cd0 100644 --- a/crates/openshell-sdk/src/oidc.rs +++ b/crates/openshell-sdk/src/oidc.rs @@ -8,7 +8,7 @@ use crate::error::{Result, SdkError}; use oauth2::basic::BasicClient; -use oauth2::{ClientId, RefreshToken, TokenResponse, TokenUrl}; +use oauth2::{ClientId, RefreshToken, Scope, TokenResponse, TokenUrl}; use serde::Deserialize; /// OIDC discovery document (subset of fields callers consume). @@ -30,6 +30,9 @@ pub struct RefreshTokenInput { pub refresh_token: String, pub issuer: String, pub client_id: String, + /// Scopes to resend with the refresh request. Some identity providers use + /// these to select the API resource for the refreshed access token. + pub scopes: Vec, pub insecure: bool, } @@ -39,6 +42,7 @@ impl std::fmt::Debug for RefreshTokenInput { f.debug_struct("RefreshTokenInput") .field("issuer", &self.issuer) .field("client_id", &self.client_id) + .field("scopes", &self.scopes) .field("insecure", &self.insecure) .finish_non_exhaustive() } @@ -54,10 +58,22 @@ impl RefreshTokenInput { refresh_token: refresh_token.into(), issuer: issuer.into(), client_id: client_id.into(), + scopes: Vec::new(), insecure: false, } } + /// Set the scopes to resend with the refresh-token grant. + #[must_use] + pub fn with_scopes(mut self, scopes: I) -> Self + where + I: IntoIterator, + S: Into, + { + self.scopes = scopes.into_iter().map(Into::into).collect(); + self + } + #[must_use] pub fn with_insecure(mut self, insecure: bool) -> Self { self.insecure = insecure; @@ -132,9 +148,11 @@ pub fn http_client(insecure: bool) -> reqwest::Client { /// Refresh an OIDC access token using the `refresh_token` grant. /// -/// The caller is responsible for preserving the prior refresh token when -/// the output's `refresh_token` is `None` — per OAuth 2.0 the server may -/// omit it from the refresh response. +/// The request resends any configured scopes so providers that use scopes to +/// select an API resource mint the correct access token. The caller is +/// responsible for preserving the prior refresh token when the output's +/// `refresh_token` is `None` — per OAuth 2.0 the server may omit it from the +/// refresh response. pub async fn refresh_token(input: &RefreshTokenInput) -> Result { let discovery = discover(&input.issuer, input.insecure).await?; @@ -143,9 +161,14 @@ pub async fn refresh_token(input: &RefreshTokenInput) -> Result Refr #[cfg(test)] mod tests { use super::*; + use wiremock::matchers::{body_string_contains, method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; #[test] fn debug_redacts_tokens() { - let input = RefreshTokenInput::new("refresh-secret", "https://idp", "cli"); + let input = RefreshTokenInput::new("refresh-secret", "https://idp", "cli") + .with_scopes(["openid", "api://cli/access_as_user"]); let rendered = format!("{input:?}"); assert!(!rendered.contains("refresh-secret")); assert!(rendered.contains("cli")); @@ -187,4 +213,44 @@ mod tests { assert!(!rendered.contains("refresh-secret")); assert!(rendered.contains("has_refresh_token")); } + + #[tokio::test] + async fn refresh_sends_configured_scopes() { + let server = MockServer::start().await; + let issuer = server.uri(); + + Mock::given(method("GET")) + .and(path("/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "authorization_endpoint": format!("{}/authorize", server.uri()), + "token_endpoint": format!("{}/token", server.uri()), + }))) + .expect(1) + .mount(&server) + .await; + + Mock::given(method("POST")) + .and(path("/token")) + .and(body_string_contains("scope=")) + .and(body_string_contains("access_as_user")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "access_token": "refreshed-access", + "token_type": "bearer", + "expires_in": 300, + }))) + .expect(1) + .mount(&server) + .await; + + let input = RefreshTokenInput::new("refresh-secret", &issuer, "client-id") + .with_scopes(["openid", "api://client-id/access_as_user"]); + let refreshed = refresh_token(&input) + .await + .expect("configured scopes should be sent on refresh"); + + assert_eq!(refreshed.access_token, "refreshed-access"); + assert!(refreshed.refresh_token.is_none()); + assert!(refreshed.expires_at.is_some()); + } } diff --git a/crates/openshell-server/src/compute/driver_config.rs b/crates/openshell-server/src/compute/driver_config.rs index fa44cd3a38..59fed439bc 100644 --- a/crates/openshell-server/src/compute/driver_config.rs +++ b/crates/openshell-server/src/compute/driver_config.rs @@ -195,7 +195,7 @@ fn apply_vm_runtime_defaults(cfg: &mut VmComputeConfig, context: DriverStartupCo fn apply_podman_env_overrides(podman: &mut PodmanComputeConfig) { if let Ok(p) = std::env::var("OPENSHELL_PODMAN_SOCKET") { - podman.socket_path = PathBuf::from(p); + podman.socket_path = Some(PathBuf::from(p)); } if let Ok(ip) = std::env::var("OPENSHELL_PODMAN_HOST_GATEWAY_IP") { podman.host_gateway_ip = ip; diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index da5d3efd45..cf6b17c7e9 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -49,7 +49,7 @@ use std::fmt; use std::net::SocketAddr; use std::path::{Path, PathBuf}; use std::pin::Pin; -use std::sync::Arc; +use std::sync::{Arc, Mutex as StdMutex, Weak}; use std::time::Duration; #[cfg(unix)] use tokio::net::UnixStream; @@ -65,6 +65,90 @@ type SharedComputeDriver = const DELETE_PHASE_CAS_RETRY_LIMIT: usize = 3; +/// Serializes request-side deletes for the same stable sandbox ID. +/// +/// Watch events deliberately do not use these gates, so a slow driver delete +/// cannot block the sequential watch loop. Weak values let entries disappear +/// after the last request using a sandbox's gate completes. +#[derive(Debug, Default)] +struct DeleteGateRegistry { + gates: StdMutex>>>, +} + +impl DeleteGateRegistry { + async fn lock_for(&self, sandbox_id: &str) -> SandboxDeleteGuard { + let gate = self.gate_for(sandbox_id); + SandboxDeleteGuard { + _guard: gate.lock_owned().await, + } + } + + fn gate_for(&self, sandbox_id: &str) -> Arc> { + let mut gates = self + .gates + .lock() + .expect("sandbox delete gate registry lock poisoned"); + gates.retain(|_, gate| gate.strong_count() > 0); + + if let Some(gate) = gates.get(sandbox_id).and_then(Weak::upgrade) { + return gate; + } + + let gate = Arc::new(Mutex::new(())); + gates.insert(sandbox_id.to_string(), Arc::downgrade(&gate)); + gate + } + + #[cfg(test)] + fn entry_count(&self) -> usize { + self.gates + .lock() + .expect("sandbox delete gate registry lock poisoned") + .len() + } +} + +/// Proof that the current delete operation holds its sandbox-ID gate. +/// +/// Delete code must acquire this guard before taking `ComputeRuntime::sync_lock`. +/// Passing it to `lock_global_for_delete` makes that ordering visible at every +/// global-lock acquisition in the delete path. +#[derive(Debug)] +struct SandboxDeleteGuard { + _guard: tokio::sync::OwnedMutexGuard<()>, +} + +#[derive(Debug)] +struct SandboxDeleteTarget { + sandbox_id: String, + sandbox_name: String, +} + +/// Identity and driver result for a completed delete request. +#[derive(Debug, Eq, PartialEq)] +pub struct DeleteSandboxResult { + pub sandbox_id: String, + pub deleted: bool, +} + +#[derive(Debug)] +struct DeleteTransition { + /// Snapshot restored if the driver result is ambiguous and no newer write + /// has replaced the exact `Deleting` version. + previous: Sandbox, + /// Exact durable `Deleting` snapshot owned by this request. + deleting: Sandbox, +} + +/// Result of trying to claim ownership of a sandbox's delete operation. +#[derive(Debug)] +enum BeginDelete { + /// This request persisted `Deleting` and owns the driver call and recovery. + Started(Box), + /// Another request already owns or completed the driver-side delete call. + AlreadyDeleting, +} + #[derive(Debug, Clone)] pub struct ComputeDriverInfoSnapshot { /// Gateway-selected driver name used for routing and `driver_config` keys. @@ -285,6 +369,7 @@ pub struct ComputeRuntime { tracing_log_bus: TracingLogBus, supervisor_sessions: Arc, sync_lock: Arc>, + delete_gates: Arc, gateway_bind_addresses: Vec, replica_id: String, } @@ -341,6 +426,7 @@ impl ComputeRuntime { tracing_log_bus, supervisor_sessions, sync_lock: Arc::new(Mutex::new(())), + delete_gates: Arc::new(DeleteGateRegistry::default()), gateway_bind_addresses, replica_id: lease::replica_id(), }) @@ -356,6 +442,21 @@ impl ComputeRuntime { self.sync_lock.clone().lock_owned().await } + /// Acquires the process-wide lock for code that already holds the + /// sandbox-ID delete gate. The guard parameter documents and enforces that + /// delete-path callers acquire locks in delete-gate -> global-lock order. + async fn lock_global_for_delete( + &self, + _delete_guard: &SandboxDeleteGuard, + ) -> tokio::sync::OwnedMutexGuard<()> { + self.sync_lock.clone().lock_owned().await + } + + #[cfg(test)] + pub(crate) fn delete_gate_entry_count(&self) -> usize { + self.delete_gates.entry_count() + } + pub async fn new_docker( config: openshell_core::Config, docker_config: DockerComputeConfig, @@ -598,63 +699,133 @@ impl ComputeRuntime { } } - pub async fn delete_sandbox(&self, workspace: &str, name: &str) -> Result { - let _guard = self.sync_lock.lock().await; - - // Resolve sandbox ID from name - let sandbox = self + pub(crate) async fn delete_sandbox( + &self, + workspace: &str, + name: &str, + ) -> Result { + // Resolve and acquire both request-side locks before spawning the + // owned worker. Cancellation while any of these awaits is pending is + // harmless because no mutation or detached work has started. + let candidate = self .store .get_message_by_name::(workspace, name) .await - .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?; - - let Some(sandbox) = sandbox else { - return Err(Status::not_found("sandbox not found")); + .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + let target = SandboxDeleteTarget { + sandbox_id: candidate.object_id().to_string(), + sandbox_name: candidate.object_name().to_string(), }; + let delete_guard = self.delete_gates.lock_for(&target.sandbox_id).await; + let global_guard = self.lock_global_for_delete(&delete_guard).await; - let id = sandbox.object_id().to_string(); - - let sandbox = self.set_sandbox_phase_deleting_with_retry(&id).await?; + // There is no await between acquiring the initial guards and spawning + // the worker. From this commitment point onward, request cancellation + // cannot stop the delete after it starts mutating durable state. + let runtime = self.clone(); + tokio::spawn(async move { + runtime + .delete_sandbox_inner(target, delete_guard, global_guard) + .await + }) + .await + .map_err(|err| { + Status::internal(format!( + "sandbox delete worker terminated unexpectedly: {err}" + )) + })? + } - self.sandbox_index.update_from_sandbox(&sandbox); - self.sandbox_watch_bus.notify(&id); - self.cleanup_sandbox_owned_records(&sandbox) + async fn delete_sandbox_inner( + &self, + target: SandboxDeleteTarget, + delete_guard: SandboxDeleteGuard, + guard: tokio::sync::OwnedMutexGuard<()>, + ) -> Result { + let current = self + .store + .get_message::(&target.sandbox_id) .await - .map_err(|e| { - Status::internal(format!("cleanup owned records for sandbox {id}: {e}")) - })?; + .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?; + let Some(current) = current else { + // A delete that owned this ID's gate completed while this request + // waited. A different sandbox may now use the old name; this + // request acknowledges only the disappearance of its original ID. + self.cleanup_removed_sandbox_state(&target.sandbox_id); + return Ok(DeleteSandboxResult { + sandbox_id: target.sandbox_id, + deleted: true, + }); + }; + if current.object_name() != target.sandbox_name { + return Err(Status::aborted( + "sandbox name changed while the delete request was waiting; retry explicitly", + )); + } + + // `Started` carries both sides of the CAS transition: the durable + // `Deleting` row used to fence recovery, and the prior row used only + // for exact-version rollback after an ambiguous driver failure. + let transition = match self + .begin_sandbox_delete_with_initial_snapshot(&target.sandbox_id, Some(current)) + .await? + { + BeginDelete::AlreadyDeleting => { + return Ok(DeleteSandboxResult { + sandbox_id: target.sandbox_id, + deleted: true, + }); + } + BeginDelete::Started(transition) => *transition, + }; + + self.sandbox_index.update_from_sandbox(&transition.deleting); + self.sandbox_watch_bus.notify(&target.sandbox_id); + drop(guard); - let deleted = self + let result = self .driver .delete_sandbox(Request::new(DeleteSandboxRequest { - sandbox_id: sandbox.object_id().to_string(), - sandbox_name: sandbox.object_name().to_string(), + sandbox_id: transition.deleting.object_id().to_string(), + sandbox_name: transition.deleting.object_name().to_string(), })) - .await - .map(|response| response.into_inner().deleted) - .map_err(|err| Status::internal(format!("delete sandbox failed: {}", err.message())))?; + .await; - if !deleted && let Err(e) = self.store.delete(Sandbox::object_type(), &id).await { - warn!(sandbox_id = %id, error = %e, "Failed to clean up store after delete"); + match result { + Ok(response) => { + let deleted = response.into_inner().deleted; + if deleted { + self.cleanup_local_state_if_sandbox_absent(&delete_guard, &target.sandbox_id) + .await?; + } else if !self + .remove_deleting_sandbox_record(&delete_guard, &target.sandbox_id) + .await + { + return Err(Status::internal( + "compute resource was absent, but gateway cleanup did not complete", + )); + } + Ok(DeleteSandboxResult { + sandbox_id: target.sandbox_id, + deleted, + }) + } + Err(err) => { + self.recover_failed_delete(&delete_guard, &transition).await; + Err(Status::internal(format!( + "delete sandbox failed: {}", + err.message() + ))) + } } - - self.cleanup_sandbox_state(&id); - Ok(deleted) - } - - async fn set_sandbox_phase_deleting_with_retry( - &self, - sandbox_id: &str, - ) -> Result { - self.set_sandbox_phase_deleting_with_initial_snapshot(sandbox_id, None) - .await } - async fn set_sandbox_phase_deleting_with_initial_snapshot( + async fn begin_sandbox_delete_with_initial_snapshot( &self, sandbox_id: &str, mut initial_snapshot: Option, - ) -> Result { + ) -> Result { let operation = "set sandbox phase to Deleting"; for attempt in 1..=DELETE_PHASE_CAS_RETRY_LIMIT { @@ -668,18 +839,29 @@ impl ComputeRuntime { .ok_or_else(|| Status::not_found("sandbox not found"))?, }; + if SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown) + == SandboxPhase::Deleting + { + return Ok(BeginDelete::AlreadyDeleting); + } + + let previous = sandbox.clone(); + match self .write_sandbox_phase_deleting_from_snapshot(sandbox) .await { - Ok(sandbox) => { + Ok(deleting) => { if attempt > 1 { debug!( sandbox_id, attempt, "Retried sandbox delete phase transition after CAS conflict" ); } - return Ok(sandbox); + return Ok(BeginDelete::Started(Box::new(DeleteTransition { + previous, + deleting, + }))); } Err(crate::persistence::PersistenceError::Conflict { current_resource_version, @@ -705,6 +887,325 @@ impl ComputeRuntime { unreachable!("delete phase retry loop always returns") } + /// Removes a durable `Deleting` row after the driver confirms that its + /// compute resource is already absent (`deleted = false`). + /// + /// Benign resource-version changes are retried, but cleanup stops if the + /// row leaves `Deleting`; that state belongs to a concurrent writer. + async fn remove_deleting_sandbox_record( + &self, + delete_guard: &SandboxDeleteGuard, + sandbox_id: &str, + ) -> bool { + let _guard = self.lock_global_for_delete(delete_guard).await; + for attempt in 1..=DELETE_PHASE_CAS_RETRY_LIMIT { + let record = match self.store.get(Sandbox::object_type(), sandbox_id).await { + Ok(Some(record)) => record, + Ok(None) => { + self.cleanup_removed_sandbox_state(sandbox_id); + return true; + } + Err(err) => { + warn!( + sandbox_id, + error = %err, + "Failed to fetch sandbox after the compute resource was absent" + ); + return false; + } + }; + let sandbox = match decode_sandbox_record(&record) { + Ok(sandbox) => sandbox, + Err(err) => { + warn!(sandbox_id, error = %err, "Failed to decode sandbox during cleanup"); + return false; + } + }; + if SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown) + != SandboxPhase::Deleting + { + debug!( + sandbox_id, + "Skipped cleanup of a sandbox no longer deleting" + ); + return false; + } + + match self + .remove_sandbox_record_if_version_locked(sandbox_id, record.resource_version) + .await + { + Ok(true) => return true, + Ok(false) if attempt < DELETE_PHASE_CAS_RETRY_LIMIT => { + tokio::task::yield_now().await; + } + Ok(false) => return false, + Err(err) => { + warn!( + sandbox_id, + error = %err, + "Failed to clean up store after the compute resource was absent" + ); + return false; + } + } + } + + false + } + + /// Removes the sandbox by stable ID only when the expected resource + /// version still owns the row. The caller holds `sync_lock`; a successful + /// delete also removes sandbox-owned records, while successful or + /// already-completed removal clears this replica's index and watch/log + /// buses. + async fn remove_sandbox_record_if_version_locked( + &self, + sandbox_id: &str, + expected_resource_version: u64, + ) -> Result { + let Some(record) = self + .store + .get(Sandbox::object_type(), sandbox_id) + .await + .map_err(|err| err.to_string())? + else { + self.cleanup_removed_sandbox_state(sandbox_id); + return Ok(false); + }; + + if record.resource_version != expected_resource_version { + debug!( + sandbox_id, + expected_resource_version, + current_resource_version = record.resource_version, + "Skipped sandbox cleanup after a concurrent state change" + ); + return Ok(false); + } + + let sandbox = decode_sandbox_record(&record)?; + self.cleanup_sandbox_owned_records(&sandbox).await?; + + match self + .store + .delete_if( + Sandbox::object_type(), + sandbox_id, + expected_resource_version, + ) + .await + { + Ok(true) => { + self.cleanup_removed_sandbox_state(sandbox_id); + Ok(true) + } + Ok(false) => { + self.cleanup_removed_sandbox_state(sandbox_id); + Ok(false) + } + Err(crate::persistence::PersistenceError::Conflict { + current_resource_version, + }) => { + debug!( + sandbox_id, + expected_resource_version, + current_resource_version, + "Skipped sandbox cleanup after a concurrent state change" + ); + Ok(false) + } + Err(err) => Err(err.to_string()), + } + } + + /// Resolves an ambiguous driver delete error without overwriting newer + /// gateway state. + /// + /// The external lookup runs without `sync_lock`. Recovery then uses the + /// exact `Deleting` resource version to apply one of three outcomes: + /// reconcile an observed backend snapshot, remove a confirmed-absent + /// backend, or restore the pre-delete snapshot when lookup is inconclusive. + async fn recover_failed_delete( + &self, + delete_guard: &SandboxDeleteGuard, + transition: &DeleteTransition, + ) { + let sandbox_id = transition.deleting.object_id(); + let sandbox_name = transition.deleting.object_name(); + let deleting_resource_version = sandbox_resource_version(&transition.deleting); + + // The driver lookup is deliberately outside the process-wide guard. + let observed = self.get_driver_sandbox(sandbox_id, sandbox_name).await; + let _guard = self.lock_global_for_delete(delete_guard).await; + + match observed { + Ok(Some(snapshot)) if snapshot.id == sandbox_id && snapshot.status.is_some() => { + let session_connected = self.supervisor_sessions.has_session(sandbox_id); + self.write_delete_recovery_with_retry( + sandbox_id, + deleting_resource_version, + "reconcile observed backend snapshot", + |sandbox| apply_driver_snapshot(sandbox, &snapshot, session_connected), + ) + .await; + } + Ok(None) => { + match self + .remove_sandbox_record_if_version_locked(sandbox_id, deleting_resource_version) + .await + { + Ok(_) => {} + Err(err) => { + debug!( + sandbox_id, + error = %err, + "Skipped absent-backend recovery after a concurrent state change" + ); + } + } + } + Ok(Some(_)) | Err(_) => { + let previous = transition.previous.clone(); + self.write_delete_recovery_with_retry( + sandbox_id, + deleting_resource_version, + "restore pre-delete snapshot", + |sandbox| *sandbox = previous.clone(), + ) + .await; + } + } + } + + /// Applies an exact-version delete recovery, retrying transient persistence + /// failures only while the durable row remains at the version this delete + /// owns. CAS conflicts belong to a newer writer and are never retried. + async fn write_delete_recovery_with_retry( + &self, + sandbox_id: &str, + deleting_resource_version: u64, + recovery_action: &'static str, + apply: F, + ) where + F: Fn(&mut Sandbox) + Send + Sync, + { + for attempt in 1..=DELETE_PHASE_CAS_RETRY_LIMIT { + match self + .store + .update_message_cas::( + sandbox_id, + deleting_resource_version, + |sandbox| apply(sandbox), + ) + .await + { + Ok(recovered) => { + self.sandbox_index.update_from_sandbox(&recovered); + self.sandbox_watch_bus.notify(sandbox_id); + return; + } + Err(error @ crate::persistence::PersistenceError::Conflict { .. }) => { + self.handle_delete_recovery_conflict(sandbox_id, error, recovery_action) + .await; + return; + } + Err(error) => match self.store.get(Sandbox::object_type(), sandbox_id).await { + Ok(None) => { + debug!( + sandbox_id, + recovery_action, + "Delete recovery found the row removed by another replica; cleaning local state" + ); + self.cleanup_removed_sandbox_state(sandbox_id); + return; + } + Ok(Some(record)) + if record.resource_version == deleting_resource_version + && attempt < DELETE_PHASE_CAS_RETRY_LIMIT => + { + debug!( + sandbox_id, + recovery_action, + attempt, + error = %error, + "Delete recovery write failed while its version was unchanged; retrying" + ); + tokio::task::yield_now().await; + } + Ok(Some(record)) if record.resource_version == deleting_resource_version => { + warn!( + sandbox_id, + recovery_action, + attempt, + error = %error, + "Delete recovery write failed after bounded retries; sandbox remains deleting" + ); + return; + } + Ok(Some(record)) => { + debug!( + sandbox_id, + recovery_action, + error = %error, + current_resource_version = record.resource_version, + "Skipped delete recovery after a concurrent state change" + ); + return; + } + Err(fetch_error) => { + warn!( + sandbox_id, + recovery_action, + error = %error, + fetch_error = %fetch_error, + "Failed to verify sandbox state after delete recovery write failure" + ); + return; + } + }, + } + } + } + + /// Handles a recovery CAS conflict while the caller holds `sync_lock`. + /// Another replica may have removed the durable row during the external + /// driver lookup; in that case this replica still needs local cleanup. + async fn handle_delete_recovery_conflict( + &self, + sandbox_id: &str, + error: crate::persistence::PersistenceError, + recovery_action: &'static str, + ) { + match self.store.get(Sandbox::object_type(), sandbox_id).await { + Ok(None) => { + debug!( + sandbox_id, + recovery_action, + "Delete recovery found the row removed by another replica; cleaning local state" + ); + self.cleanup_removed_sandbox_state(sandbox_id); + } + Ok(Some(_)) => { + debug!( + sandbox_id, + recovery_action, + error = %error, + "Skipped delete recovery after a concurrent state change" + ); + } + Err(fetch_error) => { + warn!( + sandbox_id, + recovery_action, + error = %error, + fetch_error = %fetch_error, + "Failed to verify sandbox state after delete recovery conflict" + ); + } + } + } + async fn write_sandbox_phase_deleting_from_snapshot( &self, mut sandbox: Sandbox, @@ -1164,160 +1665,36 @@ impl ComputeRuntime { incoming: DriverSandbox, existing_record: Option, ) -> Result<(), String> { - let existing = existing_record - .as_ref() - .map(decode_sandbox_record) - .transpose()?; - - // If no existing record, create initial sandbox (first watch event for this sandbox) - if existing.is_none() { - use crate::persistence::WriteCondition; - let now_ms = openshell_core::time::now_ms(); - - let session_connected = self.supervisor_sessions.has_session(&incoming.id); - let mut phase = derive_phase(incoming.status.as_ref()); - let sandbox_name = incoming.name.clone(); - - let supervisor_promoted = session_connected - && matches!(phase, SandboxPhase::Provisioning | SandboxPhase::Unknown); - if supervisor_promoted { - phase = SandboxPhase::Ready; - } - - let mut status = incoming - .status - .as_ref() - .map(|s| public_status_from_driver(s, phase, 0)); - rewrite_user_facing_conditions(&mut status, None); - if supervisor_promoted { - ensure_supervisor_ready_status(&mut status, &sandbox_name); - } - let workspace = incoming.workspace.clone(); - let mut sandbox = Sandbox { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: incoming.id.clone(), - name: sandbox_name, - created_at_ms: now_ms, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace, - deletion_timestamp_ms: 0, - }), - spec: None, - status, - }; - sandbox.set_phase(phase as i32); - - let labels_map = sandbox.object_labels(); - let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { - None - } else { - Some( - serde_json::to_string(&labels_map) - .map_err(|e| format!("failed to serialize labels: {e}"))?, - ) - }; - self.store - .put_if( - Sandbox::object_type(), - &incoming.id, - sandbox.object_name(), - sandbox.object_workspace(), - &sandbox.encode_to_vec(), - labels_json.as_deref(), - WriteCondition::MustCreate, - ) - .await - .map_err(|e| match e { - crate::persistence::PersistenceError::Conflict { - current_resource_version, - } => format!( - "concurrent modification detected during sandbox creation (current resource_version: {})", - current_resource_version - .map_or_else(|| "unknown".to_string(), |v| v.to_string()) - ), - other => other.to_string(), - })?; + let Some(existing_record) = existing_record else { + // The gateway store is authoritative. API creation persists the + // complete row before asking the driver to create its resource, so + // an unknown snapshot is either stale or unmanaged. + debug!( + sandbox_id = %incoming.id, + sandbox_name = %incoming.name, + "Ignoring driver snapshot for sandbox absent from gateway store" + ); + return Ok(()); + }; + let existing = decode_sandbox_record(&existing_record)?; - self.sandbox_index.update_from_sandbox(&sandbox); - self.sandbox_watch_bus.notify(sandbox.object_id()); + if SandboxPhase::try_from(existing.phase()).unwrap_or(SandboxPhase::Unknown) + == SandboxPhase::Deleting + { + // Ordinary driver snapshots cannot recover or regress a durable + // deletion. Delete-failure recovery is explicit and version-bound. return Ok(()); } // Single-attempt CAS: on conflict, the next watch event will naturally retry let session_connected = self.supervisor_sessions.has_session(&incoming.id); - let sandbox_name = incoming.name.clone(); - let sandbox = self .store - .update_message_cas::(&incoming.id, 0, |sandbox| { - let old_phase = - SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); - let mut phase = incoming - .status - .as_ref() - .map_or(old_phase, |status| derive_phase(Some(status))); - let supervisor_promoted = session_connected - && matches!(phase, SandboxPhase::Provisioning | SandboxPhase::Unknown); - if supervisor_promoted { - phase = SandboxPhase::Ready; - } - - let cpv = sandbox.current_policy_version(); - let mut status = incoming - .status - .as_ref() - .map(|s| public_status_from_driver(s, phase, cpv)) - .or_else(|| sandbox.status.clone()); - rewrite_user_facing_conditions(&mut status, sandbox.spec.as_ref()); - if supervisor_promoted { - ensure_supervisor_ready_status(&mut status, &sandbox_name); - } - - if let Some(s) = status.as_mut() - && s.sandbox_name.is_empty() - { - s.sandbox_name.clone_from(&sandbox_name); - } - - if old_phase != phase { - info!( - sandbox_id = %incoming.id, - sandbox_name = %sandbox_name, - old_phase = ?old_phase, - new_phase = ?phase, - "Sandbox phase changed" - ); - } - - if phase == SandboxPhase::Error - && let Some(ref status) = status - { - for condition in &status.conditions { - if condition.r#type == "Ready" - && condition.status.eq_ignore_ascii_case("false") - && is_terminal_failure_reason(&condition.reason) - { - warn!( - sandbox_id = %incoming.id, - sandbox_name = %sandbox_name, - reason = %condition.reason, - message = %condition.message, - "Sandbox failed to become ready" - ); - } - } - } - - // Update metadata fields - if let Some(metadata) = sandbox.metadata.as_mut() { - metadata.name.clone_from(&sandbox_name); - } - sandbox.status = status; - sandbox.set_phase(phase as i32); - sandbox.set_current_policy_version(cpv); - }) + .update_message_cas::( + &incoming.id, + existing_record.resource_version, + |sandbox| apply_driver_snapshot(sandbox, &incoming, session_connected), + ) .await .map_err(|e| match e { crate::persistence::PersistenceError::Conflict { @@ -1350,23 +1727,33 @@ impl ComputeRuntime { ) -> Result<(), String> { let _guard = self.sync_lock.lock().await; + let Some(existing) = self + .store + .get_message::(sandbox_id) + .await + .map_err(|err| err.to_string())? + else { + return Ok(()); + }; + let current_phase = + SandboxPhase::try_from(existing.phase()).unwrap_or(SandboxPhase::Unknown); + if current_phase == SandboxPhase::Deleting || current_phase == SandboxPhase::Error { + return Ok(()); + } + if !connected && current_phase != SandboxPhase::Ready { + return Ok(()); + } + let expected_resource_version = sandbox_resource_version(&existing); + // Use CAS to update sandbox phase based on supervisor session state let result = self .store - .update_message_cas::(sandbox_id, 0, |sandbox| { - let current_phase = - SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); - - // Skip if sandbox is in terminal state - if current_phase == SandboxPhase::Deleting || current_phase == SandboxPhase::Error { - return; - } - + .update_message_cas::(sandbox_id, expected_resource_version, |sandbox| { let sandbox_name = sandbox.object_name().to_string(); if connected { ensure_supervisor_ready_status(&mut sandbox.status, &sandbox_name); sandbox.set_phase(SandboxPhase::Ready as i32); - } else if current_phase == SandboxPhase::Ready { + } else { ensure_supervisor_not_ready_status(&mut sandbox.status, &sandbox_name); sandbox.set_phase(SandboxPhase::Provisioning as i32); } @@ -1418,9 +1805,18 @@ impl ComputeRuntime { .delete(Sandbox::object_type(), sandbox_id) .await .map_err(|e| e.to_string())?; - self.sandbox_index.remove_sandbox(sandbox_id); - self.sandbox_watch_bus.notify(sandbox_id); - self.cleanup_sandbox_state(sandbox_id); + self.cleanup_removed_sandbox_state(sandbox_id); + Ok(()) + } + + async fn apply_deleted_if_version_locked( + &self, + sandbox: &Sandbox, + expected_resource_version: u64, + ) -> Result<(), String> { + let sandbox_id = sandbox.object_id(); + self.remove_sandbox_record_if_version_locked(sandbox_id, expected_resource_version) + .await?; Ok(()) } @@ -1508,6 +1904,29 @@ impl ComputeRuntime { Ok(()) } + async fn cleanup_local_state_if_sandbox_absent( + &self, + delete_guard: &SandboxDeleteGuard, + sandbox_id: &str, + ) -> Result<(), Status> { + let _guard = self.lock_global_for_delete(delete_guard).await; + let record = self + .store + .get(Sandbox::object_type(), sandbox_id) + .await + .map_err(|err| Status::internal(format!("fetch sandbox failed: {err}")))?; + if record.is_none() { + self.cleanup_removed_sandbox_state(sandbox_id); + } + Ok(()) + } + + fn cleanup_removed_sandbox_state(&self, sandbox_id: &str) { + self.sandbox_index.remove_sandbox(sandbox_id); + self.sandbox_watch_bus.notify(sandbox_id); + self.cleanup_sandbox_state(sandbox_id); + } + fn cleanup_sandbox_state(&self, sandbox_id: &str) { self.tracing_log_bus.remove(sandbox_id); self.tracing_log_bus.platform_event_bus.remove(sandbox_id); @@ -1519,6 +1938,30 @@ impl ComputeRuntime { snapshot: DriverSandbox, sweep_started_at_ms: i64, ) -> Result<(), String> { + let expected_resource_version = { + let _guard = self.sync_lock.lock().await; + let Some(existing) = self + .store + .get(Sandbox::object_type(), &snapshot.id) + .await + .map_err(|e| e.to_string())? + else { + return Ok(()); + }; + + if existing.updated_at_ms > sweep_started_at_ms { + return Ok(()); + } + existing.resource_version + }; + + let Some(current) = self + .get_driver_sandbox(&snapshot.id, &snapshot.name) + .await? + else { + return Ok(()); + }; + let _guard = self.sync_lock.lock().await; let Some(existing) = self .store @@ -1528,18 +1971,12 @@ impl ComputeRuntime { else { return Ok(()); }; - - if existing.updated_at_ms > sweep_started_at_ms { + if existing.resource_version != expected_resource_version + || existing.updated_at_ms > sweep_started_at_ms + { return Ok(()); } - let Some(current) = self - .get_driver_sandbox(&snapshot.id, &snapshot.name) - .await? - else { - return Ok(()); - }; - self.apply_sandbox_update_locked(current, Some(existing)) .await } @@ -1550,42 +1987,68 @@ impl ComputeRuntime { sweep_started_at_ms: i64, grace_ms: i64, ) -> Result<(), String> { + let (sandbox_id, sandbox_name, expected_resource_version, age_ms) = { + let _guard = self.sync_lock.lock().await; + let Some(current_record) = self + .store + .get(Sandbox::object_type(), &record.id) + .await + .map_err(|e| e.to_string())? + else { + return Ok(()); + }; + + if current_record.updated_at_ms > sweep_started_at_ms { + return Ok(()); + } + + let sandbox = decode_sandbox_record(¤t_record)?; + let age_ms = + openshell_core::time::now_ms().saturating_sub(current_record.created_at_ms); + if age_ms < grace_ms { + return Ok(()); + } + + ( + sandbox.object_id().to_string(), + sandbox.object_name().to_string(), + current_record.resource_version, + age_ms, + ) + }; + + let current = self.get_driver_sandbox(&sandbox_id, &sandbox_name).await?; + let _guard = self.sync_lock.lock().await; let Some(current_record) = self .store - .get(Sandbox::object_type(), &record.id) + .get(Sandbox::object_type(), &sandbox_id) .await .map_err(|e| e.to_string())? else { return Ok(()); }; - - if current_record.updated_at_ms > sweep_started_at_ms { - return Ok(()); - } - - let sandbox = decode_sandbox_record(¤t_record)?; - let age_ms = openshell_core::time::now_ms().saturating_sub(current_record.created_at_ms); - if age_ms < grace_ms { + if current_record.resource_version != expected_resource_version + || current_record.updated_at_ms > sweep_started_at_ms + { return Ok(()); } - if let Some(current) = self - .get_driver_sandbox(sandbox.object_id(), sandbox.object_name()) - .await? - { + if let Some(current) = current { return self .apply_sandbox_update_locked(current, Some(current_record)) .await; } + let sandbox = decode_sandbox_record(¤t_record)?; info!( - sandbox_id = %sandbox.object_id(), - sandbox_name = %sandbox.object_name(), + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, age_secs = age_ms / 1000, "Removing sandbox from store after it disappeared from the compute driver snapshot" ); - self.apply_deleted_locked(sandbox.object_id()).await + self.apply_deleted_if_version_locked(&sandbox, expected_resource_version) + .await } async fn get_driver_sandbox( @@ -1601,7 +2064,18 @@ impl ComputeRuntime { })) .await { - Ok(response) => Ok(response.into_inner().sandbox), + Ok(response) => { + let sandbox = response.into_inner().sandbox; + if let Some(sandbox) = sandbox.as_ref() + && sandbox.id != sandbox_id + { + return Err(format!( + "compute driver returned sandbox '{}' for requested id '{sandbox_id}'", + sandbox.id + )); + } + Ok(sandbox) + } Err(status) if status.code() == Code::NotFound => Ok(None), Err(status) => Err(status.to_string()), } @@ -1933,6 +2407,13 @@ fn decode_sandbox_record(record: &ObjectRecord) -> Result { Sandbox::decode(record.payload.as_slice()).map_err(|e| e.to_string()) } +fn sandbox_resource_version(sandbox: &Sandbox) -> u64 { + sandbox + .metadata + .as_ref() + .map_or(0, |metadata| metadata.resource_version) +} + fn public_status_from_driver( status: &DriverSandboxStatus, phase: SandboxPhase, @@ -1953,6 +2434,73 @@ fn public_status_from_driver( } } +fn apply_driver_snapshot(sandbox: &mut Sandbox, incoming: &DriverSandbox, session_connected: bool) { + let old_phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); + let mut phase = incoming + .status + .as_ref() + .map_or(old_phase, |status| derive_phase(Some(status))); + let sandbox_name = &incoming.name; + let supervisor_promoted = + session_connected && matches!(phase, SandboxPhase::Provisioning | SandboxPhase::Unknown); + if supervisor_promoted { + phase = SandboxPhase::Ready; + } + + let cpv = sandbox.current_policy_version(); + let mut status = incoming + .status + .as_ref() + .map(|status| public_status_from_driver(status, phase, cpv)) + .or_else(|| sandbox.status.clone()); + rewrite_user_facing_conditions(&mut status, sandbox.spec.as_ref()); + if supervisor_promoted { + ensure_supervisor_ready_status(&mut status, sandbox_name); + } + + if let Some(status) = status.as_mut() + && status.sandbox_name.is_empty() + { + status.sandbox_name.clone_from(sandbox_name); + } + + if old_phase != phase { + info!( + sandbox_id = %incoming.id, + sandbox_name = %sandbox_name, + old_phase = ?old_phase, + new_phase = ?phase, + "Sandbox phase changed" + ); + } + + if phase == SandboxPhase::Error + && let Some(ref status) = status + { + for condition in &status.conditions { + if condition.r#type == "Ready" + && condition.status.eq_ignore_ascii_case("false") + && is_terminal_failure_reason(&condition.reason) + { + warn!( + sandbox_id = %incoming.id, + sandbox_name = %sandbox_name, + reason = %condition.reason, + message = %condition.message, + "Sandbox failed to become ready" + ); + } + } + } + + if let Some(metadata) = sandbox.metadata.as_mut() { + metadata.name.clone_from(sandbox_name); + } + sandbox.status = status; + sandbox.set_phase(phase as i32); + sandbox.set_current_policy_version(cpv); +} + fn ensure_supervisor_ready_status(status: &mut Option, sandbox_name: &str) { upsert_ready_condition( status, @@ -2212,6 +2760,7 @@ pub async fn new_test_runtime(store: Arc) -> ComputeRuntime { tracing_log_bus: TracingLogBus::new(), supervisor_sessions: Arc::new(SupervisorSessionRegistry::new()), sync_lock: Arc::new(Mutex::new(())), + delete_gates: Arc::new(DeleteGateRegistry::default()), gateway_bind_addresses: Vec::new(), replica_id: "test-replica".to_string(), } @@ -2224,10 +2773,13 @@ mod tests { use openshell_core::proto::compute::v1::{ CreateSandboxResponse, DeleteSandboxResponse, GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateResponse, + WatchSandboxesDeletedEvent, WatchSandboxesSandboxEvent, }; use std::collections::HashMap; - use std::sync::Arc; - use tokio::sync::{mpsc, oneshot}; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex as TestMutex}; + use tokio::sync::{Notify, Semaphore, mpsc, oneshot}; + use tokio_stream::wrappers::UnboundedReceiverStream; fn string_value(value: &str) -> prost_types::Value { prost_types::Value { @@ -2445,6 +2997,212 @@ mod tests { } } + #[derive(Clone)] + enum ControlledDeleteOutcome { + Ok(bool), + Error(&'static str), + } + + #[derive(Clone)] + enum ControlledGetOutcome { + Sandbox(Box), + Missing, + Error(&'static str), + } + + struct ControlledDriver { + watch_tx: mpsc::UnboundedSender>, + watch_rx: TestMutex>>>, + watch_started: Notify, + delete_started: Notify, + delete_release: Semaphore, + delete_blocked: AtomicBool, + delete_calls: AtomicUsize, + delete_outcome: TestMutex, + get_started: Notify, + get_release: Semaphore, + get_blocked: AtomicBool, + get_outcome: TestMutex, + } + + impl ControlledDriver { + fn new() -> Arc { + let (watch_tx, watch_rx) = mpsc::unbounded_channel(); + Arc::new(Self { + watch_tx, + watch_rx: TestMutex::new(Some(watch_rx)), + watch_started: Notify::new(), + delete_started: Notify::new(), + delete_release: Semaphore::new(0), + delete_blocked: AtomicBool::new(false), + delete_calls: AtomicUsize::new(0), + delete_outcome: TestMutex::new(ControlledDeleteOutcome::Ok(true)), + get_started: Notify::new(), + get_release: Semaphore::new(0), + get_blocked: AtomicBool::new(false), + get_outcome: TestMutex::new(ControlledGetOutcome::Missing), + }) + } + + fn block_delete(&self) { + self.delete_blocked.store(true, Ordering::SeqCst); + } + + fn release_delete(&self) { + self.delete_release.add_permits(1); + } + + fn block_get(&self) { + self.get_blocked.store(true, Ordering::SeqCst); + } + + fn release_get(&self) { + self.get_release.add_permits(1); + } + + fn set_delete_outcome(&self, outcome: ControlledDeleteOutcome) { + *self + .delete_outcome + .lock() + .expect("delete outcome lock poisoned") = outcome; + } + + fn set_get_outcome(&self, outcome: ControlledGetOutcome) { + *self.get_outcome.lock().expect("get outcome lock poisoned") = outcome; + } + + fn delete_calls(&self) -> usize { + self.delete_calls.load(Ordering::SeqCst) + } + + fn send_event(&self, event: WatchSandboxesEvent) { + self.watch_tx + .send(Ok(event)) + .expect("watch loop should still be receiving events"); + } + } + + #[tonic::async_trait] + impl ComputeDriver for ControlledDriver { + type WatchSandboxesStream = DriverWatchStream; + + async fn get_capabilities( + &self, + _request: Request, + ) -> Result, Status> { + Ok(tonic::Response::new(GetCapabilitiesResponse { + driver_name: "controlled-test-driver".to_string(), + driver_version: "test".to_string(), + default_image: "openshell/sandbox:test".to_string(), + })) + } + + async fn validate_sandbox_create( + &self, + _request: Request, + ) -> Result, Status> { + Ok(tonic::Response::new(ValidateSandboxCreateResponse {})) + } + + async fn get_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + self.get_started.notify_one(); + if self.get_blocked.load(Ordering::SeqCst) { + self.get_release + .acquire() + .await + .expect("get release semaphore closed") + .forget(); + } + let outcome = self + .get_outcome + .lock() + .expect("get outcome lock poisoned") + .clone(); + match outcome { + ControlledGetOutcome::Sandbox(sandbox) => { + Ok(tonic::Response::new(GetSandboxResponse { + sandbox: Some(*sandbox), + })) + } + ControlledGetOutcome::Missing => Err(Status::not_found("sandbox not found")), + ControlledGetOutcome::Error(message) => Err(Status::internal(message)), + } + } + + async fn list_sandboxes( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::ListSandboxesResponse { + sandboxes: Vec::new(), + }, + )) + } + + async fn create_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + Ok(tonic::Response::new(CreateSandboxResponse {})) + } + + async fn stop_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + Ok(tonic::Response::new(StopSandboxResponse {})) + } + + async fn delete_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + self.delete_calls.fetch_add(1, Ordering::SeqCst); + self.delete_started.notify_one(); + if self.delete_blocked.load(Ordering::SeqCst) { + self.delete_release + .acquire() + .await + .expect("delete release semaphore closed") + .forget(); + } + let outcome = self + .delete_outcome + .lock() + .expect("delete outcome lock poisoned") + .clone(); + match outcome { + ControlledDeleteOutcome::Ok(deleted) => { + Ok(tonic::Response::new(DeleteSandboxResponse { deleted })) + } + ControlledDeleteOutcome::Error(message) => Err(Status::internal(message)), + } + } + + async fn watch_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + let receiver = self + .watch_rx + .lock() + .expect("watch receiver lock poisoned") + .take() + .ok_or_else(|| Status::failed_precondition("watch already started"))?; + self.watch_started.notify_one(); + Ok(tonic::Response::new(Box::pin( + UnboundedReceiverStream::new(receiver), + ))) + } + } + async fn test_runtime(driver: SharedComputeDriver) -> ComputeRuntime { test_runtime_with_resume(driver, None).await } @@ -2471,6 +3229,7 @@ mod tests { tracing_log_bus: TracingLogBus::new(), supervisor_sessions: Arc::new(SupervisorSessionRegistry::new()), sync_lock: Arc::new(Mutex::new(())), + delete_gates: Arc::new(DeleteGateRegistry::default()), gateway_bind_addresses: Vec::new(), replica_id: "test-replica".to_string(), } @@ -2524,19 +3283,156 @@ mod tests { } } - fn make_driver_condition(reason: &str, message: &str) -> DriverCondition { - DriverCondition { - r#type: "Ready".to_string(), - status: "False".to_string(), - reason: reason.to_string(), - message: message.to_string(), - last_transition_time: String::new(), + fn service_endpoint_record(id: &str, sandbox: &Sandbox) -> ServiceEndpoint { + ServiceEndpoint { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: format!("{}--web", sandbox.object_name()), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: sandbox.object_workspace().to_string(), + deletion_timestamp_ms: 0, + }), + sandbox_id: sandbox.object_id().to_string(), + sandbox_name: sandbox.object_name().to_string(), + service_name: "web".to_string(), + target_port: 8080, + domain: true, } } - fn make_driver_status(condition: DriverCondition) -> DriverSandboxStatus { - DriverSandboxStatus { - sandbox_name: "test".to_string(), + async fn seed_sandbox_owned_records(runtime: &ComputeRuntime, sandbox: &Sandbox) -> SshSession { + runtime + .store + .put( + SANDBOX_SETTINGS_OBJECT_TYPE, + &format!("settings-{}", sandbox.object_id()), + sandbox.object_name(), + sandbox.object_workspace(), + br#"{"revision":1,"settings":{}}"#, + None, + ) + .await + .unwrap(); + let session = ssh_session_record("owned", sandbox.object_id()); + runtime.store.put_message(&session).await.unwrap(); + + let endpoint = service_endpoint_record("endpoint-owned", sandbox); + runtime.store.put_message(&endpoint).await.unwrap(); + + runtime + .store + .put_scoped( + POLICY_OBJECT_TYPE, + "policy-owned", + "policy-owned", + sandbox.object_workspace(), + sandbox.object_id(), + br#"{"version":1}"#, + None, + ) + .await + .unwrap(); + runtime + .store + .put_scoped( + DRAFT_CHUNK_OBJECT_TYPE, + "draft-owned", + "draft-owned", + sandbox.object_workspace(), + sandbox.object_id(), + br#"{"chunk":1}"#, + None, + ) + .await + .unwrap(); + session + } + + async fn remove_sandbox_owned_records_from_store(runtime: &ComputeRuntime, sandbox: &Sandbox) { + runtime + .cleanup_sandbox_owned_records(sandbox) + .await + .unwrap(); + runtime + .store + .delete(Sandbox::object_type(), sandbox.object_id()) + .await + .unwrap(); + } + + async fn assert_sandbox_owned_records( + runtime: &ComputeRuntime, + sandbox: &Sandbox, + session: &SshSession, + expected: bool, + ) { + assert_eq!( + runtime + .store + .get_by_name( + SANDBOX_SETTINGS_OBJECT_TYPE, + sandbox.object_workspace(), + sandbox.object_name(), + ) + .await + .unwrap() + .is_some(), + expected + ); + assert_eq!( + runtime + .store + .get_message::(session.object_id()) + .await + .unwrap() + .is_some(), + expected + ); + assert_eq!( + runtime + .store + .get_message::("endpoint-owned") + .await + .unwrap() + .is_some(), + expected + ); + assert_eq!( + runtime + .store + .get(POLICY_OBJECT_TYPE, "policy-owned") + .await + .unwrap() + .is_some(), + expected + ); + assert_eq!( + runtime + .store + .get(DRAFT_CHUNK_OBJECT_TYPE, "draft-owned") + .await + .unwrap() + .is_some(), + expected + ); + } + + fn make_driver_condition(reason: &str, message: &str) -> DriverCondition { + DriverCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: reason.to_string(), + message: message.to_string(), + last_transition_time: String::new(), + } + } + + fn make_driver_status(condition: DriverCondition) -> DriverSandboxStatus { + DriverSandboxStatus { + sandbox_name: "test".to_string(), instance_id: "test-pod".to_string(), agent_fd: String::new(), sandbox_fd: String::new(), @@ -2545,12 +3441,91 @@ mod tests { } } + fn ready_driver_sandbox(id: &str, name: &str) -> DriverSandbox { + DriverSandbox { + id: id.to_string(), + name: name.to_string(), + namespace: "default".to_string(), + workspace: "default".to_string(), + spec: None, + status: Some(DriverSandboxStatus { + sandbox_name: name.to_string(), + instance_id: format!("{name}-pod"), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![DriverCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "DependenciesReady".to_string(), + message: "Sandbox is ready".to_string(), + last_transition_time: String::new(), + }], + deleting: false, + }), + } + } + + fn sandbox_watch_event(sandbox: DriverSandbox) -> WatchSandboxesEvent { + WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + } + } + + fn deleted_watch_event(sandbox_id: &str) -> WatchSandboxesEvent { + WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Deleted( + WatchSandboxesDeletedEvent { + sandbox_id: sandbox_id.to_string(), + }, + )), + } + } + + async fn start_watch_loop( + runtime: &ComputeRuntime, + driver: &ControlledDriver, + ) -> (watch::Sender, tokio::task::JoinHandle<()>) { + let (shutdown_tx, shutdown_rx) = watch::channel(false); + let runtime = Arc::new(runtime.clone()); + let handle = tokio::spawn(async move { runtime.watch_loop(shutdown_rx).await }); + tokio::time::timeout(Duration::from_secs(1), driver.watch_started.notified()) + .await + .expect("watch loop did not start"); + (shutdown_tx, handle) + } + + async fn stop_watch_loop( + shutdown_tx: watch::Sender, + handle: tokio::task::JoinHandle<()>, + ) { + shutdown_tx.send(true).unwrap(); + tokio::time::timeout(Duration::from_secs(1), handle) + .await + .expect("watch loop did not stop") + .unwrap(); + } + #[tokio::test] async fn sqlite_store_is_single_replica() { let store = Arc::new(Store::connect("sqlite::memory:").await.unwrap()); assert!(store.is_single_replica()); } + #[test] + fn delete_gate_registry_removes_stale_entries() { + let registry = DeleteGateRegistry::default(); + let first = registry.gate_for("sb-1"); + assert_eq!(registry.entry_count(), 1); + drop(first); + + let _second = registry.gate_for("sb-2"); + assert_eq!(registry.entry_count(), 1); + } + #[test] fn terminal_failure_treats_unknown_reasons_as_terminal() { let terminal_cases = [ @@ -2810,85 +3785,1095 @@ mod tests { }), ); - let message = &status.unwrap().conditions[0].message; - assert_eq!( - message, - "GPU sandbox could not be scheduled on the active gateway. Another GPU sandbox may already be using the available GPU, or the gateway may not currently be able to satisfy GPU placement. Please refer to documentation and use `openshell doctor` commands to inspect GPU support and gateway configuration." + let message = &status.unwrap().conditions[0].message; + assert_eq!( + message, + "GPU sandbox could not be scheduled on the active gateway. Another GPU sandbox may already be using the available GPU, or the gateway may not currently be able to satisfy GPU placement. Please refer to documentation and use `openshell doctor` commands to inspect GPU support and gateway configuration." + ); + } + + #[test] + fn rewrite_user_facing_conditions_leaves_non_gpu_unschedulable_message_unchanged() { + let original = "0/1 nodes are available: 1 Insufficient cpu."; + let mut status = Some(SandboxStatus { + sandbox_name: "test".to_string(), + agent_pod: "test-pod".to_string(), + conditions: vec![SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: "Unschedulable".to_string(), + message: original.to_string(), + last_transition_time: String::new(), + }], + ..Default::default() + }); + + rewrite_user_facing_conditions(&mut status, Some(&SandboxSpec::default())); + + assert_eq!(status.unwrap().conditions[0].message, original); + } + + #[test] + fn compute_error_from_status_preserves_driver_status_codes() { + assert!(matches!( + compute_error_from_status(Status::already_exists("sandbox already exists")), + ComputeError::AlreadyExists + )); + + assert!(matches!( + compute_error_from_status(Status::failed_precondition("sandbox agent pod IP is not available")), + ComputeError::Precondition(message) if message == "sandbox agent pod IP is not available" + )); + } + + #[tokio::test] + async fn begin_sandbox_delete_retries_after_stale_snapshot_conflict() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let stale_snapshot = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + + runtime + .store + .update_message_cas::("sb-1", 0, |sandbox| { + sandbox.set_current_policy_version(7); + }) + .await + .unwrap(); + + let transition = runtime + .begin_sandbox_delete_with_initial_snapshot("sb-1", Some(stale_snapshot)) + .await + .unwrap(); + let BeginDelete::Started(transition) = transition else { + panic!("expected a new delete transition"); + }; + let transition = *transition; + let updated = transition.deleting; + + assert_eq!( + SandboxPhase::try_from(updated.phase()).unwrap(), + SandboxPhase::Deleting + ); + assert_eq!(updated.current_policy_version(), 7); + assert_eq!( + updated + .metadata + .as_ref() + .map_or(0, |metadata| metadata.resource_version), + 3 + ); + assert_eq!(transition.previous.current_policy_version(), 7); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Deleting + ); + assert_eq!(stored.current_policy_version(), 7); + } + + #[tokio::test] + async fn apply_sandbox_update_is_noop_while_deleting() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Deleting); + runtime.store.put_message(&sandbox).await.unwrap(); + let before = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + runtime + .apply_sandbox_update(DriverSandbox { + id: "sb-1".to_string(), + name: "sandbox-a".to_string(), + namespace: "default".to_string(), + spec: None, + status: Some(DriverSandboxStatus { + sandbox_name: "sandbox-a".to_string(), + instance_id: "agent-pod".to_string(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![DriverCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "DependenciesReady".to_string(), + message: "Pod is Ready".to_string(), + last_transition_time: String::new(), + }], + deleting: false, + }), + workspace: "default".to_string(), + }) + .await + .unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Deleting + ); + assert_eq!( + sandbox_resource_version(&stored), + sandbox_resource_version(&before) + ); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Empty) + )); + } + + #[tokio::test] + async fn supervisor_session_change_is_noop_while_deleting() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Deleting); + runtime.store.put_message(&sandbox).await.unwrap(); + let before = runtime + .store + .get(Sandbox::object_type(), "sb-1") + .await + .unwrap() + .unwrap(); + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + runtime + .supervisor_session_disconnected("sb-1") + .await + .unwrap(); + + let after = runtime + .store + .get(Sandbox::object_type(), "sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!(after.resource_version, before.resource_version); + assert_eq!(after.payload, before.payload); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Empty) + )); + } + + #[tokio::test] + async fn unknown_watch_snapshot_does_not_create_gateway_state() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-unknown"); + + runtime + .apply_sandbox_update(ready_driver_sandbox("sb-unknown", "unmanaged")) + .await + .unwrap(); + + assert!( + runtime + .store + .get_message::("sb-unknown") + .await + .unwrap() + .is_none() + ); + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "unmanaged") + .is_none() + ); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Empty) + )); + } + + #[tokio::test] + async fn watcher_update_preserves_complete_api_created_spec() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + sandbox.spec = Some(SandboxSpec { + log_level: "debug".to_string(), + template: Some(SandboxTemplate { + image: "example.test/sandbox:complete".to_string(), + ..Default::default() + }), + ..Default::default() + }); + + runtime.create_sandbox(sandbox, None).await.unwrap(); + runtime + .apply_sandbox_update(ready_driver_sandbox("sb-1", "sandbox-a")) + .await + .unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + stored + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .map(|template| template.image.as_str()), + Some("example.test/sandbox:complete") + ); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); + } + + #[tokio::test] + async fn blocked_delete_does_not_delay_unrelated_deleted_watch_event() { + let driver = ControlledDriver::new(); + driver.block_delete(); + let runtime = test_runtime(driver.clone()).await; + + for sandbox in [ + sandbox_record("sb-a", "sandbox-a", SandboxPhase::Ready), + sandbox_record("sb-b", "sandbox-b", SandboxPhase::Ready), + ] { + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + } + + let (shutdown_tx, watch_handle) = start_watch_loop(&runtime, &driver).await; + let mut sandbox_b_rx = runtime.sandbox_watch_bus.subscribe("sb-b"); + let delete_runtime = runtime.clone(); + let delete_handle = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + + let before = runtime + .store + .get(Sandbox::object_type(), "sb-a") + .await + .unwrap() + .unwrap(); + let mut stale_snapshot = ready_driver_sandbox("sb-a", "sandbox-a"); + stale_snapshot.status = Some(make_driver_status(make_driver_condition( + "ContainerExited", + "container exited during deletion", + ))); + driver.send_event(sandbox_watch_event(stale_snapshot)); + driver.send_event(deleted_watch_event("sb-b")); + + tokio::time::timeout(Duration::from_secs(1), sandbox_b_rx.recv()) + .await + .expect("sandbox B event was blocked by sandbox A deletion") + .expect("sandbox B watch bus closed before notification"); + assert!( + runtime + .store + .get_message::("sb-b") + .await + .unwrap() + .is_none() + ); + let after = runtime + .store + .get(Sandbox::object_type(), "sb-a") + .await + .unwrap() + .unwrap(); + assert_eq!(after.resource_version, before.resource_version); + assert_eq!(after.payload, before.payload); + + driver.release_delete(); + assert!( + tokio::time::timeout(Duration::from_secs(1), delete_handle) + .await + .expect("delete did not finish") + .unwrap() + .unwrap() + .deleted + ); + stop_watch_loop(shutdown_tx, watch_handle).await; + } + + #[tokio::test] + async fn concurrent_duplicate_deletes_call_driver_once() { + let driver = ControlledDriver::new(); + driver.block_delete(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let first_runtime = runtime.clone(); + let first = + tokio::spawn(async move { first_runtime.delete_sandbox("default", "sandbox-a").await }); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("first delete did not reach the driver"); + + let second_runtime = runtime.clone(); + let second = + tokio::spawn( + async move { second_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + driver.release_delete(); + + assert!( + tokio::time::timeout(Duration::from_secs(1), first) + .await + .unwrap() + .unwrap() + .unwrap() + .deleted + ); + assert!( + tokio::time::timeout(Duration::from_secs(1), second) + .await + .unwrap() + .unwrap() + .unwrap() + .deleted + ); + assert_eq!(driver.delete_calls(), 1); + } + + #[tokio::test] + async fn waiting_delete_retries_after_leader_failure_recovery() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Error("lookup failed")); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_gate = runtime.delete_gates.gate_for(sandbox.object_id()); + let first_runtime = runtime.clone(); + let first = + tokio::spawn(async move { first_runtime.delete_sandbox("default", "sandbox-a").await }); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("first delete did not reach the driver"); + + let second_runtime = runtime.clone(); + let second = + tokio::spawn( + async move { second_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), async { + while Arc::strong_count(&delete_gate) < 3 { + tokio::task::yield_now().await; + } + }) + .await + .expect("second delete did not start waiting on the sandbox gate"); + + driver.release_delete(); + let first_error = tokio::time::timeout(Duration::from_secs(1), first) + .await + .expect("first delete did not finish") + .unwrap() + .unwrap_err(); + assert!(first_error.message().contains("delete failed")); + + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("waiting delete did not retry the driver call"); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + driver.release_delete(); + + assert!( + !tokio::time::timeout(Duration::from_secs(1), second) + .await + .expect("second delete did not finish") + .unwrap() + .unwrap() + .deleted + ); + assert_eq!(driver.delete_calls(), 2); + assert!( + runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn canceled_waiting_delete_does_not_retry_after_leader_failure() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Error("lookup failed")); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_gate = runtime.delete_gates.gate_for(sandbox.object_id()); + let first_runtime = runtime.clone(); + let first = + tokio::spawn(async move { first_runtime.delete_sandbox("default", "sandbox-a").await }); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("first delete did not reach the driver"); + + let second_runtime = runtime.clone(); + let second = + tokio::spawn( + async move { second_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), async { + while Arc::strong_count(&delete_gate) < 3 { + tokio::task::yield_now().await; + } + }) + .await + .expect("second delete did not start waiting on the sandbox gate"); + + second.abort(); + assert!(second.await.unwrap_err().is_cancelled()); + driver.release_delete(); + + let first_error = tokio::time::timeout(Duration::from_secs(1), first) + .await + .expect("first delete did not finish") + .unwrap() + .unwrap_err(); + assert!(first_error.message().contains("delete failed")); + assert!( + tokio::time::timeout(Duration::from_millis(100), driver.delete_started.notified()) + .await + .is_err(), + "canceled waiting delete unexpectedly reached the driver" + ); + assert_eq!(driver.delete_calls(), 1); + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); + } + + #[tokio::test] + async fn waiting_delete_does_not_retarget_a_reused_name() { + let driver = ControlledDriver::new(); + let runtime = test_runtime(driver.clone()).await; + let original = sandbox_record("sb-original", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&original).await.unwrap(); + + // Hold the original ID's gate so the request resolves the name and + // then waits before it can revalidate the durable row. + let delete_gate = runtime.delete_gates.gate_for(original.object_id()); + let delete_guard = delete_gate.lock().await; + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), async { + while Arc::strong_count(&delete_gate) < 2 { + tokio::task::yield_now().await; + } + }) + .await + .expect("delete did not start waiting on the original ID gate"); + + runtime + .store + .delete(Sandbox::object_type(), original.object_id()) + .await + .unwrap(); + let replacement = sandbox_record("sb-replacement", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&replacement).await.unwrap(); + drop(delete_guard); + + assert!(delete.await.unwrap().unwrap().deleted); + assert_eq!(driver.delete_calls(), 0); + assert!( + runtime + .store + .get_message::(replacement.object_id()) + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test] + async fn request_cancellation_does_not_cancel_the_delete_worker() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_runtime = runtime.clone(); + let request = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + + request.abort(); + assert!(request.await.unwrap_err().is_cancelled()); + driver.release_delete(); + + tokio::time::timeout(Duration::from_secs(1), async { + loop { + if runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("detached delete worker did not finish cleanup"); + assert_eq!(driver.delete_calls(), 1); + } + + #[tokio::test] + async fn already_absent_driver_resource_is_removed_synchronously() { + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + let runtime = test_runtime(driver).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + + assert!( + !runtime + .delete_sandbox("default", "sandbox-a") + .await + .unwrap() + .deleted + ); + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "sandbox-a") + .is_none() + ); + + runtime + .apply_sandbox_update(ready_driver_sandbox("sb-1", "sandbox-a")) + .await + .unwrap(); + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + } + + #[tokio::test] + async fn already_absent_delete_retries_cleanup_after_a_deleting_version_change() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + runtime + .store + .update_message_cas::("sb-1", 0, |sandbox| { + sandbox.set_current_policy_version(9); + }) + .await + .unwrap(); + + driver.release_delete(); + assert!(!delete.await.unwrap().unwrap().deleted); + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn already_absent_delete_reports_incomplete_gateway_cleanup() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + runtime + .store + .update_message_cas::("sb-1", 0, |sandbox| { + sandbox.set_phase(SandboxPhase::Ready as i32); + }) + .await + .unwrap(); + + driver.release_delete(); + let error = delete.await.unwrap().unwrap_err(); + assert_eq!(error.code(), Code::Internal); + assert_eq!( + SandboxPhase::try_from( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap() + .phase() + ) + .unwrap(), + SandboxPhase::Ready + ); + } + + #[tokio::test] + async fn accepted_driver_delete_leaves_removal_to_watcher() { + let driver = ControlledDriver::new(); + let runtime = test_runtime(driver).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + assert!( + runtime + .delete_sandbox("default", "sandbox-a") + .await + .unwrap() + .deleted + ); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Deleting + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, true).await; + assert!(watch_rx.try_recv().is_ok()); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Empty) + )); + + runtime.apply_deleted("sb-1").await.unwrap(); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!(watch_rx.try_recv().is_ok()); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Closed) + )); + } + + #[tokio::test] + async fn accepted_delete_cleans_local_state_after_another_replica_removes_row() { + let driver = ControlledDriver::new(); + driver.block_delete(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + tokio::time::timeout(Duration::from_secs(1), watch_rx.recv()) + .await + .expect("deleting notification was not sent") + .expect("watch bus closed before the deleting notification"); + + remove_sandbox_owned_records_from_store(&runtime, &sandbox).await; + driver.release_delete(); + + assert!( + tokio::time::timeout(Duration::from_secs(1), delete) + .await + .expect("delete did not finish") + .unwrap() + .unwrap() + .deleted + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "sandbox-a") + .is_none() + ); + assert!(watch_rx.try_recv().is_ok()); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Closed) + )); + } + + #[tokio::test] + async fn absent_delete_cleans_local_state_after_another_replica_removes_row() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + tokio::time::timeout(Duration::from_secs(1), watch_rx.recv()) + .await + .expect("deleting notification was not sent") + .expect("watch bus closed before the deleting notification"); + + remove_sandbox_owned_records_from_store(&runtime, &sandbox).await; + driver.release_delete(); + + assert!( + !tokio::time::timeout(Duration::from_secs(1), delete) + .await + .expect("delete did not finish") + .unwrap() + .unwrap() + .deleted + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "sandbox-a") + .is_none() + ); + assert!(watch_rx.try_recv().is_ok()); + assert!(matches!( + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Closed) + )); + } + + #[tokio::test] + async fn delete_error_with_absent_backend_removes_gateway_row() { + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + let runtime = test_runtime(driver).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + + runtime + .delete_sandbox("default", "sandbox-a") + .await + .unwrap_err(); + + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; } - #[test] - fn rewrite_user_facing_conditions_leaves_non_gpu_unschedulable_message_unchanged() { - let original = "0/1 nodes are available: 1 Insufficient cpu."; - let mut status = Some(SandboxStatus { - sandbox_name: "test".to_string(), - agent_pod: "test-pod".to_string(), - conditions: vec![SandboxCondition { - r#type: "Ready".to_string(), - status: "False".to_string(), - reason: "Unschedulable".to_string(), - message: original.to_string(), - last_transition_time: String::new(), - }], - ..Default::default() - }); + #[tokio::test] + async fn delete_error_cleans_local_state_after_another_replica_removes_row() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); - rewrite_user_facing_conditions(&mut status, Some(&SandboxSpec::default())); + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + tokio::time::timeout(Duration::from_secs(1), watch_rx.recv()) + .await + .expect("deleting notification was not sent") + .expect("watch bus closed before the deleting notification"); - assert_eq!(status.unwrap().conditions[0].message, original); - } + remove_sandbox_owned_records_from_store(&runtime, &sandbox).await; + driver.release_delete(); - #[test] - fn compute_error_from_status_preserves_driver_status_codes() { + tokio::time::timeout(Duration::from_secs(1), delete) + .await + .expect("delete did not finish") + .unwrap() + .unwrap_err(); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "sandbox-a") + .is_none() + ); + assert!(watch_rx.try_recv().is_ok()); assert!(matches!( - compute_error_from_status(Status::already_exists("sandbox already exists")), - ComputeError::AlreadyExists + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Closed) )); + } + + async fn assert_recovery_cleans_local_state_after_row_removed_during_lookup( + get_outcome: ControlledGetOutcome, + ) { + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(get_outcome); + driver.block_get(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + runtime.sandbox_index.update_from_sandbox(&sandbox); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + let mut watch_rx = runtime.sandbox_watch_bus.subscribe("sb-1"); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.get_started.notified()) + .await + .expect("delete recovery did not reach the driver lookup"); + tokio::time::timeout(Duration::from_secs(1), watch_rx.recv()) + .await + .expect("deleting notification was not sent") + .expect("watch bus closed before the deleting notification"); + + remove_sandbox_owned_records_from_store(&runtime, &sandbox).await; + driver.release_get(); + tokio::time::timeout(Duration::from_secs(1), delete) + .await + .expect("delete did not finish") + .unwrap() + .unwrap_err(); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .sandbox_index + .sandbox_id_for_sandbox_name("default", "sandbox-a") + .is_none() + ); + assert!(watch_rx.try_recv().is_ok()); assert!(matches!( - compute_error_from_status(Status::failed_precondition("sandbox agent pod IP is not available")), - ComputeError::Precondition(message) if message == "sandbox agent pod IP is not available" + watch_rx.try_recv(), + Err(tokio::sync::broadcast::error::TryRecvError::Closed) )); } #[tokio::test] - async fn set_sandbox_phase_deleting_retries_after_stale_snapshot_conflict() { - let runtime = test_runtime(Arc::new(TestDriver::default())).await; + async fn snapshot_recovery_cleans_local_state_after_another_replica_removes_row() { + assert_recovery_cleans_local_state_after_row_removed_during_lookup( + ControlledGetOutcome::Sandbox(Box::new(ready_driver_sandbox("sb-1", "sandbox-a"))), + ) + .await; + } + + #[tokio::test] + async fn rollback_recovery_cleans_local_state_after_another_replica_removes_row() { + assert_recovery_cleans_local_state_after_row_removed_during_lookup( + ControlledGetOutcome::Error("lookup failed"), + ) + .await; + } + + #[tokio::test] + async fn delete_error_recovers_from_current_driver_snapshot() { + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + ready_driver_sandbox("sb-1", "sandbox-a"), + ))); + let runtime = test_runtime(driver).await; + let mut sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + sandbox.spec = Some(SandboxSpec { + log_level: "debug".to_string(), + ..Default::default() + }); + runtime.store.put_message(&sandbox).await.unwrap(); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + + let error = runtime + .delete_sandbox("default", "sandbox-a") + .await + .unwrap_err(); + assert_eq!(error.code(), Code::Internal); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, true).await; + assert_eq!( + stored.spec.as_ref().map(|spec| spec.log_level.as_str()), + Some("debug") + ); + } + + #[tokio::test] + async fn delete_error_rolls_back_when_driver_lookup_fails() { + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Error("lookup failed")); + let runtime = test_runtime(driver).await; let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); runtime.store.put_message(&sandbox).await.unwrap(); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; - let stale_snapshot = runtime + runtime + .delete_sandbox("default", "sandbox-a") + .await + .unwrap_err(); + + let stored = runtime .store .get_message::("sb-1") .await .unwrap() .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, true).await; + } + + #[tokio::test] + async fn delete_error_recovery_does_not_overwrite_concurrent_cas_update() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + ready_driver_sandbox("sb-1", "sandbox-a"), + ))); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); runtime .store .update_message_cas::("sb-1", 0, |sandbox| { - sandbox.set_current_policy_version(7); + sandbox.set_current_policy_version(9); }) .await .unwrap(); - let updated = runtime - .set_sandbox_phase_deleting_with_initial_snapshot("sb-1", Some(stale_snapshot)) + driver.release_delete(); + tokio::time::timeout(Duration::from_secs(1), delete) .await - .unwrap(); - - assert_eq!( - SandboxPhase::try_from(updated.phase()).unwrap(), - SandboxPhase::Deleting - ); - assert_eq!(updated.current_policy_version(), 7); - assert_eq!( - updated - .metadata - .as_ref() - .map_or(0, |metadata| metadata.resource_version), - 3 - ); + .unwrap() + .unwrap() + .unwrap_err(); let stored = runtime .store @@ -2900,39 +4885,145 @@ mod tests { SandboxPhase::try_from(stored.phase()).unwrap(), SandboxPhase::Deleting ); - assert_eq!(stored.current_policy_version(), 7); + assert_eq!(stored.current_policy_version(), 9); } #[tokio::test] - async fn apply_sandbox_update_allows_delete_failures_to_recover() { - let runtime = test_runtime(Arc::new(TestDriver::default())).await; - let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Deleting); + async fn driver_completion_tolerates_watcher_removing_row_in_flight() { + let driver = ControlledDriver::new(); + driver.block_delete(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); runtime.store.put_message(&sandbox).await.unwrap(); - runtime - .apply_sandbox_update(DriverSandbox { - id: "sb-1".to_string(), - name: "sandbox-a".to_string(), - namespace: "default".to_string(), - spec: None, - status: Some(DriverSandboxStatus { - sandbox_name: "sandbox-a".to_string(), - instance_id: "agent-pod".to_string(), - agent_fd: String::new(), - sandbox_fd: String::new(), - conditions: vec![DriverCondition { - r#type: "Ready".to_string(), - status: "True".to_string(), - reason: "DependenciesReady".to_string(), - message: "Pod is Ready".to_string(), - last_transition_time: String::new(), - }], - deleting: false, - }), - workspace: "default".to_string(), - }) + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + runtime.apply_deleted("sb-1").await.unwrap(); + + driver.release_delete(); + assert!( + tokio::time::timeout(Duration::from_secs(1), delete) + .await + .unwrap() + .unwrap() + .unwrap() + .deleted + ); + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn delete_error_does_not_resurrect_row_removed_by_watcher() { + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("delete failed")); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + ready_driver_sandbox("sb-1", "sandbox-a"), + ))); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let delete_runtime = runtime.clone(); + let delete = + tokio::spawn( + async move { delete_runtime.delete_sandbox("default", "sandbox-a").await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("delete did not reach the driver"); + runtime.apply_deleted("sb-1").await.unwrap(); + + driver.release_delete(); + tokio::time::timeout(Duration::from_secs(1), delete) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn blocked_reconciliation_lookup_does_not_delay_watch_events() { + let driver = ControlledDriver::new(); + driver.block_get(); + let snapshot = ready_driver_sandbox("sb-a", "sandbox-a"); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new(snapshot.clone()))); + let runtime = test_runtime(driver.clone()).await; + for sandbox in [ + sandbox_record("sb-a", "sandbox-a", SandboxPhase::Provisioning), + sandbox_record("sb-b", "sandbox-b", SandboxPhase::Ready), + ] { + runtime.store.put_message(&sandbox).await.unwrap(); + } + + let (shutdown_tx, watch_handle) = start_watch_loop(&runtime, &driver).await; + let mut sandbox_b_rx = runtime.sandbox_watch_bus.subscribe("sb-b"); + let reconcile_runtime = runtime.clone(); + let reconcile = tokio::spawn(async move { + reconcile_runtime + .reconcile_snapshot_sandbox(snapshot, i64::MAX) + .await + }); + tokio::time::timeout(Duration::from_secs(1), driver.get_started.notified()) + .await + .expect("reconciliation did not reach the driver lookup"); + + driver.send_event(deleted_watch_event("sb-b")); + tokio::time::timeout(Duration::from_secs(1), sandbox_b_rx.recv()) + .await + .expect("watch event was blocked by reconciliation lookup") + .expect("sandbox B watch bus closed before notification"); + assert!( + runtime + .store + .get_message::("sb-b") + .await + .unwrap() + .is_none() + ); + + driver.release_get(); + tokio::time::timeout(Duration::from_secs(1), reconcile) .await + .unwrap() + .unwrap() .unwrap(); + stop_watch_loop(shutdown_tx, watch_handle).await; + } + + #[tokio::test] + async fn non_deleting_container_exit_still_transitions_to_error() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let mut exited = ready_driver_sandbox("sb-1", "sandbox-a"); + exited.status = Some(make_driver_status(make_driver_condition( + "ContainerExited", + "container exited unexpectedly", + ))); + + runtime.apply_sandbox_update(exited).await.unwrap(); let stored = runtime .store @@ -2942,7 +5033,7 @@ mod tests { .unwrap(); assert_eq!( SandboxPhase::try_from(stored.phase()).unwrap(), - SandboxPhase::Ready + SandboxPhase::Error ); } @@ -3663,6 +5754,7 @@ mod tests { .delete_sandbox("default", "uds-sandbox") .await .unwrap() + .deleted ); let calls = driver.calls(); @@ -3852,7 +5944,7 @@ mod tests { } #[tokio::test] - async fn apply_sandbox_update_uses_workspace_from_driver() { + async fn apply_sandbox_update_ignores_unknown_workspace_snapshot() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; runtime .apply_sandbox_update(DriverSandbox { @@ -3869,18 +5961,16 @@ mod tests { .await .unwrap(); - let stored = runtime - .store - .get_message::("sb-w1") - .await - .unwrap() - .unwrap(); - assert_eq!(stored.object_workspace(), "team-ml"); + let stored = runtime.store.get_message::("sb-w1").await.unwrap(); + assert!(stored.is_none()); } #[tokio::test] async fn apply_sandbox_update_preserves_workspace() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut sandbox = sandbox_record("sb-w2", "work", SandboxPhase::Provisioning); + sandbox.metadata.as_mut().unwrap().workspace = "alpha".to_string(); + runtime.store.put_message(&sandbox).await.unwrap(); runtime .apply_sandbox_update(DriverSandbox { id: "sb-w2".to_string(), @@ -3891,7 +5981,7 @@ mod tests { "DependenciesNotReady", "Provisioning", ))), - workspace: "alpha".to_string(), + workspace: "different-workspace".to_string(), }) .await .unwrap(); diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 12f563d17b..53124261e6 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -19,7 +19,7 @@ use crate::policy_store::{AtomicPolicyRevisionWrite, PolicyStoreExt}; use crate::provider_profile_sources::EffectiveProviderProfileCatalog; #[cfg(test)] use crate::provider_profile_sources::ProviderProfileSources; -use openshell_core::net::is_internal_ip; +use openshell_core::net::{is_always_blocked_ip, is_internal_ip}; use openshell_core::proto::policy_merge_operation; use openshell_core::proto::setting_value; use openshell_core::proto::{ @@ -906,15 +906,23 @@ async fn resolve_proposal_approval_mode( Ok((false, "default")) } +struct AutoApproveChunkContext<'a> { + sandbox: &'a Sandbox, + workspace: &'a str, + source: &'a str, + resolved_from: &'a str, + current_policy: &'a ProtoSandboxPolicy, + credential_set: &'a CredentialSet, +} + async fn auto_approve_chunk( state: &Arc, - sandbox_id: &str, - workspace: &str, - sandbox_name: &str, chunk_id: &str, - source: &str, - resolved_from: &str, + context: AutoApproveChunkContext<'_>, ) -> Result<(), Status> { + let sandbox_id = context.sandbox.object_id(); + let sandbox_name = context.sandbox.object_name(); + // Same gate the human-driven approve paths apply: if a global policy is // active, sandbox-scoped chunk approvals are meaningless because // `GetSandboxConfig` prefers the global policy. Auto-approving here @@ -937,8 +945,47 @@ async fn auto_approve_chunk( return Ok(()); } + // Mechanistic dedup may return an existing row whose proposed rule was + // edited after its original validation. Re-run the prover against that + // stored rule instead of trusting the incoming proposal's verdict. + let rule = decode_draft_chunk_rule(&chunk)? + .ok_or_else(|| Status::failed_precondition("draft chunk has no proposed rule"))?; + let validation_result = validation_result_for_agent_proposal( + context.current_policy.clone(), + &chunk.rule_name, + &rule, + context.credential_set, + ); + if validation_result != "prover: no new findings" { + info!( + sandbox_id = %sandbox_id, + chunk_id = %chunk_id, + rule_name = %chunk.rule_name, + validation_result = %validation_result, + source = %context.source, + resolved_from = %context.resolved_from, + "Auto-approval skipped: current stored rule has prover findings" + ); + return Ok(()); + } + + let security_notes = current_draft_chunk_security_notes(&chunk)?; + if !security_notes.is_empty() { + info!( + sandbox_id = %sandbox_id, + chunk_id = %chunk_id, + rule_name = %chunk.rule_name, + security_notes = %security_notes, + source = %context.source, + resolved_from = %context.resolved_from, + "Auto-approval skipped: current rule is security-flagged" + ); + return Ok(()); + } + let (version, hash) = - merge_chunk_into_policy(state.store.as_ref(), sandbox_id, workspace, &chunk).await?; + merge_chunk_into_policy(state.store.as_ref(), sandbox_id, context.workspace, &chunk) + .await?; let chunk_summary = summarize_draft_chunk_rule(&chunk)?; let now_ms = current_time_ms(); @@ -950,10 +997,10 @@ async fn auto_approve_chunk( state.sandbox_watch_bus.notify(sandbox_id); - let source_label = if source.is_empty() { + let source_label = if context.source.is_empty() { "unspecified" } else { - source + context.source }; emit_gateway_policy_auto_approve_audit_log( sandbox_id, @@ -964,7 +1011,7 @@ async fn auto_approve_chunk( version, &hash, source_label, - resolved_from, + context.resolved_from, ); info!( @@ -974,7 +1021,7 @@ async fn auto_approve_chunk( version = version, policy_hash = %hash, source = %source_label, - resolved_from = %resolved_from, + resolved_from = %context.resolved_from, "Auto-approved chunk: no new prover findings" ); @@ -2669,13 +2716,15 @@ pub(super) async fn handle_submit_policy_analysis( .map(Message::encode_to_vec) .unwrap_or_default(); - let rule_ref = chunk.proposed_rule.as_ref(); + let rule_ref = chunk.proposed_rule.as_ref().expect("checked above"); let (ep_host, ep_port) = rule_ref - .and_then(|r| r.endpoints.first()) + .endpoints + .first() .map(|ep| (ep.host.to_lowercase(), ep.port as i32)) .unwrap_or_default(); let ep_binary = rule_ref - .and_then(|r| r.binaries.first()) + .binaries + .first() .map(|b| b.path.clone()) .unwrap_or_default(); @@ -2686,7 +2735,7 @@ pub(super) async fn handle_submit_policy_analysis( let validation_result = validation_result_for_agent_proposal( current_policy.clone(), &chunk.rule_name, - chunk.proposed_rule.as_ref().expect("checked above"), + rule_ref, &credential_set, ); @@ -2701,10 +2750,7 @@ pub(super) async fn handle_submit_policy_analysis( rule_name: chunk.rule_name.clone(), proposed_rule: proposed_rule_bytes, rationale: chunk.rationale.clone(), - security_notes: generate_security_notes( - &ep_host, - u16::try_from(ep_port as u32).unwrap_or(0), - ), + security_notes: generate_security_notes(rule_ref), confidence: f64::from(chunk.confidence.clamp(0.0, 1.0)), created_at_ms: now_ms, decided_at_ms: None, @@ -2784,15 +2830,17 @@ pub(super) async fn handle_submit_policy_analysis( // string means findings or infrastructure error, both of which // require human attention. if auto_approve_enabled - && validation_result == "prover: no new findings" && let Err(err) = auto_approve_chunk( state, - &sandbox_id, - &workspace, - sandbox.object_name(), &effective_id, - &req.analysis_mode, - resolved_from, + AutoApproveChunkContext { + sandbox: &sandbox, + workspace: &workspace, + source: &req.analysis_mode, + resolved_from, + current_policy: ¤t_policy, + credential_set: &credential_set, + }, ) .await { @@ -3152,12 +3200,13 @@ async fn handle_approve_all_draft_chunks_inner( let mut last_hash = String::new(); for chunk in &pending_chunks { - if !req.include_security_flagged && !chunk.security_notes.is_empty() { + let security_notes = current_draft_chunk_security_notes(chunk)?; + if !req.include_security_flagged && !security_notes.is_empty() { info!( sandbox_id = %sandbox_id, chunk_id = %chunk.id, rule_name = %chunk.rule_name, - security_notes = %chunk.security_notes, + security_notes = %security_notes, "ApproveAllDraftChunks: skipping security-flagged chunk" ); chunks_skipped += 1; @@ -3577,17 +3626,27 @@ fn compute_config_revision( u64::from_le_bytes(bytes) } -fn draft_chunk_record_to_proto(record: &DraftChunkRecord) -> Result { - use openshell_core::proto::NetworkPolicyRule; - - let proposed_rule = if record.proposed_rule.is_empty() { - None +fn decode_draft_chunk_rule(record: &DraftChunkRecord) -> Result, Status> { + if record.proposed_rule.is_empty() { + Ok(None) } else { - Some( - NetworkPolicyRule::decode(record.proposed_rule.as_slice()) - .map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?, - ) - }; + NetworkPolicyRule::decode(record.proposed_rule.as_slice()) + .map(Some) + .map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}"))) + } +} + +fn current_draft_chunk_security_notes(record: &DraftChunkRecord) -> Result { + Ok(decode_draft_chunk_rule(record)? + .as_ref() + .map_or_else(String::new, generate_security_notes)) +} + +fn draft_chunk_record_to_proto(record: &DraftChunkRecord) -> Result { + let proposed_rule = decode_draft_chunk_rule(record)?; + let security_notes = proposed_rule + .as_ref() + .map_or_else(String::new, generate_security_notes); Ok(PolicyChunk { id: record.id.clone(), @@ -3595,7 +3654,7 @@ fn draft_chunk_record_to_proto(record: &DraftChunkRecord) -> Result San } } +fn allowed_ip_is_internal(entry: &str) -> bool { + use openshell_core::net::{is_always_blocked_net, is_internal_net}; + + let parsed = entry.parse::().or_else(|_| { + entry.parse::().map(|ip| match ip { + IpAddr::V4(v4) => ipnet::IpNet::V4(ipnet::Ipv4Net::from(v4)), + IpAddr::V6(v6) => ipnet::IpNet::V6(ipnet::Ipv6Net::from(v6)), + }) + }); + let Ok(net) = parsed else { + return false; + }; + + // These remain hard validation failures, not warning-only destinations. + if is_always_blocked_net(net) { + return false; + } + + is_internal_net(net) +} + /// Re-validate security notes server-side for a proposed policy chunk. -fn generate_security_notes(host: &str, port: u16) -> String { +fn generate_security_notes(rule: &NetworkPolicyRule) -> String { let mut notes = Vec::new(); - // Flag destinations that are an internal/private address. Parse the host as - // an IP literal and defer to the canonical RFC-accurate classifier - // (openshell-core net::is_internal_ip) rather than naive string prefixes: - // `starts_with("172.")` wrongly matched 172.0-15 / 172.32-255 (RFC 1918 is - // only 172.16.0.0/12) and missed CGNAT (100.64.0.0/10), IPv6 ULA, etc. The - // "localhost" hostname is not an IP literal, so it is checked separately. - // See #1777. - let resolves_internal = host.parse::().is_ok_and(is_internal_ip); - if resolves_internal || host == "localhost" { - notes.push(format!( - "Destination '{host}' appears to be an internal/private address." - )); - } + for endpoint in &rule.endpoints { + let host = endpoint.host.to_lowercase(); + + // Flag destinations that are an internal/private address. Parse the host as + // an IP literal and defer to the canonical RFC-accurate classifier + // (openshell-core net::is_internal_ip) rather than naive string prefixes: + // `starts_with("172.")` wrongly matched 172.0-15 / 172.32-255 (RFC 1918 is + // only 172.16.0.0/12) and missed CGNAT (100.64.0.0/10), IPv6 ULA, etc. The + // Always-blocked destinations are rejected by merge validation instead + // of being presented as advisory findings. See #1777. + let resolves_advisory_internal = host + .parse::() + .is_ok_and(|ip| is_internal_ip(ip) && !is_always_blocked_ip(ip)); + if resolves_advisory_internal { + notes.push(format!( + "Destination '{host}' appears to be an internal/private address." + )); + } - if host.contains('*') { - notes.push(format!( - "Host '{host}' contains a wildcard — this may match unintended destinations." - )); - } + if host.contains('*') { + notes.push(format!( + "Host '{host}' contains a wildcard — this may match unintended destinations." + )); + } - if port > 49152 { - notes.push(format!( - "Port {port} is in the ephemeral range — this may be a temporary service." - )); - } + for allowed_ip in &endpoint.allowed_ips { + if allowed_ip_is_internal(allowed_ip) { + notes.push(format!( + "allowed_ips includes private/internal range '{allowed_ip}'." + )); + } + } + if host.trim().is_empty() && !endpoint.allowed_ips.is_empty() { + notes.push( + "allowed_ips allowlist is hostless and may match any hostname resolving within the configured range." + .to_string(), + ); + } - const DB_PORTS: [u16; 7] = [5432, 3306, 6379, 27017, 9200, 11211, 5672]; - if DB_PORTS.contains(&port) { - notes.push(format!( - "Port {port} is a well-known database/service port." - )); + let ports = if endpoint.ports.is_empty() { + std::slice::from_ref(&endpoint.port) + } else { + endpoint.ports.as_slice() + }; + for raw_port in ports { + let port = u16::try_from(*raw_port).unwrap_or(0); + if port > 49152 { + notes.push(format!( + "Port {port} is in the ephemeral range — this may be a temporary service." + )); + } + + const DB_PORTS: [u16; 7] = [5432, 3306, 6379, 27017, 9200, 11211, 5672]; + if DB_PORTS.contains(&port) { + notes.push(format!( + "Port {port} is a well-known database/service port." + )); + } + } } notes.join(" ") @@ -4444,25 +4552,159 @@ mod tests { request } + fn security_notes_for_host(host: &str) -> String { + generate_security_notes(&NetworkPolicyRule { + endpoints: vec![NetworkEndpoint { + host: host.to_string(), + port: 80, + ..Default::default() + }], + ..Default::default() + }) + } + #[test] fn security_notes_use_canonical_internal_ip_classifier() { // RFC 1918 is 172.16.0.0/12 only: the old starts_with("172.") prefix // wrongly flagged 172.15/172.32 and missed CGNAT (100.64.0.0/10). #1777. - assert!(generate_security_notes("172.16.0.1", 80).contains("internal/private")); - assert!(!generate_security_notes("172.15.0.1", 80).contains("internal/private")); - assert!(!generate_security_notes("172.32.0.1", 80).contains("internal/private")); - assert!(generate_security_notes("100.64.0.1", 80).contains("internal/private")); - assert!(generate_security_notes("10.0.0.1", 80).contains("internal/private")); - assert!(generate_security_notes("192.168.1.1", 80).contains("internal/private")); - assert!(generate_security_notes("127.0.0.1", 80).contains("internal/private")); - assert!(generate_security_notes("localhost", 80).contains("internal/private")); - assert!(!generate_security_notes("8.8.8.8", 80).contains("internal/private")); + assert!(security_notes_for_host("172.16.0.1").contains("internal/private")); + assert!(!security_notes_for_host("172.15.0.1").contains("internal/private")); + assert!(!security_notes_for_host("172.32.0.1").contains("internal/private")); + assert!(security_notes_for_host("100.64.0.1").contains("internal/private")); + assert!(security_notes_for_host("10.0.0.1").contains("internal/private")); + assert!(security_notes_for_host("192.168.1.1").contains("internal/private")); + assert!(!security_notes_for_host("8.8.8.8").contains("internal/private")); // Hostnames that merely start with a private-range prefix must NOT be // flagged: classification parses an IP literal, not a string prefix. #1824. - assert!(!generate_security_notes("10.example.com", 80).contains("internal/private")); - assert!(!generate_security_notes("172.example.com", 80).contains("internal/private")); + assert!(!security_notes_for_host("10.example.com").contains("internal/private")); + assert!(!security_notes_for_host("172.example.com").contains("internal/private")); // IPv6 ULA (fc00::/7, RFC 4193) is internal/private. - assert!(generate_security_notes("fd00::1", 80).contains("internal/private")); + assert!(security_notes_for_host("fd00::1").contains("internal/private")); + } + + #[test] + fn security_notes_exclude_always_blocked_destinations() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![ + NetworkEndpoint { + host: "127.0.0.1".to_string(), + port: 443, + ..Default::default() + }, + NetworkEndpoint { + host: "169.254.169.254".to_string(), + port: 443, + ..Default::default() + }, + NetworkEndpoint { + host: "localhost".to_string(), + port: 443, + ..Default::default() + }, + NetworkEndpoint { + host: "metadata.google.internal".to_string(), + port: 443, + allowed_ips: vec![ + "127.0.0.0/8".to_string(), + "169.254.0.0/16".to_string(), + "::/128".to_string(), + ], + ..Default::default() + }, + ], + ..Default::default() + }); + + assert!(notes.is_empty(), "{notes}"); + } + + #[test] + fn security_notes_flag_private_allowed_ips() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["10.0.0.0/8".to_string()], + ..Default::default() + }], + ..Default::default() + }); + + assert!(notes.contains("allowed_ips includes private/internal range '10.0.0.0/8'.")); + } + + #[test] + fn security_notes_flag_cidr_overlapping_cgnat() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["100.0.0.0/9".to_string()], + ..Default::default() + }], + ..Default::default() + }); + + assert!(notes.contains("allowed_ips includes private/internal range '100.0.0.0/9'.")); + } + + #[test] + fn security_notes_do_not_flag_large_public_allowed_ips() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["8.8.0.0/16".to_string(), "8.8.8.0/24".to_string()], + ..Default::default() + }], + ..Default::default() + }); + + assert!(notes.is_empty(), "{notes}"); + } + + #[test] + fn security_notes_flag_hostless_allowed_ips() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![NetworkEndpoint { + allowed_ips: vec!["192.168.0.0/16".to_string()], + port: 443, + ..Default::default() + }], + ..Default::default() + }); + + assert!(notes.contains("allowed_ips includes private/internal range '192.168.0.0/16'.")); + assert!(notes.contains( + "allowed_ips allowlist is hostless and may match any hostname resolving within the configured range." + )); + } + + #[test] + fn security_notes_inspect_all_endpoints_and_effective_ports() { + let notes = generate_security_notes(&NetworkPolicyRule { + endpoints: vec![ + NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + ..Default::default() + }, + NetworkEndpoint { + host: "*.internal.example".to_string(), + port: 3306, + ports: vec![5432, 50_000], + allowed_ips: vec!["172.16.0.0/12".to_string()], + ..Default::default() + }, + ], + ..Default::default() + }); + + assert!(notes.contains("Host '*.internal.example' contains a wildcard")); + assert!(notes.contains("allowed_ips includes private/internal range '172.16.0.0/12'.")); + assert!(notes.contains("Port 5432 is a well-known database/service port.")); + assert!(notes.contains("Port 50000 is in the ephemeral range")); + assert!(!notes.contains("Port 3306")); } #[test] @@ -6672,6 +6914,497 @@ mod tests { .unwrap(); } + #[tokio::test] + async fn approve_all_skips_private_allowed_ips_unless_included() { + let state = test_server_state().await; + let sandbox_name = "private-allowed-ips"; + state + .store + .put_message(&test_sandbox( + "sb-private-allowed-ips", + sandbox_name, + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + + let submit = handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "agent_authored".to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: "private_service".to_string(), + proposed_rule: Some(NetworkPolicyRule { + name: "private_service".to_string(), + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["10.0.0.0/8".to_string()], + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }), + ..Default::default() + }], + ..Default::default() + })), + ) + .await + .unwrap() + .into_inner(); + let chunk_id = submit.accepted_chunk_ids[0].clone(); + let chunk = state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap(); + assert!( + chunk + .security_notes + .contains("allowed_ips includes private/internal range '10.0.0.0/8'.") + ); + + let skipped = handle_approve_all_draft_chunks( + &state, + with_user(Request::new(ApproveAllDraftChunksRequest { + name: sandbox_name.to_string(), + include_security_flagged: false, + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(skipped.chunks_approved, 0); + assert_eq!(skipped.chunks_skipped, 1); + assert_eq!( + state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap() + .status, + "pending" + ); + + let approved = handle_approve_all_draft_chunks( + &state, + with_user(Request::new(ApproveAllDraftChunksRequest { + name: sandbox_name.to_string(), + include_security_flagged: true, + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(approved.chunks_approved, 1); + assert_eq!(approved.chunks_skipped, 0); + assert_eq!( + state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap() + .status, + "approved" + ); + } + + #[tokio::test] + async fn edit_recomputes_security_notes_before_read_and_approve_all() { + let state = test_server_state().await; + let sandbox_id = "sb-edit-security-notes"; + let sandbox_name = "edit-security-notes"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + + let safe_rule = NetworkPolicyRule { + name: "edited_service".to_string(), + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }; + let submit = handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "agent_authored".to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: "edited_service".to_string(), + proposed_rule: Some(safe_rule.clone()), + ..Default::default() + }], + ..Default::default() + })), + ) + .await + .unwrap() + .into_inner(); + let chunk_id = submit.accepted_chunk_ids[0].clone(); + + let mut private_rule = safe_rule; + private_rule.endpoints[0].allowed_ips = vec!["10.0.0.0/8".to_string()]; + handle_edit_draft_chunk( + &state, + with_user(Request::new(EditDraftChunkRequest { + name: sandbox_name.to_string(), + chunk_id: chunk_id.clone(), + proposed_rule: Some(private_rule), + workspace: "default".to_string(), + })), + ) + .await + .unwrap(); + + let stored = state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap(); + assert!(stored.security_notes.is_empty()); + + let draft = handle_get_draft_policy( + &state, + with_user(Request::new(GetDraftPolicyRequest { + name: sandbox_name.to_string(), + status_filter: String::new(), + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert!( + draft.chunks[0] + .security_notes + .contains("allowed_ips includes private/internal range '10.0.0.0/8'.") + ); + + let skipped = handle_approve_all_draft_chunks( + &state, + with_user(Request::new(ApproveAllDraftChunksRequest { + name: sandbox_name.to_string(), + include_security_flagged: false, + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(skipped.chunks_approved, 0); + assert_eq!(skipped.chunks_skipped, 1); + assert_eq!( + state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap() + .status, + "pending" + ); + } + + #[tokio::test] + async fn approve_all_recomputes_empty_legacy_security_notes() { + let state = test_server_state().await; + let sandbox_id = "sb-stale-security-notes"; + let sandbox_name = "stale-security-notes"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + + let rule = NetworkPolicyRule { + name: "legacy_private".to_string(), + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["10.0.0.0/8".to_string()], + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }; + let mut chunk = pending_draft_chunk("legacy-private", sandbox_id); + chunk.rule_name = rule.name.clone(); + chunk.proposed_rule = rule.encode_to_vec(); + chunk.host = "service.example.com".to_string(); + chunk.port = 443; + state + .store + .put_draft_chunk(&chunk, None, "default") + .await + .unwrap(); + + let skipped = handle_approve_all_draft_chunks( + &state, + with_user(Request::new(ApproveAllDraftChunksRequest { + name: sandbox_name.to_string(), + include_security_flagged: false, + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(skipped.chunks_approved, 0); + assert_eq!(skipped.chunks_skipped, 1); + assert_eq!( + state + .store + .get_draft_chunk(&chunk.id) + .await + .unwrap() + .unwrap() + .status, + "pending" + ); + } + + #[tokio::test] + async fn security_flagged_empty_delta_does_not_auto_approve() { + let state = test_server_state().await; + let sandbox_name = "security-flagged-auto"; + state + .store + .put_message(&test_sandbox( + "sb-security-flagged-auto", + sandbox_name, + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + seed_sandbox_approval_mode(&state, sandbox_name, "auto").await; + + handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "mechanistic".to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: "private_service".to_string(), + proposed_rule: Some(NetworkPolicyRule { + name: "private_service".to_string(), + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + allowed_ips: vec!["10.0.0.0/8".to_string()], + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }), + ..Default::default() + }], + ..Default::default() + })), + ) + .await + .unwrap(); + + let draft = handle_get_draft_policy( + &state, + with_user(Request::new(GetDraftPolicyRequest { + name: sandbox_name.to_string(), + status_filter: String::new(), + workspace: "default".to_string(), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(draft.chunks[0].validation_result, "prover: no new findings"); + assert_eq!(draft.chunks[0].status, "pending"); + assert!( + draft.chunks[0] + .security_notes + .contains("allowed_ips includes private/internal range '10.0.0.0/8'.") + ); + } + + #[tokio::test] + async fn mechanistic_dedup_auto_approval_rechecks_edited_stored_rule() { + let state = test_server_state().await; + let sandbox_id = "sb-stored-prover-verdict"; + let sandbox_name = "stored-prover-verdict"; + state + .store + .put_message(&test_provider("github-pat", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + ProtoSandboxPolicy::default(), + vec!["github-pat".to_string()], + )) + .await + .unwrap(); + + // The first observation is clean and remains pending in the default + // manual mode. Its normalized endpoint columns become the dedup key. + let safe_rule = NetworkPolicyRule { + name: "safe_service".to_string(), + endpoints: vec![NetworkEndpoint { + host: "service.example.com".to_string(), + port: 443, + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }; + let first = handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "mechanistic".to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: safe_rule.name.clone(), + proposed_rule: Some(safe_rule.clone()), + ..Default::default() + }], + ..Default::default() + })), + ) + .await + .unwrap() + .into_inner(); + let chunk_id = first.accepted_chunk_ids[0].clone(); + let original = state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap(); + assert_eq!(original.status, "pending"); + assert_eq!(original.validation_result, "prover: no new findings"); + + // Editing only replaces the stored protobuf payload; the normalized + // dedup columns remain service.example.com:443 + /usr/bin/curl. This + // replacement has no advisory notes but does add credentialed reach. + let finding_rule = NetworkPolicyRule { + name: safe_rule.name.clone(), + endpoints: vec![NetworkEndpoint { + host: "api.github.com".to_string(), + port: 443, + ..Default::default() + }], + binaries: safe_rule.binaries.clone(), + }; + assert!(generate_security_notes(&finding_rule).is_empty()); + let credential_set = CredentialSet { + credentials: vec![Credential { + name: "github-pat".to_string(), + cred_type: "github".to_string(), + scopes: Vec::new(), + injected_via: String::new(), + target_hosts: vec!["api.github.com".to_string()], + }], + api_registries: HashMap::new(), + }; + assert!( + validation_result_for_agent_proposal( + ProtoSandboxPolicy::default(), + &finding_rule.name, + &finding_rule, + &credential_set, + ) + .contains("credential_reach_expansion") + ); + handle_edit_draft_chunk( + &state, + with_user(Request::new(EditDraftChunkRequest { + name: sandbox_name.to_string(), + chunk_id: chunk_id.clone(), + proposed_rule: Some(finding_rule), + workspace: "default".to_string(), + })), + ) + .await + .unwrap(); + + seed_sandbox_approval_mode(&state, sandbox_name, "auto").await; + + // The duplicate incoming proposal is clean. Dedup returns the edited + // row, so auto-approval must evaluate that stored payload instead. + let duplicate = handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "mechanistic".to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: safe_rule.name.clone(), + proposed_rule: Some(safe_rule), + ..Default::default() + }], + ..Default::default() + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(duplicate.accepted_chunk_ids, vec![chunk_id.clone()]); + + let stored = state + .store + .get_draft_chunk(&chunk_id) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.status, "pending"); + let stored_rule = NetworkPolicyRule::decode(stored.proposed_rule.as_slice()).unwrap(); + assert_eq!(stored_rule.endpoints[0].host, "api.github.com"); + assert!(generate_security_notes(&stored_rule).is_empty()); + assert!( + state + .store + .get_latest_policy(sandbox_id) + .await + .unwrap() + .is_none() + ); + } + #[tokio::test] async fn draft_chunk_handler_lifecycle_round_trip() { use openshell_core::proto::{ @@ -8511,6 +9244,7 @@ mod tests { host: "api.github.com".to_string(), port: 443, protocol: "rest".to_string(), + access: "full".to_string(), deny_rules: vec![L7DenyRule { method: "DELETE".to_string(), path: "/repos/*".to_string(), @@ -9106,9 +9840,9 @@ mod tests { seed_sandbox_approval_mode(&state, &sandbox_name, "auto").await; let proposed_rule = NetworkPolicyRule { - name: "allow_10_0_0_5_8080".to_string(), + name: "allow_example_8080".to_string(), endpoints: vec![NetworkEndpoint { - host: "10.0.0.5".to_string(), + host: "example.com".to_string(), port: 8080, ..Default::default() }], @@ -9128,7 +9862,7 @@ mod tests { name: sandbox_name, analysis_mode: "mechanistic".to_string(), proposed_chunks: vec![PolicyChunk { - rule_name: "allow_10_0_0_5_8080".to_string(), + rule_name: "allow_example_8080".to_string(), proposed_rule: Some(rule), ..Default::default() }], @@ -9207,7 +9941,7 @@ mod tests { .expect("auto-approve must have persisted a policy revision"); let policy = SandboxPolicy::decode(latest.policy_payload.as_slice()).unwrap(); assert!( - policy.network_policies.contains_key("allow_10_0_0_5_8080"), + policy.network_policies.contains_key("allow_example_8080"), "approved rule must stay merged after resubmit; keys: {:?}", policy.network_policies.keys().collect::>() ); diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index e8f67d0fa8..46d3a31cd4 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -3076,6 +3076,7 @@ mod tests { port, path: path.to_string(), protocol: "rest".to_string(), + access: "full".to_string(), ..Default::default() }]; handle_import_provider_profiles( @@ -3214,6 +3215,7 @@ mod tests { port: 443, path: "/v1/**".to_string(), protocol: "rest".to_string(), + access: "full".to_string(), ..Default::default() }]; let response = handle_import_provider_profiles( @@ -3549,6 +3551,7 @@ mod tests { port: 443, path: "/v1/**".to_string(), protocol: "rest".to_string(), + access: "full".to_string(), ..Default::default() }]; let response = handle_update_provider_profiles( diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 19238b1ea3..57bf421b88 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -574,19 +574,14 @@ async fn handle_delete_sandbox_inner( .await? .name; - let sandbox_id = state - .store - .get_message_by_name::(&workspace, &name) - .await - .ok() - .flatten() - .map(|sandbox| sandbox.object_id().to_string()); - let deleted = state.compute.delete_sandbox(&workspace, &name).await?; - if deleted && let Some(sandbox_id) = sandbox_id { - state.telemetry.end_sandbox_session(&sandbox_id); + let result = state.compute.delete_sandbox(&workspace, &name).await?; + if result.deleted { + state.telemetry.end_sandbox_session(&result.sandbox_id); } info!(sandbox_name = %name, "DeleteSandbox request completed successfully"); - Ok(Response::new(DeleteSandboxResponse { deleted })) + Ok(Response::new(DeleteSandboxResponse { + deleted: result.deleted, + })) } async fn sandbox_by_name( @@ -2461,6 +2456,61 @@ mod tests { sandbox } + #[tokio::test] + async fn delete_handler_ends_telemetry_for_the_resolved_sandbox_id() { + let state = test_server_state().await; + let mut original = test_sandbox("reused-name", Vec::new()); + original.metadata.as_mut().unwrap().id = "sandbox-original".to_string(); + state.store.put_message(&original).await.unwrap(); + + // Hold the global guard so the handler can resolve the original ID and + // acquire its delete gate, but cannot yet revalidate or mutate it. + let global_guard = state.compute.sandbox_sync_guard().await; + let delete_state = state.clone(); + let delete = tokio::spawn(async move { + handle_delete_sandbox_inner( + &delete_state, + Request::new(DeleteSandboxRequest { + name: "reused-name".to_string(), + workspace: "default".to_string(), + }), + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(1), async { + while state.compute.delete_gate_entry_count() == 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("delete did not resolve and acquire the original sandbox gate"); + + state + .store + .delete(Sandbox::object_type(), original.object_id()) + .await + .unwrap(); + let mut replacement = test_sandbox("reused-name", Vec::new()); + replacement.metadata.as_mut().unwrap().id = "sandbox-replacement".to_string(); + state.store.put_message(&replacement).await.unwrap(); + drop(global_guard); + + let response = delete.await.unwrap().unwrap().into_inner(); + assert!(response.deleted); + assert!( + state + .store + .get_message::(replacement.object_id()) + .await + .unwrap() + .is_some() + ); + assert_eq!( + state.telemetry.ended_sandbox_sessions(), + [original.object_id().to_string()] + ); + } + #[tokio::test] async fn attach_sandbox_provider_persists_current_provider_list() { let state = test_server_state().await; diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 83e27d27cb..a58f66c916 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -1724,16 +1724,21 @@ mod tests { .with_span_events(FmtSpan::CLOSE); let subscriber = tracing_subscriber::registry().with(fmt_layer); - let _guard = tracing::subscriber::set_default(subscriber); - - let req = Request::builder() - .uri("/test-path") - .header("x-request-id", "trace-test-id-12345") - .body(Empty::::new()) - .unwrap(); - let span = make_request_span(&req); - drop(span.enter()); - drop(span); + tracing::subscriber::with_default(subscriber, || { + // Other parallel tests may register this callsite while no subscriber + // is active. Refresh the process-wide cache after installing this + // thread-local subscriber so the span cannot remain disabled. + tracing::callsite::rebuild_interest_cache(); + + let req = Request::builder() + .uri("/test-path") + .header("x-request-id", "trace-test-id-12345") + .body(Empty::::new()) + .unwrap(); + let span = make_request_span(&req); + drop(span.enter()); + drop(span); + }); let output = String::from_utf8(log_buf.lock().unwrap().clone()).unwrap(); assert!( diff --git a/crates/openshell-server/src/telemetry.rs b/crates/openshell-server/src/telemetry.rs index 7de21154e7..c0778b1ff8 100644 --- a/crates/openshell-server/src/telemetry.rs +++ b/crates/openshell-server/src/telemetry.rs @@ -9,19 +9,38 @@ use openshell_core::telemetry::DenyGroup; use std::collections::HashMap; #[derive(Debug, Default)] -pub struct TelemetryState; +pub struct TelemetryState { + #[cfg(test)] + ended_sandbox_sessions: std::sync::Mutex>, +} #[allow(clippy::unused_self)] impl TelemetryState { pub fn new() -> Self { - Self + Self::default() } pub fn sandbox_session_connected(&self, _sandbox_id: &str) {} pub fn sandbox_session_disconnected(&self, _sandbox_id: &str) {} - pub fn end_sandbox_session(&self, _sandbox_id: &str) {} + pub fn end_sandbox_session(&self, sandbox_id: &str) { + #[cfg(test)] + self.ended_sandbox_sessions + .lock() + .expect("telemetry session lock poisoned") + .push(sandbox_id.to_string()); + #[cfg(not(test))] + let _ = sandbox_id; + } + + #[cfg(test)] + pub(crate) fn ended_sandbox_sessions(&self) -> Vec { + self.ended_sandbox_sessions + .lock() + .expect("telemetry session lock poisoned") + .clone() + } pub fn record_network_activity(&self, sandbox_id: &str, summary: &NetworkActivitySummary) { if sandbox_id.is_empty() || !openshell_core::telemetry::enabled() { diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 1449276f4f..008bf1104b 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -50,7 +50,6 @@ uuid = { workspace = true } webpki-roots = { workspace = true } [dev-dependencies] -openshell-core = { path = "../openshell-core", features = ["test-helpers"] } openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } tempfile = "3" tonic = { workspace = true } diff --git a/crates/openshell-supervisor-network/src/l7/graphql.rs b/crates/openshell-supervisor-network/src/l7/graphql.rs index 12979f0b19..2569bc68fa 100644 --- a/crates/openshell-supervisor-network/src/l7/graphql.rs +++ b/crates/openshell-supervisor-network/src/l7/graphql.rs @@ -801,9 +801,7 @@ network_policies: ancestors: Vec::new(), cmdline_paths: Vec::new(), secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let request_info = crate::l7::L7RequestInfo { action: req.action, diff --git a/crates/openshell-supervisor-network/src/l7/middleware.rs b/crates/openshell-supervisor-network/src/l7/middleware.rs index 263d886f28..28f91c3bb7 100644 --- a/crates/openshell-supervisor-network/src/l7/middleware.rs +++ b/crates/openshell-supervisor-network/src/l7/middleware.rs @@ -215,11 +215,7 @@ pub async fn send_middleware_rejection_response, redacted_target: &str, ) -> Result<()> { - let context = Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }); + let context = Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)); if let Some(denial) = denial { crate::l7::rest::send_middleware_deny_response( req, @@ -562,9 +558,7 @@ mod tests { ancestors: Vec::new(), cmdline_paths: Vec::new(), secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let req = crate::l7::provider::L7Request { action: "POST".into(), @@ -602,8 +596,6 @@ mod tests { #[tokio::test] async fn middleware_failure_uses_platform_response_without_policy_guidance() { - let _proposals = - openshell_core::proposals::test_helpers::ProposalsFlagGuard::set(true).await; let ctx = L7EvalContext { host: "api.example.test".into(), port: 443, @@ -612,9 +604,7 @@ mod tests { ancestors: Vec::new(), cmdline_paths: Vec::new(), secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let req = crate::l7::provider::L7Request { action: "POST".into(), diff --git a/crates/openshell-supervisor-network/src/l7/mod.rs b/crates/openshell-supervisor-network/src/l7/mod.rs index f054695a4f..64f4ae3741 100644 --- a/crates/openshell-supervisor-network/src/l7/mod.rs +++ b/crates/openshell-supervisor-network/src/l7/mod.rs @@ -21,34 +21,8 @@ pub mod tls; pub(crate) mod token_grant_injection; pub(crate) mod websocket; -/// Application-layer protocol for L7 inspection. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum L7Protocol { - Rest, - Websocket, - Graphql, - Sql, - JsonRpc, - Mcp, -} - -impl L7Protocol { - pub fn parse(s: &str) -> Option { - match s.to_ascii_lowercase().as_str() { - "rest" => Some(Self::Rest), - "websocket" => Some(Self::Websocket), - "graphql" => Some(Self::Graphql), - "sql" => Some(Self::Sql), - "json-rpc" => Some(Self::JsonRpc), - "mcp" => Some(Self::Mcp), - _ => None, - } - } - - pub fn is_jsonrpc_family(self) -> bool { - matches!(self, Self::JsonRpc | Self::Mcp) - } -} +pub use openshell_policy::L7Protocol; +use openshell_policy::{L7EndpointFields, validate_l7_endpoint_semantics}; /// TLS handling mode for proxy connections. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] @@ -1031,40 +1005,51 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< )); } - // rules + access mutual exclusion - if has_rules && !access.is_empty() { - errors.push(format!("{loc}: rules and access are mutually exclusive")); - } - - if jsonrpc_family && !access.is_empty() { - if protocol == "mcp" { - errors.push(format!( - "{loc}: protocol {protocol} does not support access presets; use rules/deny_rules or set mcp.allow_all_known_mcp_methods: true for an allow-all MCP policy" - )); - } else { - errors.push(format!( - "{loc}: protocol {protocol} does not support access presets; use explicit rules with allow.method such as \"*\"" - )); - } - } - - if protocol == "json-rpc" && !has_rules { - errors.push(format!( - "{loc}: protocol {protocol} requires explicit rules with allow.method" - )); - } - - // protocol requires rules or access - if !protocol.is_empty() && protocol != "mcp" && !has_rules && access.is_empty() { - errors.push(format!( - "{loc}: protocol requires rules or access to define allowed traffic" - )); - } - - if !protocol.is_empty() && l7_protocol.is_none() { - errors.push(format!( - "{loc}: unknown protocol '{protocol}' (expected rest, websocket, graphql, sql, json-rpc, or mcp)" - )); + // L7 endpoint semantic validation (shared with profile lint). + // Computed lazily: has_deny_rules and rules_would_deny_all are + // needed here but also referenced by per-rule checks below. + let has_deny_rules = ep + .get("deny_rules") + .and_then(|v| v.as_array()) + .is_some_and(|a| !a.is_empty()); + let rules_would_deny_all = + ep.get("rules") + .and_then(|v| v.as_array()) + .is_some_and(|rules| { + !rules.is_empty() + && rules.iter().all(|rule| { + let allow = rule.get("allow"); + allow.is_none_or(serde_json::Value::is_null) + || allow.is_some_and(|v| { + let str_empty = |key| { + v.get(key) + .and_then(|s| s.as_str()) + .unwrap_or("") + .is_empty() + }; + str_empty("method") + && str_empty("path") + && str_empty("command") + && str_empty("operation_type") + && str_empty("operation_name") + && !mcp_rule_has_tool_selector(v) + }) + }) + }); + let mcp_allow_all_known_mcp_methods = ep + .get("mcp_allow_all_known_mcp_methods") + .and_then(serde_json::Value::as_bool) + .unwrap_or(false); + let l7_fields = L7EndpointFields { + protocol, + access, + has_rules, + has_deny_rules, + rules_would_deny_all, + allow_all_known_mcp_methods: mcp_allow_all_known_mcp_methods, + }; + for msg in validate_l7_endpoint_semantics(&l7_fields) { + errors.push(format!("{loc}: {msg}")); } if let Some(mode) = ep.get("persisted_queries").and_then(|v| v.as_str()) @@ -1154,20 +1139,6 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< .get("mcp_strict_tool_names") .and_then(serde_json::Value::as_bool) .unwrap_or(true); - let mcp_allow_all_known_mcp_methods = ep - .get("mcp_allow_all_known_mcp_methods") - .and_then(serde_json::Value::as_bool) - .unwrap_or(false); - if protocol == "mcp" - && !has_rules - && access.is_empty() - && !mcp_allow_all_known_mcp_methods - { - errors.push(format!( - "{loc}: protocol mcp requires rules when mcp.allow_all_known_mcp_methods is false" - )); - } - if ep .get("websocket_credential_rewrite") .and_then(serde_json::Value::as_bool) @@ -1225,41 +1196,13 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< )); } - // rules with empty list - if ep - .get("rules") - .and_then(|v| v.as_array()) - .is_some_and(Vec::is_empty) - { - errors.push(format!( - "{loc}: rules list cannot be empty (would deny all traffic). Use `access: full` or remove rules." - )); - } - // port 443 + rest + tls: skip — L7 won't work (already handled above) // The old warning about missing `tls: terminate` is no longer needed // because TLS termination is now automatic. - // Validate deny_rules - let has_deny_rules = ep - .get("deny_rules") - .and_then(|v| v.as_array()) - .is_some_and(|a| !a.is_empty()); + // Per-rule deny_rules validation (semantic checks handled by + // shared validator above). if has_deny_rules { - // deny_rules require L7 inspection - if protocol.is_empty() { - errors.push(format!( - "{loc}: deny_rules require protocol (L7 inspection must be enabled)" - )); - } - - // deny_rules require some allow base (access or rules) - if protocol != "mcp" && !has_rules && access.is_empty() { - errors.push(format!( - "{loc}: deny_rules require rules or access to define the base allow set" - )); - } - let has_mcp_tool_allow_selectors = protocol == "mcp" && mcp_endpoint_has_tool_allow_selectors(ep); @@ -1356,6 +1299,17 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< } } + // Empty rules list (explicitly set but empty) + if ep + .get("rules") + .and_then(|v| v.as_array()) + .is_some_and(Vec::is_empty) + { + errors.push(format!( + "{loc}: rules list cannot be empty (would deny all traffic). Use `access: full` or remove rules." + )); + } + // Empty deny_rules list (explicitly set but empty) if ep .get("deny_rules") @@ -1409,16 +1363,20 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< for (rule_idx, rule) in rules.iter().enumerate() { let allow = rule.get("allow").unwrap_or(rule); let rule_loc = format!("{loc}.rules[{rule_idx}].allow"); - if has_mcp_tool_allow_selectors - && allow + if has_mcp_tool_allow_selectors && !mcp_rule_has_tool_selector(allow) { + let method = allow .get("method") .and_then(serde_json::Value::as_str) - .is_some_and(method_matcher_matches_tools_call) - && !mcp_rule_has_tool_selector(allow) - { - errors.push(format!( - "{rule_loc}: method matcher allows every tool call and conflicts with MCP tool allow rules; add tool or params.name to narrow tools/call, or remove the tool allow rules" - )); + .unwrap_or(""); + if method_matcher_matches_tools_call(method) + || (method.is_empty() && mcp_allow_all_known_mcp_methods) + { + errors.push(format!( + "{rule_loc}: method matcher allows every tool call and conflicts \ + with MCP tool allow rules; add tool or params.name to narrow \ + tools/call, or remove the tool allow rules" + )); + } } validate_jsonrpc_rule_fields( &mut errors, @@ -2392,6 +2350,46 @@ mod tests { ); } + #[test] + fn validate_mcp_empty_allow_with_allow_all_conflicts_with_tool_rules() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "mcp.example.com", + "port": 443, + "protocol": "mcp", + "mcp_allow_all_known_mcp_methods": true, + "rules": [{ + "allow": { + "params": { "name": { "glob": "read_*" } } + } + }, { + "allow": { + "method": "", + "path": "", + "command": "", + "operation_type": "", + "operation_name": "" + } + }] + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + errors.iter().any(|e| { + e.contains("rules[1].allow") + && e.contains("allows every tool call") + && e.contains("conflicts with MCP tool allow rules") + }), + "empty allow with allow_all_known_mcp_methods normalizes to method:* and should \ + conflict with tool-specific rules: {errors:?}" + ); + } + #[test] fn validate_jsonrpc_rejects_params_matchers() { let data = serde_json::json!({ @@ -3503,6 +3501,140 @@ mod tests { ); } + #[test] + fn validate_rules_empty_list_rejected() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "api.example.com", + "port": 443, + "protocol": "rest", + "access": "full", + "rules": [] + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + errors + .iter() + .any(|e| e.contains("rules list cannot be empty")), + "should reject empty rules: {errors:?}" + ); + } + + #[test] + fn validate_rules_deny_all_with_empty_allow_object() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "api.example.com", + "port": 443, + "protocol": "rest", + "rules": [{"allow": {"method": "", "path": ""}}] + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + errors.iter().any(|e| e.contains("would deny all traffic")), + "should detect deny-all with empty allow object: {errors:?}" + ); + } + + #[test] + fn validate_mcp_params_selector_not_classified_as_deny_all() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "mcp.example.com", + "port": 443, + "protocol": "mcp", + "mcp_allow_all_known_mcp_methods": true, + "rules": [{ + "allow": { + "method": "", + "path": "", + "command": "", + "operation_type": "", + "operation_name": "", + "params": { "name": { "glob": "read_*" } } + } + }], + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + errors.is_empty(), + "MCP rule with params.name selector should not be classified as deny-all: {errors:?}" + ); + } + + #[test] + fn validate_mcp_tool_selector_not_classified_as_deny_all() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "mcp.example.com", + "port": 443, + "protocol": "mcp", + "mcp_allow_all_known_mcp_methods": true, + "rules": [{ + "allow": { + "method": "", + "path": "", + "command": "", + "operation_type": "", + "operation_name": "", + "tool": "my-tool" + } + }], + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + errors.is_empty(), + "MCP rule with tool selector should not be classified as deny-all: {errors:?}" + ); + } + + #[test] + fn validate_mcp_allow_all_with_empty_allow_not_denied() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "mcp.example.com", + "port": 443, + "protocol": "mcp", + "mcp_allow_all_known_mcp_methods": true, + "rules": [{"allow": {}}], + }], + "binaries": [] + } + } + }); + let (errors, _) = validate_l7_policies(&data); + assert!( + !errors.iter().any(|e| e.contains("would deny all traffic")), + "MCP with allow_all and empty allow should not be rejected as deny-all: {errors:?}" + ); + } + #[test] fn validate_deny_rules_empty_list_rejected() { let data = serde_json::json!({ diff --git a/crates/openshell-supervisor-network/src/l7/relay.rs b/crates/openshell-supervisor-network/src/l7/relay.rs index 4bf1bb0bc0..fa2eab4ad7 100644 --- a/crates/openshell-supervisor-network/src/l7/relay.rs +++ b/crates/openshell-supervisor-network/src/l7/relay.rs @@ -34,6 +34,7 @@ use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt}; use tracing::{debug, warn}; /// Context for L7 request policy evaluation. +#[cfg_attr(test, derive(Default))] pub struct L7EvalContext { /// Host from the CONNECT request. pub host: String, @@ -62,6 +63,8 @@ pub struct L7EvalContext { /// Dynamic token grant resolver for endpoint-bound credentials. pub(crate) token_grant_resolver: Option>, + /// Shared feature state for agent-driven policy proposals. + pub(crate) agent_proposals: openshell_core::proposals::AgentProposals, } #[derive(Default)] @@ -176,11 +179,13 @@ where engine_type_for_protocol(config.protocol), ); crate::l7::rest::RestProvider::default() - .deny( + .deny_with_redacted_target( req, &ctx.policy_name, crate::l7::rest::UNSUPPORTED_H2C_UPGRADE_DETAIL, client, + None, + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; Ok(true) @@ -296,11 +301,13 @@ where let Some(config) = select_l7_config_for_path(configs, &req.target) else { crate::l7::rest::RestProvider::default() - .deny( + .deny_with_redacted_target( &req, &ctx.policy_name, "no L7 endpoint path matched request", client, + None, + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -416,11 +423,7 @@ where "websocket endpoint requires a valid WebSocket upgrade request", client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -551,11 +554,7 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -884,11 +883,7 @@ where "websocket endpoint requires a valid WebSocket upgrade request", client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -1067,11 +1062,7 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -1299,11 +1290,7 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -1535,11 +1522,7 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext { - host: Some(&ctx.host), - port: Some(ctx.port), - binary: Some(&ctx.binary_path), - }), + Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), ) .await?; return Ok(()); @@ -2294,9 +2277,9 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, dynamic_credentials: Some(fixture.dynamic_credentials()), token_grant_resolver: Some(fixture.resolver()), + ..Default::default() }; (config, tunnel_engine, ctx, fixture) @@ -2361,9 +2344,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) @@ -2404,9 +2385,9 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, dynamic_credentials: Some(fixture.dynamic_credentials()), token_grant_resolver: Some(fixture.resolver()), + ..Default::default() }; (generation_guard, ctx, fixture) @@ -2451,9 +2432,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) } @@ -2497,9 +2476,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) } @@ -3012,9 +2989,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); @@ -3129,9 +3104,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let req = || crate::l7::provider::L7Request { action: "POST".into(), @@ -3342,9 +3315,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) } @@ -3506,9 +3477,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); @@ -3607,9 +3576,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) } @@ -3962,9 +3929,7 @@ network_policies: ancestors: Vec::new(), cmdline_paths: Vec::new(), secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let input = middleware_request_input( @@ -3996,9 +3961,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let req = crate::l7::provider::L7Request { action: "POST".into(), @@ -4187,9 +4150,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -4299,9 +4260,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -4514,9 +4473,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let request = L7RequestInfo { action: "WEBSOCKET_TEXT".into(), @@ -4591,9 +4548,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let mut request = L7RequestInfo { action: "POST".into(), @@ -4714,9 +4669,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let mut request = L7RequestInfo { action: "POST".into(), @@ -4779,9 +4732,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let mut request = L7RequestInfo { action: "POST".into(), @@ -4942,9 +4893,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); @@ -5041,9 +4990,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -5153,9 +5100,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: resolver.map(Arc::new), - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -5278,9 +5223,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: resolver.map(Arc::new), - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -5451,9 +5394,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); @@ -5544,9 +5485,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut app, mut relay_client) = tokio::io::duplex(8192); diff --git a/crates/openshell-supervisor-network/src/l7/rest.rs b/crates/openshell-supervisor-network/src/l7/rest.rs index 0c1346bf2e..fd9d373f81 100644 --- a/crates/openshell-supervisor-network/src/l7/rest.rs +++ b/crates/openshell-supervisor-network/src/l7/rest.rs @@ -130,6 +130,19 @@ pub(crate) struct DenyResponseContext<'a> { pub(crate) host: Option<&'a str>, pub(crate) port: Option, pub(crate) binary: Option<&'a str>, + pub(crate) agent_proposals_enabled: bool, +} + +impl<'a> DenyResponseContext<'a> { + #[must_use] + pub(crate) fn from_l7_context(ctx: &'a crate::l7::relay::L7EvalContext) -> Self { + Self { + host: Some(&ctx.host), + port: Some(ctx.port), + binary: Some(&ctx.binary_path), + agent_proposals_enabled: ctx.agent_proposals.enabled(), + } + } } impl RestProvider { @@ -2360,9 +2373,10 @@ fn deny_response_body( // module; this side picks up the change automatically. body.insert( "next_steps".to_string(), - crate::policy_local::agent_next_steps(), + crate::policy_local::agent_next_steps(context.agent_proposals_enabled), ); - if let Some(guidance) = crate::policy_local::agent_guidance() { + if let Some(guidance) = crate::policy_local::agent_guidance_for(context.agent_proposals_enabled) + { body.insert("agent_guidance".to_string(), serde_json::json!(guidance)); } @@ -3183,6 +3197,7 @@ mod tests { use super::*; use crate::opa::OpaEngine; use flate2::{Compress, Compression, Decompress, FlushCompress, FlushDecompress, Status}; + use openshell_core::proposals::AgentProposals; use openshell_core::secrets::SecretResolver; use std::pin::Pin; use std::sync::Arc; @@ -3350,6 +3365,31 @@ mod tests { .expect("WebSocket frame should arrive") } + async fn policy_local_json_response( + ctx: Arc, + ) -> serde_json::Value { + let (mut client, mut server) = tokio::io::duplex(4096); + let task = tokio::spawn(async move { + crate::policy_local::handle_forward_request( + ctx.as_ref(), + "GET", + "/v1/policy/current", + b"GET http://policy.local/v1/policy/current HTTP/1.1\r\nHost: policy.local\r\n\r\n", + &mut server, + ) + .await + .unwrap(); + }); + + let mut received = Vec::new(); + client.read_to_end(&mut received).await.unwrap(); + task.await.unwrap(); + + let response = String::from_utf8(received).unwrap(); + let (_, body) = response.split_once("\r\n\r\n").unwrap(); + serde_json::from_str(body).unwrap() + } + fn masked_frame_with_rsv(opcode: u8, rsv: u8, payload: &[u8]) -> Vec { let mask_key = [0x37, 0xfa, 0x21, 0x3d]; let mut frame = Vec::new(); @@ -3565,9 +3605,6 @@ mod tests { #[test] fn deny_response_body_is_agent_readable_and_redacted() { - // Agent-readable next_steps is gated on the proposals feature flag. - let _proposals = - openshell_core::proposals::test_helpers::ProposalsFlagGuard::set_blocking(true); let req = L7Request { action: "PUT".to_string(), target: "/repos/NVIDIA/OpenShell/contents/README.md?access_token=secret-token" @@ -3586,6 +3623,7 @@ mod tests { host: Some("api.github.com"), port: Some(443), binary: Some("/usr/bin/gh"), + agent_proposals_enabled: true, }), ); @@ -3632,8 +3670,6 @@ mod tests { #[test] fn deny_response_body_omits_agent_guidance_when_policy_advisor_is_off() { - let _proposals = - openshell_core::proposals::test_helpers::ProposalsFlagGuard::set_blocking(false); let req = L7Request { action: "GET".to_string(), target: "/gists".to_string(), @@ -3651,6 +3687,7 @@ mod tests { host: Some("api.github.com"), port: Some(443), binary: Some("/usr/bin/gh"), + agent_proposals_enabled: false, }), ); @@ -3662,10 +3699,76 @@ mod tests { ); } + #[tokio::test] + async fn shared_agent_proposals_state_gates_policy_local_and_l7_next_steps() { + let agent_proposals = AgentProposals::new(false); + let (_workspace_tx, workspace_rx) = tokio::sync::watch::channel(String::new()); + let policy_local_ctx = Arc::new(crate::policy_local::PolicyLocalContext::new( + Some(openshell_core::proto::SandboxPolicy { + version: 1, + ..Default::default() + }), + None, + None, + agent_proposals.clone(), + workspace_rx, + )); + let l7_ctx = crate::l7::relay::L7EvalContext { + host: "api.github.com".to_string(), + port: 443, + binary_path: "/usr/bin/gh".to_string(), + agent_proposals: agent_proposals.clone(), + ..Default::default() + }; + let req = L7Request { + action: "POST".to_string(), + target: "/user/repos".to_string(), + query_params: HashMap::new(), + raw_header: Vec::new(), + body_length: BodyLength::None, + }; + + let policy_body = policy_local_json_response(policy_local_ctx.clone()).await; + assert_eq!(policy_body["error"], "feature_disabled"); + let deny_body = deny_response_body( + &req, + "github-readonly", + "no matching L7 allow rule", + None, + Some(DenyResponseContext::from_l7_context(&l7_ctx)), + ); + assert_eq!(deny_body["next_steps"], serde_json::json!([])); + assert!(deny_body.get("agent_guidance").is_none()); + + agent_proposals.set_enabled(true); + + let policy_body = policy_local_json_response(policy_local_ctx).await; + assert_eq!(policy_body["format"], "yaml"); + assert!( + policy_body["policy_yaml"] + .as_str() + .unwrap() + .contains("version: 1") + ); + let deny_body = deny_response_body( + &req, + "github-readonly", + "no matching L7 allow rule", + None, + Some(DenyResponseContext::from_l7_context(&l7_ctx)), + ); + assert_eq!(deny_body["next_steps"][0]["action"], "read_skill"); + assert_eq!(deny_body["next_steps"][3]["action"], "submit_proposal"); + assert!( + deny_body["agent_guidance"] + .as_str() + .unwrap() + .contains("http://policy.local/v1/proposals") + ); + } + #[test] fn middleware_deny_response_identifies_config_without_policy_remediation() { - let _proposals = - openshell_core::proposals::test_helpers::ProposalsFlagGuard::set_blocking(true); let req = L7Request { action: "POST".to_string(), target: "/v1/messages?token=secret-token".to_string(), @@ -3687,6 +3790,7 @@ mod tests { host: Some("api.example.com"), port: Some(443), binary: Some("/usr/bin/curl"), + agent_proposals_enabled: true, }), ); @@ -3705,9 +3809,6 @@ mod tests { #[tokio::test] async fn send_deny_response_writes_structured_json_403() { - // Agent-readable next_steps is gated on the proposals feature flag. - let _proposals = - openshell_core::proposals::test_helpers::ProposalsFlagGuard::set(true).await; let (mut client, mut server) = tokio::io::duplex(4096); let send = tokio::spawn(async move { let req = L7Request { @@ -3727,6 +3828,7 @@ mod tests { host: Some("api.github.com"), port: Some(443), binary: Some("/usr/bin/gh"), + agent_proposals_enabled: true, }), ) .await diff --git a/crates/openshell-supervisor-network/src/l7/tls.rs b/crates/openshell-supervisor-network/src/l7/tls.rs index 70e198f420..f7c923c690 100644 --- a/crates/openshell-supervisor-network/src/l7/tls.rs +++ b/crates/openshell-supervisor-network/src/l7/tls.rs @@ -184,7 +184,7 @@ pub async fn tls_terminate_client( /// /// Returns a TLS stream for re-encrypted upstream communication. pub async fn tls_connect_upstream( - upstream: TcpStream, + upstream: impl AsyncRead + AsyncWrite + Unpin + Send, hostname: &str, client_config: &Arc, ) -> Result { diff --git a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs index 07a8c641f1..2ca25e6e81 100644 --- a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs +++ b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs @@ -721,9 +721,9 @@ mod tests { ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, dynamic_credentials: Some(fixture.dynamic_credentials()), token_grant_resolver: Some(fixture.resolver()), + ..Default::default() }; let req = L7Request { action: "GET".to_string(), @@ -758,9 +758,9 @@ mod tests { ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, dynamic_credentials: Some(fixture.dynamic_credentials()), token_grant_resolver: Some(fixture.resolver()), + ..Default::default() }; let req = L7Request { action: "GET".to_string(), diff --git a/crates/openshell-supervisor-network/src/l7/websocket.rs b/crates/openshell-supervisor-network/src/l7/websocket.rs index e1f92e6ece..bb59c5227b 100644 --- a/crates/openshell-supervisor-network/src/l7/websocket.rs +++ b/crates/openshell-supervisor-network/src/l7/websocket.rs @@ -1273,9 +1273,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let (mut client_write, mut relay_read) = tokio::io::duplex(MAX_TEXT_MESSAGE_BYTES + 1024); let (mut relay_write, mut upstream_read) = tokio::io::duplex(MAX_TEXT_MESSAGE_BYTES + 1024); diff --git a/crates/openshell-supervisor-network/src/lib.rs b/crates/openshell-supervisor-network/src/lib.rs index c6c1af5aed..ac0cb120a2 100644 --- a/crates/openshell-supervisor-network/src/lib.rs +++ b/crates/openshell-supervisor-network/src/lib.rs @@ -19,3 +19,4 @@ pub mod run; pub mod sigv4; mod spiffe_endpoint; mod token_grant; +pub mod upstream_proxy; diff --git a/crates/openshell-supervisor-network/src/policy_local.rs b/crates/openshell-supervisor-network/src/policy_local.rs index 53c038dec1..e915c18c9b 100644 --- a/crates/openshell-supervisor-network/src/policy_local.rs +++ b/crates/openshell-supervisor-network/src/policy_local.rs @@ -4,6 +4,7 @@ //! Sandbox-local policy advisor HTTP API. use miette::{IntoDiagnostic, Result}; +use openshell_core::proposals::AgentProposals; use openshell_core::proto::{ L7Allow, L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, PolicyChunk, SandboxPolicy as ProtoSandboxPolicy, @@ -85,6 +86,7 @@ const MAX_DENIAL_LINE_BYTES: usize = 4096; #[derive(Debug)] pub struct PolicyLocalContext { current_policy: Arc>>, + agent_proposals: AgentProposals, gateway_endpoint: Option, sandbox_name: Option, shorthand_log_dir: PathBuf, @@ -96,12 +98,14 @@ impl PolicyLocalContext { current_policy: Option, gateway_endpoint: Option, sandbox_name: Option, + agent_proposals: AgentProposals, workspace_rx: tokio::sync::watch::Receiver, ) -> Self { Self::with_log_dir( current_policy, gateway_endpoint, sandbox_name, + agent_proposals, PathBuf::from(LOG_DIR), workspace_rx, ) @@ -111,11 +115,13 @@ impl PolicyLocalContext { current_policy: Option, gateway_endpoint: Option, sandbox_name: Option, + agent_proposals: AgentProposals, shorthand_log_dir: PathBuf, workspace_rx: tokio::sync::watch::Receiver, ) -> Self { Self { current_policy: Arc::new(RwLock::new(current_policy)), + agent_proposals, gateway_endpoint, sandbox_name, shorthand_log_dir, @@ -126,6 +132,16 @@ impl PolicyLocalContext { pub async fn set_current_policy(&self, policy: ProtoSandboxPolicy) { *self.current_policy.write().await = Some(policy); } + + #[must_use] + pub fn agent_proposals(&self) -> AgentProposals { + self.agent_proposals.clone() + } + + #[must_use] + pub fn agent_proposals_enabled(&self) -> bool { + self.agent_proposals.enabled() + } } pub async fn handle_forward_request( @@ -154,7 +170,7 @@ async fn route_request( // when the flag is off — including the diagnostic `current_policy` and // `denials` routes. The skill is also not installed in that mode, so a // disabled sandbox has no entry point into this API at all. - if !openshell_core::proposals::agent_proposals_enabled() { + if !ctx.agent_proposals_enabled() { return ( 404, serde_json::json!({ @@ -223,12 +239,12 @@ fn not_found_payload(path: &str) -> (u16, serde_json::Value) { /// the deny body and the actual route table from drifting — adding or /// renaming a route only requires touching the route constants above. /// -/// Returns an empty array when `agent_proposals_enabled()` is false so a +/// Returns an empty array when agent policy proposals are disabled so a /// disabled sandbox doesn't advertise a surface that 404s. The deny body /// caller still emits the field (with `[]`) so the wire shape is stable. #[must_use] -pub fn agent_next_steps() -> serde_json::Value { - if !openshell_core::proposals::agent_proposals_enabled() { +pub fn agent_next_steps(agent_proposals_enabled: bool) -> serde_json::Value { + if !agent_proposals_enabled { return serde_json::json!([]); } let host = POLICY_LOCAL_HOST; @@ -258,8 +274,8 @@ pub fn agent_next_steps() -> serde_json::Value { /// Build the optional natural-language guidance embedded in L7 deny bodies. #[must_use] -pub fn agent_guidance() -> Option<&'static str> { - openshell_core::proposals::agent_proposals_enabled().then_some(AGENT_GUIDANCE) +pub fn agent_guidance_for(agent_proposals_enabled: bool) -> Option<&'static str> { + agent_proposals_enabled.then_some(AGENT_GUIDANCE) } async fn current_policy_response(ctx: &PolicyLocalContext) -> (u16, serde_json::Value) { @@ -1355,6 +1371,7 @@ struct L7DenyRuleJson { #[cfg(test)] mod tests { use super::*; + use openshell_core::proposals::AgentProposals; fn test_workspace_rx() -> tokio::sync::watch::Receiver { tokio::sync::watch::channel(String::new()).1 @@ -1505,6 +1522,7 @@ mod tests { None, None, None, + AgentProposals::new(true), dir.path().to_path_buf(), test_workspace_rx(), ); @@ -1542,6 +1560,7 @@ mod tests { None, None, None, + AgentProposals::new(true), dir.path().to_path_buf(), test_workspace_rx(), ); @@ -1558,6 +1577,7 @@ mod tests { None, None, None, + AgentProposals::new(true), dir.path().to_path_buf(), test_workspace_rx(), ); @@ -1627,6 +1647,7 @@ mod tests { None, None, None, + AgentProposals::new(true), dir.path().to_path_buf(), test_workspace_rx(), ); @@ -1638,12 +1659,9 @@ mod tests { assert!(surfaced.ends_with("...[truncated]")); } - use openshell_core::proposals::test_helpers::ProposalsFlagGuard; - #[test] fn agent_next_steps_returns_empty_when_flag_off() { - let _guard = ProposalsFlagGuard::set_blocking(false); - let steps = agent_next_steps(); + let steps = agent_next_steps(false); let arr = steps.as_array().expect("agent_next_steps is an array"); assert!( arr.is_empty(), @@ -1653,8 +1671,7 @@ mod tests { #[test] fn agent_next_steps_returns_full_array_when_flag_on() { - let _guard = ProposalsFlagGuard::set_blocking(true); - let steps = agent_next_steps(); + let steps = agent_next_steps(true); let arr = steps.as_array().expect("agent_next_steps is an array"); assert_eq!(arr.len(), 4, "expected 4 next_steps when feature is on"); let actions: Vec<&str> = arr @@ -1667,14 +1684,12 @@ mod tests { #[test] fn agent_guidance_is_absent_when_flag_off() { - let _guard = ProposalsFlagGuard::set_blocking(false); - assert!(agent_guidance().is_none()); + assert!(agent_guidance_for(false).is_none()); } #[test] fn agent_guidance_points_to_policy_advisor_when_flag_on() { - let _guard = ProposalsFlagGuard::set_blocking(true); - let guidance = agent_guidance().expect("guidance when proposals are enabled"); + let guidance = agent_guidance_for(true).expect("guidance when proposals are enabled"); assert!(guidance.contains("do not stop")); assert!(guidance.contains("/etc/openshell/skills/policy_advisor.md")); assert!(guidance.contains("http://policy.local/v1/proposals")); @@ -1683,7 +1698,6 @@ mod tests { #[tokio::test] async fn route_request_returns_feature_disabled_when_flag_off() { - let _guard = ProposalsFlagGuard::set(false).await; let ctx = PolicyLocalContext::new( Some(ProtoSandboxPolicy { version: 1, @@ -1691,6 +1705,7 @@ mod tests { }), None, None, + AgentProposals::new(false), test_workspace_rx(), ); @@ -1711,7 +1726,6 @@ mod tests { #[tokio::test] async fn current_policy_route_returns_yaml_envelope() { - let _guard = ProposalsFlagGuard::set(true).await; let ctx = PolicyLocalContext::new( Some(ProtoSandboxPolicy { version: 1, @@ -1719,6 +1733,7 @@ mod tests { }), None, None, + AgentProposals::new(true), test_workspace_rx(), ); @@ -1812,8 +1827,13 @@ mod tests { #[tokio::test] async fn proposal_routes_reject_malformed_paths() { - let _guard = ProposalsFlagGuard::set(true).await; - let ctx = PolicyLocalContext::new(None, None, None, test_workspace_rx()); + let ctx = PolicyLocalContext::new( + None, + None, + None, + AgentProposals::new(true), + test_workspace_rx(), + ); // Empty chunk_id after the prefix is 404, not a wildcard list. let (status, _) = route_request(&ctx, "GET", "/v1/proposals/", &[]).await; @@ -1833,11 +1853,11 @@ mod tests { #[tokio::test] async fn proposal_status_route_returns_503_when_no_gateway() { - let _guard = ProposalsFlagGuard::set(true).await; let ctx = PolicyLocalContext::new( None, None, Some("test-sandbox".to_string()), + AgentProposals::new(true), test_workspace_rx(), ); @@ -1848,11 +1868,11 @@ mod tests { #[tokio::test] async fn proposal_wait_route_returns_503_when_no_gateway() { - let _guard = ProposalsFlagGuard::set(true).await; let ctx = PolicyLocalContext::new( None, None, Some("test-sandbox".to_string()), + AgentProposals::new(true), test_workspace_rx(), ); @@ -1864,11 +1884,11 @@ mod tests { #[tokio::test] async fn proposal_routes_return_feature_disabled_when_flag_off() { - let _guard = ProposalsFlagGuard::set(false).await; let ctx = PolicyLocalContext::new( None, None, Some("test-sandbox".to_string()), + AgentProposals::new(false), test_workspace_rx(), ); @@ -1950,6 +1970,7 @@ mod tests { Some(policy_with_rule(proposed.clone())), None, None, + AgentProposals::new(false), test_workspace_rx(), ); let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(2); @@ -1977,7 +1998,13 @@ mod tests { version: 1, ..Default::default() }; - let ctx = PolicyLocalContext::new(Some(initial), None, None, test_workspace_rx()); + let ctx = PolicyLocalContext::new( + Some(initial), + None, + None, + AgentProposals::new(false), + test_workspace_rx(), + ); // Concurrently, an unrelated rule lands. We must not return. let unrelated_load = { @@ -2028,6 +2055,7 @@ mod tests { }), None, None, + AgentProposals::new(false), test_workspace_rx(), ); @@ -2066,6 +2094,7 @@ mod tests { }), None, None, + AgentProposals::new(false), test_workspace_rx(), ); let deadline = tokio::time::Instant::now() + std::time::Duration::from_millis(300); diff --git a/crates/openshell-supervisor-network/src/procfs.rs b/crates/openshell-supervisor-network/src/procfs.rs index 3ac8dbe141..8bc2fbb110 100644 --- a/crates/openshell-supervisor-network/src/procfs.rs +++ b/crates/openshell-supervisor-network/src/procfs.rs @@ -9,6 +9,9 @@ use miette::Result; #[cfg(target_os = "linux")] use std::collections::HashSet; +use std::net::SocketAddr; +#[cfg(target_os = "linux")] +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; use std::path::Path; #[cfg(target_os = "linux")] use std::path::PathBuf; @@ -40,6 +43,46 @@ pub struct TcpPeerSocketOwners { pub owners: Vec, } +/// TCP endpoints for a workload connection accepted by the sandbox proxy. +/// +/// `/proc//net/tcp{,6}` reports the socket from the workload side, so its +/// local endpoint should match `workload` and its remote endpoint should match +/// `proxy`. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct WorkloadProxyTcpConnection { + pub workload: SocketAddr, + pub proxy: SocketAddr, +} + +impl WorkloadProxyTcpConnection { + pub fn new(workload: SocketAddr, proxy: SocketAddr) -> Self { + Self { workload, proxy } + } +} + +impl std::fmt::Display for WorkloadProxyTcpConnection { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{} -> {}", self.workload, self.proxy) + } +} + +#[cfg(target_os = "linux")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum ProcTcpAddressFamily { + Ipv4, + Ipv6, +} + +#[cfg(target_os = "linux")] +impl ProcTcpAddressFamily { + fn table_name(self) -> &'static str { + match self { + Self::Ipv4 => "tcp", + Self::Ipv6 => "tcp6", + } + } +} + #[cfg(target_os = "linux")] #[derive(Clone, Debug, Eq, PartialEq)] struct DescendantPid { @@ -116,11 +159,14 @@ pub fn binary_path(pid: i32) -> Result { /// Resolve the binary path of the TCP peer inside a sandbox network namespace. /// /// Uses `/proc//net/tcp` to find the socket inode for the given -/// ephemeral port, then scans the entrypoint process tree to find which PID owns -/// that socket, and finally reads `/proc//exe` to get the binary path. +/// connection tuple, then scans the entrypoint process tree to find which PID +/// owns that socket, and finally reads `/proc//exe` to get the binary path. #[cfg(target_os = "linux")] -pub fn resolve_tcp_peer_binary(entrypoint_pid: u32, peer_port: u16) -> Result { - let owner = resolve_single_tcp_peer_owner(entrypoint_pid, peer_port)?; +pub fn resolve_tcp_peer_binary( + entrypoint_pid: u32, + connection: WorkloadProxyTcpConnection, +) -> Result { + let owner = resolve_single_tcp_peer_owner(entrypoint_pid, connection)?; binary_path(owner.pid.cast_signed()) } @@ -132,17 +178,20 @@ pub fn resolve_tcp_peer_binary(entrypoint_pid: u32, peer_port: u16) -> Result Result { - let inode = parse_proc_net_tcp(entrypoint_pid, peer_port)?; + let inode = parse_proc_net_tcp(entrypoint_pid, connection)?; let owners = find_socket_inode_owners(inode, entrypoint_pid)?; Ok(TcpPeerSocketOwners { inode, owners }) } /// Resolve exactly one owner for the TCP peer, failing closed on ambiguity. #[cfg(target_os = "linux")] -fn resolve_single_tcp_peer_owner(entrypoint_pid: u32, peer_port: u16) -> Result { - let socket_owners = resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port)?; +fn resolve_single_tcp_peer_owner( + entrypoint_pid: u32, + connection: WorkloadProxyTcpConnection, +) -> Result { + let socket_owners = resolve_tcp_peer_socket_owners(entrypoint_pid, connection)?; match socket_owners.owners.as_slice() { [owner] => Ok(owner.clone()), owners => { @@ -164,8 +213,11 @@ fn resolve_single_tcp_peer_owner(entrypoint_pid: u32, peer_port: u16) -> Result< /// /// Needed for the ancestor walk: we must know the PID to walk `/proc//status` `PPid` chain. #[cfg(target_os = "linux")] -pub fn resolve_tcp_peer_identity(entrypoint_pid: u32, peer_port: u16) -> Result<(PathBuf, u32)> { - let owner = resolve_single_tcp_peer_owner(entrypoint_pid, peer_port)?; +pub fn resolve_tcp_peer_identity( + entrypoint_pid: u32, + connection: WorkloadProxyTcpConnection, +) -> Result<(PathBuf, u32)> { + let owner = resolve_single_tcp_peer_owner(entrypoint_pid, connection)?; let path = binary_path(owner.pid.cast_signed())?; Ok((path, owner.pid)) } @@ -274,7 +326,7 @@ pub fn collect_cmdline_paths(pid: u32, stop_pid: u32, exclude: &[PathBuf]) -> Ve } /// Parse `/proc//net/tcp` (and `/proc//net/tcp6`) to find the socket -/// inode for a given local port. +/// inode for a complete local-to-remote TCP tuple. /// /// Checks both IPv4 and IPv6 tables because some clients (notably gRPC C-core) /// use `AF_INET6` sockets with IPv4-mapped addresses even for IPv4 connections. @@ -288,52 +340,96 @@ pub fn collect_cmdline_paths(pid: u32, stop_pid: u32, exclude: &[PathBuf]) -> Ve /// - State `01` = ESTABLISHED /// - Inode is field index 9 (0-indexed) #[cfg(target_os = "linux")] -fn parse_proc_net_tcp(pid: u32, peer_port: u16) -> Result { +fn parse_proc_net_tcp(pid: u32, connection: WorkloadProxyTcpConnection) -> Result { // Check IPv4 first (most common), then IPv6. - for suffix in &["tcp", "tcp6"] { - let path = format!("/proc/{pid}/net/{suffix}"); + for address_family in [ProcTcpAddressFamily::Ipv4, ProcTcpAddressFamily::Ipv6] { + let path = format!("/proc/{pid}/net/{}", address_family.table_name()); let Ok(content) = std::fs::read_to_string(&path) else { continue; }; - for line in content.lines().skip(1) { - let fields: Vec<&str> = line.split_whitespace().collect(); - if fields.len() < 10 { - continue; - } + if let Some(inode) = find_proc_net_tcp_inode(&content, address_family, connection)? { + return Ok(inode); + } + } - // Parse local_address to extract port. - // IPv4 format: AABBCCDD:PORT - // IPv6 format: 00000000000000000000000000000000:PORT - let local_addr = fields[1]; - let local_port = match local_addr.rsplit_once(':') { - Some((_, port_hex)) => u16::from_str_radix(port_hex, 16).unwrap_or(0), - None => continue, - }; - - // Check state is ESTABLISHED (01) - let state = fields[3]; - if state != "01" { - continue; - } + Err(miette::miette!( + "No ESTABLISHED TCP connection found for {connection} in /proc/{pid}/net/tcp{{,6}}" + )) +} - if local_port == peer_port { - let inode: u64 = fields[9] - .parse() - .map_err(|_| miette::miette!("Failed to parse inode from {}", fields[9]))?; - if inode == 0 { - continue; - } - return Ok(inode); +#[cfg(target_os = "linux")] +fn find_proc_net_tcp_inode( + content: &str, + address_family: ProcTcpAddressFamily, + connection: WorkloadProxyTcpConnection, +) -> Result> { + for line in content.lines().skip(1) { + let fields: Vec<&str> = line.split_whitespace().collect(); + if fields.len() < 10 || fields[3] != "01" { + continue; + } + + let Some(local_addr) = parse_proc_socket_addr(fields[1], address_family) else { + continue; + }; + let Some(remote_addr) = parse_proc_socket_addr(fields[2], address_family) else { + continue; + }; + + if socket_addrs_match(local_addr, connection.workload) + && socket_addrs_match(remote_addr, connection.proxy) + { + let inode: u64 = fields[9] + .parse() + .map_err(|_| miette::miette!("Failed to parse inode from {}", fields[9]))?; + if inode != 0 { + return Ok(Some(inode)); } } } - Err(miette::miette!( - "No ESTABLISHED TCP connection found for port {} in /proc/{}/net/tcp{{,6}}", - peer_port, - pid - )) + Ok(None) +} + +#[cfg(target_os = "linux")] +fn parse_proc_socket_addr(value: &str, address_family: ProcTcpAddressFamily) -> Option { + let (address, port) = value.rsplit_once(':')?; + let port = u16::from_str_radix(port, 16).ok()?; + + let ip = match address_family { + ProcTcpAddressFamily::Ipv4 => { + let address = u32::from_str_radix(address, 16).ok()?; + IpAddr::V4(Ipv4Addr::from(address.to_le_bytes())) + } + ProcTcpAddressFamily::Ipv6 => { + if address.len() != 32 { + return None; + } + let mut bytes = [0u8; 16]; + for (index, chunk) in address.as_bytes().chunks_exact(8).enumerate() { + let chunk = std::str::from_utf8(chunk).ok()?; + let word = u32::from_str_radix(chunk, 16).ok()?; + bytes[index * 4..index * 4 + 4].copy_from_slice(&word.to_le_bytes()); + } + IpAddr::V6(Ipv6Addr::from(bytes)) + } + }; + + Some(SocketAddr::new(ip, port)) +} + +#[cfg(target_os = "linux")] +fn socket_addrs_match(left: SocketAddr, right: SocketAddr) -> bool { + left.port() == right.port() && normalize_ip(left.ip()) == normalize_ip(right.ip()) +} + +#[cfg(target_os = "linux")] +fn normalize_ip(ip: IpAddr) -> IpAddr { + match ip { + IpAddr::V6(ipv6) => ipv6.to_ipv4_mapped().map_or(IpAddr::V6(ipv6), IpAddr::V4), + ipv4 @ IpAddr::V4(_) => ipv4, + } } /// Scan `/proc` to find every PID that owns a given socket inode. @@ -756,6 +852,38 @@ mod tests { assert_eq!(pids.len(), unique.len()); } + #[cfg(target_os = "linux")] + #[test] + fn parse_proc_socket_addr_decodes_ipv4_and_mapped_ipv6() { + let ipv4 = parse_proc_socket_addr("0100007F:C350", ProcTcpAddressFamily::Ipv4).unwrap(); + let mapped_ipv6 = parse_proc_socket_addr( + "0000000000000000FFFF00000100007F:C350", + ProcTcpAddressFamily::Ipv6, + ) + .unwrap(); + + assert_eq!(ipv4, "127.0.0.1:50000".parse().unwrap()); + assert!(socket_addrs_match(ipv4, mapped_ipv6)); + } + + #[cfg(target_os = "linux")] + #[test] + fn find_proc_net_tcp_inode_matches_complete_tuple() { + let content = "\ + sl local_address rem_address st tx_queue:rx_queue tr:tm->when retrnsmt uid timeout inode\n\ + 0: 0100007F:C350 0100007F:1F91 01 00000000:00000000 00:00000000 00000000 1000 0 11111\n\ + 1: 0100007F:C350 0100007F:1F90 01 00000000:00000000 00:00000000 00000000 1000 0 22222\n"; + let connection = WorkloadProxyTcpConnection::new( + "127.0.0.1:50000".parse().unwrap(), + "127.0.0.1:8080".parse().unwrap(), + ); + + let inode = + find_proc_net_tcp_inode(content, ProcTcpAddressFamily::Ipv4, connection).unwrap(); + + assert_eq!(inode, Some(22222)); + } + #[cfg(target_os = "linux")] #[test] fn resolve_tcp_peer_socket_owners_returns_all_forked_socket_holders() { @@ -774,9 +902,10 @@ mod tests { } let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); - let listener_port = listener.local_addr().unwrap().port(); - let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); - let peer_port = stream.local_addr().unwrap().port(); + let proxy_addr = listener.local_addr().unwrap(); + let stream = TcpStream::connect(proxy_addr).expect("connect"); + let workload_addr = stream.local_addr().unwrap(); + let connection = WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); let (_accepted, _) = listener.accept().expect("accept"); // libc/syscall FFI requires unsafe @@ -797,7 +926,7 @@ mod tests { let entrypoint_pid = std::process::id(); let deadline = Instant::now() + Duration::from_secs(5); let owners = loop { - let owners = resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port) + let owners = resolve_tcp_peer_socket_owners(entrypoint_pid, connection) .expect("resolve socket owners"); let owner_pids = owners .owners diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index e7609811ff..69245e11e1 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -7,6 +7,7 @@ use crate::identity::BinaryIdentityCache; use crate::l7::tls::ProxyTlsState; use crate::opa::{NetworkAction, OpaEngine, PolicyGenerationGuard}; use crate::policy_local::{POLICY_LOCAL_HOST, PolicyLocalContext}; +use crate::upstream_proxy::{self, UpstreamProxyConfig}; use miette::{IntoDiagnostic, Result}; use openshell_core::activity::{ActivitySender, try_record_activity}; use openshell_core::denial::DenialEvent; @@ -200,6 +201,7 @@ impl ProxyHandle { denial_tx: Option>, activity_tx: Option, engine_ready: tokio::sync::watch::Receiver, + upstream_proxy_args: &upstream_proxy::UpstreamProxyArgs, ) -> Result { // Use override bind_addr, fall back to policy http_addr, then default // to loopback:3128. The default allows the proxy to function when no @@ -239,6 +241,48 @@ impl ProxyHandle { host-gateway aliases exempt from SSRF always-blocked check" ); } + let agent_proposals = policy_local_ctx + .as_ref() + .map_or_else(Default::default, |ctx| ctx.agent_proposals()); + + // Corporate egress proxy configured by the operator and delivered on + // the supervisor's command line by the compute driver; the + // conventional HTTPS_PROXY/NO_PROXY variables the sandbox controls + // are ignored here. + // + // This is an operator-owned security boundary, so a present-but-invalid + // value (bad proxy URL, unreadable auth file, malformed credential) is + // fatal to proxy startup: failing closed prevents a misconfiguration + // from silently degrading to direct dialing or unauthenticated proxy + // access. + let upstream_proxy: Arc> = Arc::new( + UpstreamProxyConfig::from_args(upstream_proxy_args).map_err(|err| { + let event = + openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(openshell_ocsf::StateId::Disabled, "invalid") + .message(format!( + "Upstream corporate proxy configuration invalid; \ + refusing to start: {err}" + )) + .build(); + ocsf_emit!(event); + miette::miette!("invalid upstream corporate proxy configuration: {err}") + })?, + ); + if let Some(cfg) = upstream_proxy.as_ref() { + let event = openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(openshell_ocsf::StateId::Enabled, "enabled") + .message(format!( + "Upstream corporate proxy enabled: {}", + cfg.summary() + )) + .build(); + ocsf_emit!(event); + } let join = tokio::spawn(async move { // Wait for the OPA engine's symlink resolution reload to complete @@ -274,7 +318,9 @@ impl ProxyHandle { let tls = tls_state.clone(); let inf = inference_ctx.clone(); let policy_local = policy_local_ctx.clone(); + let proposals = agent_proposals.clone(); let gw = trusted_host_gateway.clone(); + let up_proxy = upstream_proxy.clone(); let resolver = provider_credentials .as_ref() .and_then(ProviderCredentialState::resolver); @@ -295,7 +341,9 @@ impl ProxyHandle { tls, inf, policy_local, + proposals, gw, + up_proxy, resolver, dynamic_credentials, dtx, @@ -626,7 +674,9 @@ async fn handle_tcp_connection( tls_state: Option>, inference_ctx: Option>, policy_local_ctx: Option>, + agent_proposals: openshell_core::proposals::AgentProposals, trusted_host_gateway: Arc>, + upstream_proxy: Arc>, secret_resolver: Option>, dynamic_credentials: Option< Arc< @@ -694,6 +744,7 @@ async fn handle_tcp_connection( identity_cache, entrypoint_pid, policy_local_ctx, + agent_proposals, trusted_host_gateway, secret_resolver, dynamic_credentials, @@ -733,8 +784,9 @@ async fn handle_tcp_connection( return Ok(()); } - let peer_addr = client.peer_addr().into_diagnostic()?; - let _local_addr = client.local_addr().into_diagnostic()?; + let workload_addr = client.peer_addr().into_diagnostic()?; + let proxy_addr = client.local_addr().into_diagnostic()?; + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); // Evaluate OPA policy with process-identity binding. // Wrapped in spawn_blocking because identity resolution does heavy sync I/O: @@ -745,7 +797,7 @@ async fn handle_tcp_connection( let host_clone = host_lc.clone(); let decision = tokio::task::spawn_blocking(move || { evaluate_opa_tcp( - peer_addr, + connection, &opa_clone, &cache_clone, &pid_clone, @@ -803,7 +855,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -825,11 +877,12 @@ async fn handle_tcp_connection( emit_activity(&activity_tx, true, "connect_policy"); respond( &mut client, - &build_json_error_response( + &build_json_error_response_with_reason( 403, "Forbidden", "policy_denied", &format!("CONNECT {host_lc}:{port} not permitted by policy"), + &deny_reason, ), ) .await?; @@ -885,7 +938,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -939,7 +992,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -987,7 +1040,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1039,7 +1092,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1089,7 +1142,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1144,7 +1197,7 @@ async fn handle_tcp_connection( .severity(SeverityId::High) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1169,7 +1222,7 @@ async fn handle_tcp_connection( return Ok(()); } - let mut upstream = TcpStream::connect(validated_addrs.as_slice()) + let mut upstream = dial_upstream(&upstream_proxy, &host_lc, port, &validated_addrs) .await .into_diagnostic()?; @@ -1200,7 +1253,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Informational) .status(StatusId::Success) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1241,6 +1294,7 @@ async fn handle_tcp_connection( token_grant_resolver: dynamic_credentials .as_ref() .map(|_| crate::l7::token_grant_injection::default_resolver()), + agent_proposals, }; if effective_tls_skip { @@ -1393,7 +1447,7 @@ async fn handle_tcp_connection( .severity(SeverityId::High) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1516,7 +1570,7 @@ async fn handle_tcp_connection( .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .src_endpoint_addr(workload_addr.ip(), workload_addr.port()) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -1717,16 +1771,16 @@ fn collect_ancestor_identities(start_pid: u32, stop_pid: u32) -> Vec<(u32, PathB #[cfg(target_os = "linux")] fn resolve_process_identity( entrypoint_pid: u32, - peer_port: u16, + connection: crate::procfs::WorkloadProxyTcpConnection, identity_cache: &BinaryIdentityCache, ) -> std::result::Result { - let socket_owners = crate::procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port) + let socket_owners = crate::procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, connection) .map_err(|e| IdentityError { - reason: format!("failed to resolve peer binary: {e}"), - binary: None, - binary_pid: None, - ancestors: vec![], - })?; + reason: format!("failed to resolve peer binary: {e}"), + binary: None, + binary_pid: None, + ancestors: vec![], + })?; let mut identities = Vec::with_capacity(socket_owners.owners.len()); for owner in &socket_owners.owners { @@ -1785,7 +1839,7 @@ fn resolve_process_identity( /// Evaluate OPA policy for a TCP connection with identity binding via /proc/net/tcp. #[cfg(target_os = "linux")] fn evaluate_opa_tcp( - peer_addr: SocketAddr, + connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, identity_cache: &BinaryIdentityCache, entrypoint_pid: &AtomicU32, @@ -1832,9 +1886,7 @@ fn evaluate_opa_tcp( }; let total_start = std::time::Instant::now(); - let peer_port = peer_addr.port(); - - let identity = match resolve_process_identity(proc_net_anchor_pid, peer_port, identity_cache) { + let identity = match resolve_process_identity(proc_net_anchor_pid, connection, identity_cache) { Ok(id) => id, Err(err) => { return deny( @@ -1937,7 +1989,7 @@ fn evaluate_endpoint_only_opa(engine: &OpaEngine, host: &str, port: u16) -> Conn /// Non-Linux stub: OPA identity binding requires /proc. #[cfg(not(target_os = "linux"))] fn evaluate_opa_tcp( - _peer_addr: SocketAddr, + _connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, _identity_cache: &BinaryIdentityCache, _entrypoint_pid: &AtomicU32, @@ -2903,6 +2955,67 @@ fn validate_declared_endpoint_resolved_addrs( Ok(()) } +/// Dial a validated upstream destination for a TLS (CONNECT) tunnel. +/// +/// Connects directly to the SSRF-checked resolved addresses, or chains +/// through the corporate proxy (HTTP CONNECT) when one is configured for +/// this destination via the driver-supplied upstream proxy arguments +/// and not excluded by the operator `NO_PROXY` list. Policy evaluation and +/// SSRF validation must have already succeeded; only the final TCP dial +/// changes. Plain-HTTP requests never take this path: they always dial the +/// destination directly. +/// +/// `NO_PROXY` evaluation is port-aware and sees the validated resolved +/// addresses: an entry with a `:port` qualifier only bypasses that port, +/// and an IP/CIDR entry that matches through resolution limits the direct +/// dial to the addresses it contains. +/// +/// The CONNECT target sent to the corporate proxy is a validated resolved +/// address, so the proxy performs no DNS resolution of its own and the +/// tunnel stays bound to the answer that passed SSRF and `allowed_ips` +/// validation; the hostname still travels inside the tunnel (TLS SNI, +/// application `Host`). The operator opt-in `proxy_connect_by_hostname` +/// sends the client-requested hostname instead, for proxies whose ACLs +/// filter on hostnames, at the cost of re-opening proxy-side resolution. +/// +/// Both paths return a [`upstream_proxy::PrefixedStream`]: for proxied +/// dials it replays any tunneled bytes that arrived in the same read as the +/// CONNECT response; for direct dials it is a plain passthrough. +async fn dial_upstream( + upstream_proxy: &Option, + host_lc: &str, + port: u16, + addrs: &[SocketAddr], +) -> std::io::Result { + if let Some(cfg) = upstream_proxy.as_ref() { + return match cfg.decision(host_lc, port, addrs) { + upstream_proxy::ProxyDecision::Proxy(endpoint) => { + if cfg.connect_by_hostname() { + upstream_proxy::connect_via( + endpoint, + host_lc, + port, + upstream_proxy::ConnectTarget::Hostname, + ) + .await + } else { + // Try every validated address in order, matching the + // fallback the direct path's `TcpStream::connect` does. + upstream_proxy::connect_via_validated(endpoint, host_lc, port, addrs).await + } + } + upstream_proxy::ProxyDecision::Direct(direct_addrs) => { + Ok(upstream_proxy::PrefixedStream::without_prefix( + TcpStream::connect(&direct_addrs[..]).await?, + )) + } + }; + } + Ok(upstream_proxy::PrefixedStream::without_prefix( + TcpStream::connect(addrs).await?, + )) +} + /// Resolve a host:port using sandbox `/etc/hosts` first (when available), then /// reject if any resolved address is internal. /// @@ -3541,6 +3654,7 @@ async fn handle_forward_proxy( identity_cache: Arc, entrypoint_pid: Arc, policy_local_ctx: Option>, + agent_proposals: openshell_core::proposals::AgentProposals, trusted_host_gateway: Arc>, secret_resolver: Option>, dynamic_credentials: Option< @@ -3644,8 +3758,9 @@ async fn handle_forward_proxy( canonicalize_forward_host_header(&buf[..used], &canonical_authority)?; // 2. Evaluate OPA policy (same identity binding as CONNECT) - let peer_addr = client.peer_addr().into_diagnostic()?; - let _local_addr = client.local_addr().into_diagnostic()?; + let workload_addr = client.peer_addr().into_diagnostic()?; + let proxy_addr = client.local_addr().into_diagnostic()?; + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); let opa_clone = opa_engine.clone(); let cache_clone = identity_cache.clone(); @@ -3653,7 +3768,7 @@ async fn handle_forward_proxy( let host_clone = host_lc.clone(); let decision = tokio::task::spawn_blocking(move || { evaluate_opa_tcp( - peer_addr, + connection, &opa_clone, &cache_clone, &pid_clone, @@ -3709,7 +3824,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -3820,6 +3935,7 @@ async fn handle_forward_proxy( token_grant_resolver: dynamic_credentials .as_ref() .map(|_| crate::l7::token_grant_injection::default_resolver()), + agent_proposals, }; let mut l7_activity_pending = false; @@ -3970,7 +4086,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4212,7 +4328,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4293,7 +4409,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4350,7 +4466,10 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip( + workload_addr.ip(), + workload_addr.port(), + )) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4402,7 +4521,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4459,7 +4578,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4513,7 +4632,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4575,8 +4694,12 @@ async fn handle_forward_proxy( return Ok(()); } - // 6. Connect upstream - let mut upstream = match TcpStream::connect(addrs.as_slice()).await { + // 6. Connect upstream. Plain-HTTP requests always dial the destination + // directly: only TLS (CONNECT) tunnels chain through the corporate + // proxy, since plain-HTTP forwarding would need absolute-form requests + // rather than a CONNECT tunnel. + let dial_result = TcpStream::connect(addrs.as_slice()).await; + let mut upstream = match dial_result { Ok(s) => s, Err(e) => { let event = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -4588,7 +4711,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4795,7 +4918,7 @@ async fn handle_forward_proxy( OcsfUrl::new("http", &host_lc, &path, port), )) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint(Endpoint::from_ip(peer_addr.ip(), peer_addr.port())) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) .actor_process( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), @@ -4845,10 +4968,26 @@ async fn handle_forward_proxy( Ok(()) } +/// Parse a CONNECT authority into `(host, port)`. +/// +/// IPv6 literals use the RFC 3986 bracketed form (`[::1]:443`); the returned +/// host is bracket-free, matching what DNS resolution, SSRF validation, and +/// `NO_PROXY` matching expect. Host content is otherwise passed through +/// unvalidated — policy and resolution decide what it means. fn parse_target(target: &str) -> Result<(String, u16)> { - let (host, port_str) = target - .split_once(':') - .ok_or_else(|| miette::miette!("CONNECT target missing port: {target}"))?; + let (host, port_str) = if let Some(rest) = target.strip_prefix('[') { + let (host, after) = rest + .split_once(']') + .ok_or_else(|| miette::miette!("CONNECT target has unclosed '[': {target}"))?; + let port_str = after + .strip_prefix(':') + .ok_or_else(|| miette::miette!("CONNECT target missing port: {target}"))?; + (host, port_str) + } else { + target + .split_once(':') + .ok_or_else(|| miette::miette!("CONNECT target missing port: {target}"))? + }; let port: u16 = port_str .parse() .map_err(|_| miette::miette!("Invalid port in CONNECT target: {target}"))?; @@ -4884,6 +5023,34 @@ fn build_json_error_response(status: u16, status_text: &str, error: &str, detail .into_bytes() } +fn build_json_error_response_with_reason( + status: u16, + status_text: &str, + error: &str, + detail: &str, + reason: &str, +) -> Vec { + let mut body = serde_json::json!({ + "error": error, + "detail": detail, + }); + if !reason.is_empty() { + body["reason"] = serde_json::json!(reason); + } + let body_str = body.to_string(); + format!( + "HTTP/1.1 {status} {status_text}\r\n\ + Content-Type: application/json\r\n\ + Content-Length: {}\r\n\ + Connection: close\r\n\ + \r\n\ + {}", + body_str.len(), + body_str, + ) + .into_bytes() +} + fn build_middleware_deny_response( policy_name: &str, denial: &openshell_supervisor_middleware::MiddlewareDenial, @@ -5006,6 +5173,7 @@ fn is_benign_relay_error(err: &miette::Report) -> bool { )] mod tests { use super::*; + use openshell_core::proposals::AgentProposals; use std::future::Future; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::Arc; @@ -5042,6 +5210,8 @@ network_policies: {} None, None, None, + AgentProposals::default(), + Arc::new(None), Arc::new(None), None, None, @@ -5133,6 +5303,48 @@ network_policies: {} assert!(body.get("agent_guidance").is_none()); } + #[test] + fn policy_deny_response_includes_reason() { + let response = build_json_error_response_with_reason( + 403, + "Forbidden", + "policy_denied", + "CONNECT api.example.com:443 not permitted by policy", + "binary '/usr/bin/node' not allowed in policy 'allow_api' (ancestors: [/usr/local/bin/claude])", + ); + let response = String::from_utf8(response).expect("UTF-8 error response"); + assert!(response.starts_with("HTTP/1.1 403 Forbidden")); + let (_, body) = response.split_once("\r\n\r\n").expect("HTTP response"); + let body: serde_json::Value = serde_json::from_str(body).expect("JSON response"); + + assert_eq!(body["error"], "policy_denied"); + assert_eq!( + body["detail"], + "CONNECT api.example.com:443 not permitted by policy" + ); + assert_eq!( + body["reason"], + "binary '/usr/bin/node' not allowed in policy 'allow_api' (ancestors: [/usr/local/bin/claude])" + ); + } + + #[test] + fn policy_deny_response_omits_empty_reason() { + let response = build_json_error_response_with_reason( + 403, + "Forbidden", + "policy_denied", + "CONNECT api.example.com:443 not permitted by policy", + "", + ); + let response = String::from_utf8(response).expect("UTF-8 error response"); + let (_, body) = response.split_once("\r\n\r\n").expect("HTTP response"); + let body: serde_json::Value = serde_json::from_str(body).expect("JSON response"); + + assert_eq!(body["error"], "policy_denied"); + assert!(body.get("reason").is_none()); + } + #[test] fn endpoint_only_opa_allows_declared_endpoint_without_process_identity() { let policy = include_str!("../data/sandbox-policy.rego"); @@ -5508,9 +5720,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let runner = openshell_supervisor_middleware::ChainRunner::new( openshell_supervisor_middleware_builtins::services() @@ -5737,9 +5947,9 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, dynamic_credentials: Some(fixture.dynamic_credentials()), token_grant_resolver: Some(fixture.resolver()), + ..Default::default() }; (ctx, fixture) @@ -5795,9 +6005,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; (config, tunnel_engine, ctx) } @@ -5970,9 +6178,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: resolver, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let query_params = std::collections::HashMap::new(); @@ -6013,9 +6219,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let query_params = std::collections::HashMap::new(); let config = websocket_l7_config(crate::l7::L7Protocol::Rest, false); @@ -8233,11 +8437,30 @@ network_policies: } #[test] - fn test_parse_target_ipv6_bracket_notation_fails() { - assert!( - parse_target("[::1]:443").is_err(), - "split_once splits at first colon inside brackets — port parse fails" - ); + fn test_parse_target_ipv6_bracket_notation() { + let (host, port) = parse_target("[::1]:443").unwrap(); + assert_eq!(host, "::1", "brackets are stripped from the parsed host"); + assert_eq!(port, 443); + + let (host, port) = parse_target("[2001:db8::1]:8443").unwrap(); + assert_eq!(host, "2001:db8::1"); + assert_eq!(port, 8443); + } + + #[test] + fn test_parse_target_rejects_malformed_ipv6_brackets() { + for target in [ + // Unclosed bracket. + "[::1:443", + // No port after the bracket. + "[::1]", + "[::1]443", + // Empty or non-numeric port. + "[::1]:", + "[::1]:notaport", + ] { + assert!(parse_target(target).is_err(), "{target} should be rejected"); + } } // -- parse_proxy_uri: hostname parser regression tests -- @@ -8421,9 +8644,7 @@ network_policies: ancestors: vec![], cmdline_paths: vec![], secret_resolver: None, - activity_tx: None, - dynamic_credentials: None, - token_grant_resolver: None, + ..Default::default() }; let runner = openshell_supervisor_middleware::ChainRunner::new( openshell_supervisor_middleware_builtins::services() @@ -9325,14 +9546,16 @@ network_policies: engine, cache, entrypoint_pid, - None, // tls_state — ephemeral CA unavailable - None, // inference_ctx - None, // policy_local_ctx - Arc::new(None), // trusted_host_gateway - None, // secret_resolver - None, // dynamic_credentials - Some(denial_tx), // denial_tx — positive allow/deny signal - None, // activity_tx + None, // tls_state — ephemeral CA unavailable + None, // inference_ctx + None, // policy_local_ctx + AgentProposals::default(), // agent_proposals + Arc::new(None), // trusted_host_gateway + Arc::new(None), // upstream_proxy + None, // secret_resolver + None, // dynamic_credentials + Some(denial_tx), // denial_tx — positive allow/deny signal + None, // activity_tx )), ) .await @@ -9555,9 +9778,9 @@ network_policies: /// 4. Spawn the temp bash as a child with a `/dev/tcp` one-liner that /// opens a real TCP connection to the listener and holds it open /// inside the bash process. - /// 5. Accept the connection on the listener side and capture the peer's - /// ephemeral port — that's what `resolve_process_identity` uses to - /// walk `/proc/net/tcp` back to the child PID. + /// 5. Accept the connection on the listener side and capture both socket + /// endpoints — that's what `resolve_process_identity` uses to walk + /// `/proc/net/tcp` back to the child PID. /// 6. Overwrite the temp bash on disk with different bytes to simulate /// a `docker cp` hot-swap. The running child is unaffected (it still /// executes from its in-memory image), but `/proc//exe` will @@ -9585,7 +9808,7 @@ network_policies: // 1. Start a listener on loopback. let listener = TcpListener::bind("127.0.0.1:0").expect("bind"); - let listener_port = listener.local_addr().unwrap().port(); + let proxy_addr = listener.local_addr().unwrap(); // 2. Copy /bin/bash to a temp path. let tmp = tempfile::TempDir::new().unwrap(); @@ -9605,8 +9828,10 @@ network_policies: // to keep it open. Do not use an external command like `sleep`: // it inherits the socket fd and intentionally trips the shared // socket ambiguity guard instead of exercising the hot-swap path. - let script = - format!("exec 3<>/dev/tcp/127.0.0.1/{listener_port}; read -r -t 30 _ <&3 || true"); + let script = format!( + "exec 3<>/dev/tcp/127.0.0.1/{}; read -r -t 30 _ <&3 || true", + proxy_addr.port() + ); let mut child = Command::new(&bash_v1) .arg("-c") .arg(&script) @@ -9616,9 +9841,9 @@ network_policies: .spawn() .expect("spawn hotswap-bash child"); - // 5. Accept on the listener side, capture the peer port. + // 5. Accept on the listener side and capture the peer endpoint. listener.set_nonblocking(false).expect("blocking listener"); - let (mut stream, peer_addr) = match listener.accept() { + let (mut stream, workload_addr) = match listener.accept() { Ok(pair) => pair, Err(e) => { let _ = child.kill(); @@ -9626,7 +9851,7 @@ network_policies: panic!("failed to accept child connection: {e}"); } }; - let peer_port = peer_addr.port(); + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); // Drain any spurious data; we just need the socket open. stream .set_read_timeout(Some(Duration::from_millis(50))) @@ -9653,7 +9878,7 @@ network_policies: // contract: hash the live executable via /proc//exe while // returning a clean display path for policy/logging. let test_pid = std::process::id(); - let result = resolve_process_identity(test_pid, peer_port, &cache); + let result = resolve_process_identity(test_pid, connection, &cache); let child_pid = child.id(); // Always clean up the child before asserting so a failure doesn't @@ -9728,9 +9953,10 @@ network_policies: } let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); - let listener_port = listener.local_addr().unwrap().port(); - let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); - let peer_port = stream.local_addr().unwrap().port(); + let proxy_addr = listener.local_addr().unwrap(); + let stream = TcpStream::connect(proxy_addr).expect("connect"); + let workload_addr = stream.local_addr().unwrap(); + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); let (_accepted, _) = listener.accept().expect("accept"); let fd = stream.as_raw_fd(); @@ -9786,7 +10012,7 @@ network_policies: let cache = BinaryIdentityCache::new(); - let mut result = resolve_process_identity(entrypoint_pid, peer_port, &cache); + let mut result = resolve_process_identity(entrypoint_pid, connection, &cache); for _ in 0..10 { match &result { Err(err) @@ -9795,7 +10021,7 @@ network_policies: { // /proc//fd scan transiently failed; give procfs time to settle. std::thread::sleep(Duration::from_millis(50)); - result = resolve_process_identity(entrypoint_pid, peer_port, &cache); + result = resolve_process_identity(entrypoint_pid, connection, &cache); } Ok(_) => { // On arm64 under heavy CI load the /proc fd scan can transiently @@ -9803,7 +10029,7 @@ network_policies: // the child as owner and yielding a spurious Ok. Retry to give // both owners time to appear consistently in /proc//fd. std::thread::sleep(Duration::from_millis(50)); - result = resolve_process_identity(entrypoint_pid, peer_port, &cache); + result = resolve_process_identity(entrypoint_pid, connection, &cache); } _ => break, } diff --git a/crates/openshell-supervisor-network/src/run.rs b/crates/openshell-supervisor-network/src/run.rs index 9284a55ca1..5047ad7bdb 100644 --- a/crates/openshell-supervisor-network/src/run.rs +++ b/crates/openshell-supervisor-network/src/run.rs @@ -26,6 +26,7 @@ use openshell_ocsf::{ use openshell_core::activity::ActivitySender; use openshell_core::denial::DenialEvent; +use openshell_core::proposals::AgentProposals; use tokio::sync::mpsc::UnboundedSender; use crate::identity::BinaryIdentityCache; @@ -86,7 +87,9 @@ pub async fn run_networking( #[allow(unused_variables)] inference_routes: Option<&str>, denial_tx: Option>, activity_tx: Option, + agent_proposals: AgentProposals, workspace_rx: tokio::sync::watch::Receiver, + upstream_proxy_args: &crate::upstream_proxy::UpstreamProxyArgs, ) -> Result { // Build the policy-local route context. The orchestrator's policy poll // loop also holds an `Arc` clone (via `Networking::policy_local_ctx`) so @@ -97,6 +100,7 @@ pub async fn run_networking( sandbox_name .map(str::to_string) .or_else(|| sandbox_id.map(str::to_string)), + agent_proposals, workspace_rx, )); @@ -311,6 +315,7 @@ pub async fn run_networking( denial_tx, activity_tx, engine_ready_rx, + upstream_proxy_args, ) .await?; Some(proxy_handle) diff --git a/crates/openshell-supervisor-network/src/upstream_proxy.rs b/crates/openshell-supervisor-network/src/upstream_proxy.rs new file mode 100644 index 0000000000..85e57c14c5 --- /dev/null +++ b/crates/openshell-supervisor-network/src/upstream_proxy.rs @@ -0,0 +1,1877 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Upstream corporate proxy chaining for the sandbox egress proxy. +//! +//! In proxy-required enterprise networks (issue #1792) the supervisor cannot +//! dial policy-approved destinations directly: all outbound traffic must go +//! through a corporate forward proxy. The operator-owned proxy configuration +//! reaches the supervisor as command-line arguments ([`UpstreamProxyArgs`]) +//! that the compute driver sets when it launches the supervisor — never +//! through the environment, which sandbox images could bake `ENV` values +//! into — and this module chains approved TLS tunnels through the corporate +//! proxy with HTTP CONNECT. +//! +//! Only TLS (CONNECT) egress is chained: plain-HTTP requests always dial the +//! destination directly. Forwarding plain HTTP through a corporate proxy +//! requires absolute-form request forwarding rather than CONNECT tunneling +//! and is out of scope for this feature. +//! +//! The conventional `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` / `NO_PROXY` +//! variables are intentionally ignored: those are controlled by the sandbox +//! creator and are rewritten separately to point the workload child at the +//! local policy proxy, so honoring them would let a sandbox pick an arbitrary +//! upstream proxy or disable proxying with `NO_PROXY=*`. +//! +//! Scope and invariants: +//! - Only `http://` proxy URLs are supported. Configuration is fail-closed: +//! any present-but-invalid driver-supplied setting — an empty argument +//! value, an unsupported (`https://`, SOCKS) or malformed proxy URL, an +//! unreadable auth file, a malformed credential, or an auth file without the explicit +//! cleartext-credential acknowledgement — is a fatal startup error rather +//! than being silently ignored, so a typo can never quietly downgrade the +//! operator's egress boundary to direct dialing or unauthenticated proxy +//! access. Validation semantics are shared with the compute driver via +//! [`openshell_core::driver_utils::parse_upstream_proxy_url`] and +//! [`openshell_core::driver_utils::parse_upstream_proxy_credential`]. +//! - Policy evaluation, DNS resolution, and SSRF checks run exactly as in the +//! direct-dial path; the corporate proxy only replaces the final TCP dial. +//! - CONNECT requests target a validated resolved address by default, so the +//! proxy performs no DNS resolution and the tunnel stays bound to the +//! answer that passed SSRF/`allowed_ips` validation. The +//! `--upstream-proxy-connect-by-hostname` opt-in sends the hostname +//! instead, for proxies whose ACLs filter on hostnames. +//! - The operator `NO_PROXY` list decides which destinations bypass the +//! corporate proxy and keep dialing directly (cluster-internal services, +//! host gateway, etc.). Loopback destinations always bypass the proxy. + +use std::io::{Error as IoError, ErrorKind}; +use std::net::{IpAddr, SocketAddr}; +use std::pin::Pin; +use std::task::{Context, Poll}; +use std::time::Duration; + +use base64::Engine as _; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, ReadBuf}; +use tokio::net::TcpStream; +use tracing::debug; + +/// Upper bound on the corporate proxy's CONNECT response header block. +const MAX_CONNECT_RESPONSE_BYTES: usize = 8 * 1024; + +/// End-to-end budget for dialing the corporate proxy and completing the +/// CONNECT handshake. +const CONNECT_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(30); + +/// A parsed corporate proxy endpoint. +#[derive(Clone)] +pub struct ProxyEndpoint { + host: String, + port: u16, + /// Pre-computed `Basic ` header value from the proxy auth file. + /// Never logged. + proxy_authorization: Option, +} + +impl ProxyEndpoint { + /// `host:port` label for logs. Excludes credentials. + #[must_use] + pub fn display_addr(&self) -> String { + format!("{}:{}", self.host, self.port) + } +} + +impl std::fmt::Debug for ProxyEndpoint { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ProxyEndpoint") + .field("host", &self.host) + .field("port", &self.port) + .field("proxy_authorization", &self.proxy_authorization.is_some()) + .finish() + } +} + +/// The pattern half of one parsed `NO_PROXY` entry. +#[derive(Debug, Clone)] +enum NoProxyPattern { + /// `*` — bypass the proxy for every destination. + Wildcard, + /// Domain suffix match: `corp.com` matches `corp.com` and `x.corp.com`. + Domain(String), + /// Exact IP match, against an IP-literal host or its resolved addresses. + Ip(IpAddr), + /// CIDR match, against an IP-literal host or its resolved addresses. + Cidr(ipnet::IpNet), +} + +/// One parsed `NO_PROXY` entry. +#[derive(Debug, Clone)] +struct NoProxyEntry { + pattern: NoProxyPattern, + /// When set (`internal.corp:8443`), the entry only applies to this + /// destination port instead of every port. + port: Option, +} + +/// Parsed `NO_PROXY` list. +#[derive(Debug, Clone, Default)] +struct NoProxy { + entries: Vec, +} + +impl NoProxy { + fn parse(raw: &str) -> Self { + let mut entries = Vec::new(); + for item in raw.split(',') { + let item = item.trim(); + if item.is_empty() { + continue; + } + if item == "*" { + entries.push(NoProxyEntry { + pattern: NoProxyPattern::Wildcard, + port: None, + }); + continue; + } + // Whole-item IP/CIDR forms first: a bare IPv6 literal contains + // colons that must never be misread as a port qualifier. + if let Ok(net) = item.parse::() { + entries.push(NoProxyEntry { + pattern: NoProxyPattern::Cidr(net), + port: None, + }); + continue; + } + if let Ok(ip) = item.trim_matches(['[', ']']).parse::() { + entries.push(NoProxyEntry { + pattern: NoProxyPattern::Ip(ip), + port: None, + }); + continue; + } + // Optional `:port` qualifier: only a valid trailing u16 counts; + // anything else stays part of the pattern. + let (head, port) = item + .rsplit_once(':') + .map_or((item, None), |(head, port_str)| { + port_str + .parse::() + .map_or((item, None), |port| (head, Some(port))) + }); + let pattern = if let Ok(net) = head.parse::() { + NoProxyPattern::Cidr(net) + } else if let Ok(ip) = head.trim_matches(['[', ']']).parse::() { + NoProxyPattern::Ip(ip) + } else { + // Domain entry. Strip any leading `*.` or `.` so + // `.corp.com`, `*.corp.com`, and `corp.com` all behave + // identically. + let name = head + .strip_prefix("*.") + .or_else(|| head.strip_prefix('.')) + .unwrap_or(head) + .to_ascii_lowercase(); + if name.is_empty() { + continue; + } + NoProxyPattern::Domain(name) + }; + entries.push(NoProxyEntry { pattern, port }); + } + Self { entries } + } + + /// The validated addresses that may be dialed directly for + /// `(host, port)`, or `None` when no entry matches and the corporate + /// proxy must be used. + /// + /// Hostname-level matches — `*`, a domain entry, or an IP/CIDR entry + /// matching an IP-literal host — authorize every validated address. + /// IP/CIDR entries match hostnames through their *resolved* addresses and + /// authorize only the addresses they contain, so a bypass scoped to an + /// internal range can never widen into a direct dial of an address + /// outside that range. + /// + /// The automatic loopback bypass is derived from resolved addresses, never + /// from the host string: `resolved` may come from the sandbox's own + /// `/etc/hosts` (see `resolve_socket_addrs`), which the workload controls, + /// so treating the name `localhost` as proof of loopback would let a + /// spoofed mapping dial a policy-allowed public address while escaping the + /// operator proxy. A spoofed `localhost` falls through to the entries below + /// and is proxied unless an explicit `NO_PROXY` entry matches. + fn direct_addrs( + &self, + host: &str, + port: u16, + resolved: &[SocketAddr], + ) -> Option> { + let host_ip = host.trim_matches(['[', ']']).parse::().ok(); + if let Some(ip) = host_ip { + // An IP literal is its own destination, so the host string is + // authoritative here. + if ip.is_loopback() { + return Some(resolved.to_vec()); + } + } else if host == "localhost" + && !resolved.is_empty() + && resolved.iter().all(|addr| addr.ip().is_loopback()) + { + return Some(resolved.to_vec()); + } + let mut subset: Vec = Vec::new(); + for entry in &self.entries { + if entry.port.is_some_and(|entry_port| entry_port != port) { + continue; + } + match &entry.pattern { + NoProxyPattern::Wildcard => return Some(resolved.to_vec()), + NoProxyPattern::Domain(suffix) => { + if host == suffix + || host + .strip_suffix(suffix) + .is_some_and(|prefix| prefix.ends_with('.')) + { + return Some(resolved.to_vec()); + } + } + NoProxyPattern::Ip(ip) => { + if host_ip == Some(*ip) { + return Some(resolved.to_vec()); + } + for addr in resolved { + if addr.ip() == *ip && !subset.contains(addr) { + subset.push(*addr); + } + } + } + NoProxyPattern::Cidr(net) => { + if host_ip.is_some_and(|ip| net.contains(&ip)) { + return Some(resolved.to_vec()); + } + for addr in resolved { + if net.contains(&addr.ip()) && !subset.contains(addr) { + subset.push(*addr); + } + } + } + } + } + if subset.is_empty() { + None + } else { + Some(subset) + } + } +} + +/// How a validated destination must be dialed, per the operator `NO_PROXY` +/// contract. Produced by [`UpstreamProxyConfig::decision`]. +#[derive(Debug)] +pub enum ProxyDecision<'a> { + /// Chain through the corporate proxy with HTTP CONNECT. + Proxy(&'a ProxyEndpoint), + /// Dial directly, restricted to this subset of the validated addresses: + /// all of them for hostname-level matches (loopback, `*`, domain + /// entries, IP-literal hosts), only the addresses contained in the + /// matching IP/CIDR entries otherwise. + Direct(Vec), +} + +/// What the supervisor puts in the CONNECT request line sent to the +/// corporate proxy. +#[derive(Debug, Clone, Copy)] +pub enum ConnectTarget { + /// CONNECT to this validated address; the proxy performs no DNS + /// resolution, so the tunnel is bound to the answer that already passed + /// SSRF and `allowed_ips` validation. The destination hostname still + /// travels inside the tunnel (TLS SNI, application `Host`). + Ip(IpAddr), + /// CONNECT by hostname; the proxy resolves the name itself. Operator + /// opt-in for hostname-filtering proxy ACLs + /// (`--upstream-proxy-connect-by-hostname`). + Hostname, +} + +/// Corporate proxy configuration built from the driver-supplied +/// command-line arguments ([`UpstreamProxyArgs`]). +#[derive(Debug, Clone)] +pub struct UpstreamProxyConfig { + https: ProxyEndpoint, + no_proxy: NoProxy, + connect_by_hostname: bool, +} + +/// Operator-owned corporate proxy settings passed to the supervisor as +/// command-line arguments by the compute driver. +/// +/// Argv is set by the driver when it creates the container/VM and cannot be +/// influenced by sandbox environment or image `ENV`, so a field left `None` +/// / `false` genuinely means "not configured by the operator". +#[derive(Debug, Clone, Default)] +pub struct UpstreamProxyArgs { + /// `http://host:port` corporate proxy URL, or `None` for direct egress. + pub https_proxy: Option, + /// Comma-separated `NO_PROXY` list. + pub no_proxy: Option, + /// Path to the root-only credential mount (`user:pass`). + pub proxy_auth_file: Option, + /// Operator acknowledgement that credentials travel as cleartext Basic + /// auth over the plain-TCP proxy connection. + pub proxy_auth_allow_insecure: bool, + /// Send the destination hostname in CONNECT instead of a validated IP. + pub proxy_connect_by_hostname: bool, +} + +// Supervisor CLI flag names for the corporate-proxy settings, used as the +// dispatch keys in `from_lookup` and in operator-facing error messages. +const ARG_HTTPS_PROXY: &str = "--upstream-proxy"; +const ARG_NO_PROXY: &str = "--upstream-no-proxy"; +const ARG_PROXY_AUTH_FILE: &str = "--upstream-proxy-auth-file"; +const ARG_PROXY_AUTH_ALLOW_INSECURE: &str = "--upstream-proxy-auth-allow-insecure"; +const ARG_PROXY_CONNECT_BY_HOSTNAME: &str = "--upstream-proxy-connect-by-hostname"; + +impl UpstreamProxyConfig { + /// Build the corporate proxy configuration from the driver-supplied + /// command-line [`UpstreamProxyArgs`]. Returns `Ok(None)` when no proxy + /// is configured. + /// + /// The conventional `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` / + /// `NO_PROXY` environment variables are intentionally never consulted: + /// they are controlled by the sandbox creator (and rewritten to point + /// workload children at the local policy proxy), so honoring them would + /// let a sandbox choose an arbitrary upstream proxy or disable proxying + /// entirely. + /// + /// # Errors + /// + /// This is an operator-owned security boundary, so a present-but-invalid + /// value is fatal instead of being treated as unset: an empty (or + /// whitespace-only) argument, an invalid or unsupported proxy URL, an + /// auth file that is set but unreadable or holds a malformed credential, + /// an auth file without the cleartext-credential acknowledgement, or an + /// auth file / `NO_PROXY` list / acknowledgement / connect-by-hostname + /// flag with no proxy configured. Failing closed prevents a + /// misconfiguration from silently degrading to direct dialing or + /// unauthenticated proxy access. + pub fn from_args(args: &UpstreamProxyArgs) -> Result, String> { + // Present the typed argv fields under their canonical identifiers so + // the shared validation runs once. Booleans map to Some("true") / + // None, so every pairing rule (auth file needs the acknowledgement, + // no auxiliary setting without a proxy) applies unchanged. + Self::from_lookup(|name| { + if name == ARG_HTTPS_PROXY { + args.https_proxy.clone() + } else if name == ARG_NO_PROXY { + args.no_proxy.clone() + } else if name == ARG_PROXY_AUTH_FILE { + args.proxy_auth_file.clone() + } else if name == ARG_PROXY_AUTH_ALLOW_INSECURE { + args.proxy_auth_allow_insecure.then(|| "true".to_string()) + } else if name == ARG_PROXY_CONNECT_BY_HOSTNAME { + args.proxy_connect_by_hostname.then(|| "true".to_string()) + } else { + None + } + }) + } + + fn from_lookup(lookup: impl Fn(&str) -> Option) -> Result, String> { + // A missing setting means "not configured". A present-but-empty value + // is a misconfiguration (the driver never emits one), so it is fatal + // rather than silently downgrading the boundary to direct dialing or + // unauthenticated proxy access. + let var = |name: &str| -> Result, String> { + match lookup(name) { + None => Ok(None), + Some(value) if value.trim().is_empty() => { + Err(format!("{name} must not be empty when set")) + } + Some(value) => Ok(Some(value)), + } + }; + let https = var(ARG_HTTPS_PROXY)? + .map(|url| parse_proxy_url(&url, ARG_HTTPS_PROXY)) + .transpose()?; + let auth_file = var(ARG_PROXY_AUTH_FILE)?; + let auth_allow_insecure = var(ARG_PROXY_AUTH_ALLOW_INSECURE)?; + let connect_by_hostname_raw = var(ARG_PROXY_CONNECT_BY_HOSTNAME)?; + let no_proxy_list = var(ARG_NO_PROXY)?; + let Some(mut https) = https else { + // Auxiliary proxy settings without a proxy mean the operator + // believed a proxy boundary was in effect; refuse rather than + // silently running with direct egress. + for (name, value) in [ + (ARG_PROXY_AUTH_FILE, &auth_file), + (ARG_PROXY_AUTH_ALLOW_INSECURE, &auth_allow_insecure), + (ARG_PROXY_CONNECT_BY_HOSTNAME, &connect_by_hostname_raw), + (ARG_NO_PROXY, &no_proxy_list), + ] { + if value.is_some() { + return Err(format!("{name} is set but no upstream proxy is configured")); + } + } + return Ok(None); + }; + + // CONNECT-target mode. The default binds the tunnel to a validated + // address; hostname CONNECT re-opens proxy-side DNS resolution and + // is only honored as the exact opt-in value the driver writes. + let connect_by_hostname = match connect_by_hostname_raw.as_deref().map(str::trim) { + None => false, + Some("true") => true, + Some(_) => { + return Err(format!( + "{ARG_PROXY_CONNECT_BY_HOSTNAME} must be 'true' when set" + )); + } + }; + + // Cleartext-credential acknowledgement. `Proxy-Authorization: Basic` + // travels over plain TCP to the http:// proxy, so credentials are + // only sent when the operator explicitly opted in. The compute driver + // enforces the same pairing at sandbox-create time; enforcing it here + // as well keeps the supervisor fail-closed against a bypassed or + // foreign driver. + let allow_insecure = match auth_allow_insecure.as_deref().map(str::trim) { + None => false, + Some("true") => true, + Some(_) => { + return Err(format!( + "{ARG_PROXY_AUTH_ALLOW_INSECURE} must be 'true' when set" + )); + } + }; + if auth_file.is_none() && auth_allow_insecure.is_some() { + return Err(format!( + "{ARG_PROXY_AUTH_ALLOW_INSECURE} is set but no {ARG_PROXY_AUTH_FILE} is configured" + )); + } + if auth_file.is_some() && !allow_insecure { + return Err(format!( + "{ARG_PROXY_AUTH_FILE} sends the credential as cleartext Basic auth \ + over the plain-TCP proxy connection; refusing without \ + {ARG_PROXY_AUTH_ALLOW_INSECURE}" + )); + } + + // Load proxy credentials from the configured auth file, if any. + // The file is delivered through a root-only secret mount so the + // credentials never appear in the environment or container metadata. + if let Some(path) = auth_file { + let credential = + openshell_core::driver_utils::read_upstream_proxy_credential_file(&path)?; + let header = basic_auth_header(&credential).map_err(|err| { + format!("invalid credential in upstream proxy auth file '{path}': {err}") + })?; + https.proxy_authorization = Some(header); + } + + Ok(Some(Self { + https, + no_proxy: NoProxy::parse(&no_proxy_list.unwrap_or_default()), + connect_by_hostname, + })) + } + + /// Whether CONNECT requests carry the destination hostname instead of a + /// validated IP (operator opt-in for hostname-filtering proxy ACLs). + #[must_use] + pub fn connect_by_hostname(&self) -> bool { + self.connect_by_hostname + } + + /// How to dial the validated destination `(host, port, resolved)`, + /// honoring the operator `NO_PROXY` list. + /// + /// Entries may carry a `:port` qualifier that limits them to that + /// destination port. IP/CIDR entries also match a hostname through its + /// already-validated resolved addresses; such a match authorizes a + /// direct dial of only the addresses inside the entry (see + /// [`ProxyDecision::Direct`]). + #[must_use] + pub fn decision<'a>( + &'a self, + host: &str, + port: u16, + resolved: &[SocketAddr], + ) -> ProxyDecision<'a> { + self.no_proxy + .direct_addrs(host, port, resolved) + .map_or(ProxyDecision::Proxy(&self.https), ProxyDecision::Direct) + } + + /// Credential-free summary for startup logging. + #[must_use] + pub fn summary(&self) -> String { + format!( + "https_proxy={} no_proxy_entries={} connect_target={}", + self.https.display_addr(), + self.no_proxy.entries.len(), + if self.connect_by_hostname { + "hostname" + } else { + "validated-ip" + } + ) + } +} + +/// Parse an `http://host:port` proxy URL with the same validation rules the +/// compute driver applies at sandbox-create time +/// ([`parse_upstream_proxy_url`](openshell_core::driver_utils::parse_upstream_proxy_url)). +/// +/// Credentials are never taken from the URL: they are delivered out of band +/// through the root-only auth-file mount (`--upstream-proxy-auth-file`) so +/// they never appear in config or container metadata. +/// +/// # Errors +/// +/// Rejects unsupported schemes (TLS or SOCKS proxies), inline `user:pass@` +/// credentials, and malformed addresses. The error names `var_name` so the +/// operator can locate the offending setting. +fn parse_proxy_url(raw: &str, var_name: &str) -> Result { + let addr = openshell_core::driver_utils::parse_upstream_proxy_url(raw) + .map_err(|err| format!("{var_name} is invalid: {err}"))?; + Ok(ProxyEndpoint { + host: addr.host, + port: addr.port, + proxy_authorization: None, + }) +} + +/// Build a `Proxy-Authorization: Basic ` header value from a raw +/// `user:pass` credential. +/// +/// The credential is used verbatim after trimming: it is delivered through a +/// trusted operator file, not a URL, so there is no percent-encoding to +/// decode. Validation is shared with the compute driver through +/// [`parse_upstream_proxy_credential`](openshell_core::driver_utils::parse_upstream_proxy_credential), +/// so a credential the driver staged at sandbox-create time is never rejected +/// here. +/// +/// # Errors +/// +/// Rejects an empty credential, one containing control characters that could +/// inject additional HTTP headers, and one not in `user:pass` form. Error +/// messages never include the credential content. +fn basic_auth_header(credential: &str) -> Result { + let credential = openshell_core::driver_utils::parse_upstream_proxy_credential(credential) + .map_err(|err| err.to_string())?; + Ok(format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode(credential) + )) +} + +/// A `TcpStream` that first replays bytes already read from the socket. +/// +/// The CONNECT handshake reads from the proxy socket in chunks, so the read +/// that completes the response header block may also contain the first +/// tunneled payload bytes (a server-speaks-first destination, or a proxy +/// that coalesces writes). Those bytes belong to the destination byte +/// stream and must reach the caller rather than being discarded; this +/// wrapper yields them before reading from the socket again. Writes pass +/// straight through. +#[derive(Debug)] +pub struct PrefixedStream { + inner: TcpStream, + /// Bytes read past the CONNECT header terminator, replayed first. + prefix: Vec, + /// Read offset into `prefix`. + pos: usize, +} + +impl PrefixedStream { + /// Wrap `inner`, replaying `prefix` before socket reads. + #[must_use] + pub fn new(inner: TcpStream, prefix: Vec) -> Self { + Self { + inner, + prefix, + pos: 0, + } + } + + /// Wrap a directly dialed stream with nothing to replay. + #[must_use] + pub fn without_prefix(inner: TcpStream) -> Self { + Self::new(inner, Vec::new()) + } +} + +impl AsyncRead for PrefixedStream { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + let this = self.get_mut(); + if this.pos < this.prefix.len() { + let n = buf.remaining().min(this.prefix.len() - this.pos); + buf.put_slice(&this.prefix[this.pos..this.pos + n]); + this.pos += n; + if this.pos == this.prefix.len() { + // Drained: release the buffer instead of holding it for the + // tunnel's lifetime. + this.prefix = Vec::new(); + this.pos = 0; + } + return Poll::Ready(Ok(())); + } + Pin::new(&mut this.inner).poll_read(cx, buf) + } +} + +impl AsyncWrite for PrefixedStream { + fn poll_write( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + Pin::new(&mut self.get_mut().inner).poll_write(cx, buf) + } + + fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.get_mut().inner).poll_flush(cx) + } + + fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.get_mut().inner).poll_shutdown(cx) + } + + fn poll_write_vectored( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + bufs: &[std::io::IoSlice<'_>], + ) -> Poll> { + Pin::new(&mut self.get_mut().inner).poll_write_vectored(cx, bufs) + } + + fn is_write_vectored(&self) -> bool { + self.inner.is_write_vectored() + } +} + +/// Open a tunnel to the destination through the corporate proxy with HTTP +/// CONNECT. +/// +/// Returns the connected stream once the proxy answers 200; after that the +/// stream is a transparent byte pipe to the destination. Any tunneled bytes +/// received in the same read as the CONNECT response are preserved and +/// replayed by the returned [`PrefixedStream`]. +/// +/// `target` selects the CONNECT request target. The default mode sends a +/// validated IP ([`ConnectTarget::Ip`]) so the proxy performs no DNS +/// resolution and the tunnel is bound to an address that already passed +/// SSRF and `allowed_ips` validation; `host` then only labels logs and is +/// used by the caller for TLS SNI inside the tunnel. With the operator +/// opt-in ([`ConnectTarget::Hostname`]) the hostname is sent instead so +/// hostname-filtering proxy ACLs keep working, at the cost of proxy-side +/// resolution. Local DNS resolution and SSRF validation must already have +/// happened at the call site in both modes. +/// +/// # Errors +/// +/// Returns an error when the proxy is unreachable, the handshake times out, +/// or the proxy answers with a non-200 status. +pub async fn connect_via( + endpoint: &ProxyEndpoint, + host: &str, + port: u16, + target: ConnectTarget, +) -> std::io::Result { + tokio::time::timeout( + CONNECT_HANDSHAKE_TIMEOUT, + connect_via_inner(endpoint, host, port, target), + ) + .await + .map_err(|_| { + IoError::new( + ErrorKind::TimedOut, + format!( + "upstream proxy {} CONNECT handshake timed out", + endpoint.display_addr() + ), + ) + })? +} + +/// Open a validated-IP CONNECT tunnel, trying each validated address in +/// order until one succeeds. +/// +/// The direct-dial path hands `TcpStream::connect` the whole validated list +/// and it falls back across addresses; this is the proxied equivalent, so a +/// dual-stack destination whose first validated address is unreachable +/// through the corporate proxy still connects via a later one. All attempts +/// share one aggregate handshake budget ([`CONNECT_HANDSHAKE_TIMEOUT`]), +/// matching the single-attempt paths; within it, each attempt is capped at +/// its fair share of the remaining time (`remaining / attempts_left`) so a +/// proxy that accepts a CONNECT but never answers cannot starve the +/// remaining validated addresses. Time an attempt fails without using +/// rolls over to later attempts. +/// +/// # Errors +/// +/// Returns an error when `addrs` is empty, or when every attempt fails or +/// times out (the last attempt's error, annotated with the attempt count +/// when there were several). +pub async fn connect_via_validated( + endpoint: &ProxyEndpoint, + host: &str, + port: u16, + addrs: &[SocketAddr], +) -> std::io::Result { + connect_via_validated_with_budget(endpoint, host, port, addrs, CONNECT_HANDSHAKE_TIMEOUT).await +} + +/// [`connect_via_validated`] with an explicit aggregate budget, split out so +/// tests can exercise the hang-fallback behavior without real 30s waits. +async fn connect_via_validated_with_budget( + endpoint: &ProxyEndpoint, + host: &str, + port: u16, + addrs: &[SocketAddr], + budget: Duration, +) -> std::io::Result { + if addrs.is_empty() { + return Err(IoError::new( + ErrorKind::InvalidInput, + format!("no validated addresses to CONNECT to for {host}"), + )); + } + let deadline = tokio::time::Instant::now() + budget; + let mut last_err = None; + let mut attempted = 0usize; + for (index, addr) in addrs.iter().enumerate() { + let remaining = deadline.saturating_duration_since(tokio::time::Instant::now()); + if remaining.is_zero() { + break; + } + let attempts_left = u32::try_from(addrs.len() - index).unwrap_or(u32::MAX); + let attempt_budget = remaining / attempts_left; + attempted += 1; + match tokio::time::timeout( + attempt_budget, + connect_via_inner(endpoint, host, port, ConnectTarget::Ip(addr.ip())), + ) + .await + { + Ok(Ok(stream)) => return Ok(stream), + Ok(Err(err)) => last_err = Some(err), + Err(_) => { + last_err = Some(IoError::new( + ErrorKind::TimedOut, + format!( + "upstream proxy {} CONNECT to {} timed out", + endpoint.display_addr(), + addr.ip(), + ), + )); + } + } + } + // The first iteration always runs (`remaining` starts at the full + // budget), so at least one attempt recorded an error. + let last = last_err.expect("at least one CONNECT attempt"); + if addrs.len() == 1 { + return Err(last); + } + let scope = if attempted == addrs.len() { + format!("all {} validated addresses", addrs.len()) + } else { + format!( + "{attempted} of {} validated addresses (handshake budget exhausted)", + addrs.len() + ) + }; + Err(IoError::new( + last.kind(), + format!("CONNECT failed for {scope} of {host}; last: {last}"), + )) +} + +async fn connect_via_inner( + endpoint: &ProxyEndpoint, + host: &str, + port: u16, + target: ConnectTarget, +) -> std::io::Result { + let mut stream = TcpStream::connect((endpoint.host.as_str(), endpoint.port)).await?; + + let target = match target { + ConnectTarget::Ip(IpAddr::V6(ip)) => format!("[{ip}]:{port}"), + ConnectTarget::Ip(ip) => format!("{ip}:{port}"), + ConnectTarget::Hostname if host.contains(':') => format!("[{host}]:{port}"), + ConnectTarget::Hostname => format!("{host}:{port}"), + }; + let mut request = format!("CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n"); + if let Some(auth) = &endpoint.proxy_authorization { + request.push_str("Proxy-Authorization: "); + request.push_str(auth); + request.push_str("\r\n"); + } + request.push_str("\r\n"); + stream.write_all(request.as_bytes()).await?; + + // Read the proxy's response header block. A read may run past the + // `\r\n\r\n` terminator into tunneled payload; those bytes are preserved + // below, never discarded. + let mut buf = vec![0u8; MAX_CONNECT_RESPONSE_BYTES]; + let mut used = 0; + let header_end = loop { + if used == buf.len() { + return Err(IoError::other(format!( + "upstream proxy {} CONNECT response headers exceed {MAX_CONNECT_RESPONSE_BYTES} bytes", + endpoint.display_addr() + ))); + } + let n = stream.read(&mut buf[used..]).await?; + if n == 0 { + return Err(IoError::new( + ErrorKind::UnexpectedEof, + format!( + "upstream proxy {} closed the connection during CONNECT", + endpoint.display_addr() + ), + )); + } + used += n; + if let Some(pos) = buf[..used].windows(4).position(|win| win == b"\r\n\r\n") { + break pos + 4; + } + }; + + let response = String::from_utf8_lossy(&buf[..header_end]); + let status_line = response.lines().next().unwrap_or_default(); + let status_code = status_line + .split_whitespace() + .nth(1) + .and_then(|code| code.parse::().ok()); + match status_code { + Some(200) => { + debug!( + proxy = %endpoint.display_addr(), + target = %target, + "upstream proxy CONNECT tunnel established" + ); + buf.truncate(used); + let overflow = buf.split_off(header_end); + Ok(PrefixedStream::new(stream, overflow)) + } + Some(code) => Err(IoError::other(format!( + "upstream proxy {} refused CONNECT to {target}: HTTP {code}", + endpoint.display_addr() + ))), + None => Err(IoError::new( + ErrorKind::InvalidData, + format!( + "upstream proxy {} sent a malformed CONNECT response", + endpoint.display_addr() + ), + )), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use super::{ + ARG_HTTPS_PROXY as HTTPS_PROXY, ARG_NO_PROXY as NO_PROXY, + ARG_PROXY_AUTH_ALLOW_INSECURE as PROXY_AUTH_ALLOW_INSECURE, + ARG_PROXY_AUTH_FILE as PROXY_AUTH_FILE, + ARG_PROXY_CONNECT_BY_HOSTNAME as PROXY_CONNECT_BY_HOSTNAME, + }; + + fn config_from(pairs: &[(&str, &str)]) -> Result, String> { + UpstreamProxyConfig::from_lookup(|name| { + pairs + .iter() + .find(|(k, _)| *k == name) + .map(|(_, v)| (*v).to_string()) + }) + } + + /// Shorthand for tests exercising a configuration that must load. + fn config_ok(pairs: &[(&str, &str)]) -> UpstreamProxyConfig { + config_from(pairs).unwrap().unwrap() + } + + #[test] + fn no_env_yields_none() { + assert!(config_from(&[]).unwrap().is_none()); + } + + #[test] + fn from_args_maps_cli_arguments_to_config() { + // The driver-supplied argv is the authoritative source; the shared + // validation applies to it exactly as in the flag-keyed lookup. + let args = UpstreamProxyArgs { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + no_proxy: Some("*.svc.cluster.local".to_string()), + proxy_connect_by_hostname: true, + ..UpstreamProxyArgs::default() + }; + let cfg = UpstreamProxyConfig::from_args(&args).unwrap().unwrap(); + assert!(cfg.connect_by_hostname()); + assert!(bypasses(&cfg, "kubernetes.default.svc.cluster.local")); + + // No proxy URL means no configuration. + assert!( + UpstreamProxyConfig::from_args(&UpstreamProxyArgs::default()) + .unwrap() + .is_none() + ); + } + + #[test] + fn from_args_enforces_the_auth_file_acknowledgement_pairing() { + // An auth file without the acknowledgement is fatal, exactly as in + // the flag-keyed lookup the driver validation mirrors. + let args = UpstreamProxyArgs { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + proxy_auth_file: Some("/etc/openshell/auth/upstream-proxy".to_string()), + ..UpstreamProxyArgs::default() + }; + let err = UpstreamProxyConfig::from_args(&args).unwrap_err(); + assert!(err.contains(PROXY_AUTH_ALLOW_INSECURE), "{err}"); + + // Auxiliary settings without a proxy are fatal. + let args = UpstreamProxyArgs { + proxy_connect_by_hostname: true, + ..UpstreamProxyArgs::default() + }; + let err = UpstreamProxyConfig::from_args(&args).unwrap_err(); + assert!(err.contains("no upstream proxy"), "{err}"); + } + + #[test] + fn conventional_proxy_vars_are_ignored() { + // The sandbox creator controls these names; they must not steer the + // supervisor's operator-owned egress boundary. + assert!( + config_from(&[ + ("HTTPS_PROXY", "http://attacker:9999"), + ("HTTP_PROXY", "http://attacker:9999"), + ("ALL_PROXY", "http://attacker:9999"), + ("https_proxy", "http://attacker:9999"), + ]) + .unwrap() + .is_none() + ); + } + + #[test] + fn present_but_empty_values_are_fatal() { + // A setting the operator did not configure is absent, never empty: + // the driver only passes configured values. A present-but-blank + // value is therefore a misconfiguration and must not silently mean + // "no proxy". + for (name, value) in [ + (HTTPS_PROXY, ""), + (HTTPS_PROXY, " "), + (NO_PROXY, " "), + (PROXY_AUTH_FILE, ""), + (PROXY_AUTH_ALLOW_INSECURE, ""), + ] { + let err = config_from(&[(name, value)]).unwrap_err(); + assert!(err.contains(name), "{err}"); + assert!(err.contains("empty"), "{err}"); + } + } + + /// Shorthand: the proxy endpoint chosen for `host:443` with no resolved + /// addresses in play, or `None` when the destination dials directly. + fn proxy_endpoint<'a>(cfg: &'a UpstreamProxyConfig, host: &str) -> Option<&'a ProxyEndpoint> { + match cfg.decision(host, 443, &[]) { + ProxyDecision::Proxy(ep) => Some(ep), + ProxyDecision::Direct(_) => None, + } + } + + #[test] + fn https_proxy_parsed_with_port() { + let cfg = config_ok(&[(HTTPS_PROXY, "http://proxy.corp.com:8080")]); + let ep = proxy_endpoint(&cfg, "api.stripe.com").unwrap(); + assert_eq!(ep.display_addr(), "proxy.corp.com:8080"); + assert!(ep.proxy_authorization.is_none()); + } + + #[test] + fn scheme_less_or_port_less_proxy_url_is_fatal() { + // The accepted grammar is exactly `http://host:port`; lenient + // defaulting would let a typo silently target the wrong proxy. + for url in [ + "proxy.corp.com", + "proxy.corp.com:3128", + "http://proxy.corp.com", + ] { + let err = config_from(&[(HTTPS_PROXY, url)]).unwrap_err(); + assert!(err.contains(HTTPS_PROXY), "{url}: {err}"); + assert!(err.contains("explicit"), "{url}: {err}"); + } + } + + // -- Fail-closed configuration validation -- + // + // Present-but-invalid settings must be fatal, never silently treated + // as unset: a typo must not downgrade the operator's egress boundary + // to direct dialing or unauthenticated proxy access. + + #[test] + fn tls_and_socks_proxies_are_fatal() { + for url in ["https://proxy:443", "socks5://proxy:1080"] { + let err = config_from(&[(HTTPS_PROXY, url)]).unwrap_err(); + assert!(err.contains(HTTPS_PROXY), "{err}"); + assert!(err.contains("scheme"), "{err}"); + } + } + + #[test] + fn url_userinfo_is_fatal_not_used_as_credentials() { + // Inline credentials in the URL must never become the proxy auth; + // credentials come only from the auth file. Matching the compute + // driver, a URL that embeds them is rejected outright. + let err = config_from(&[(HTTPS_PROXY, "http://user:secret@proxy:8080")]).unwrap_err(); + assert!(err.contains(HTTPS_PROXY), "{err}"); + assert!( + !err.contains("secret"), + "error must not leak the credential: {err}" + ); + } + + #[test] + fn malformed_proxy_address_is_fatal() { + let err = config_from(&[(HTTPS_PROXY, "http://proxy:notaport")]).unwrap_err(); + assert!(err.contains(HTTPS_PROXY), "{err}"); + // A path/query/fragment addresses an endpoint, not a proxy; it is + // rejected rather than silently discarded. + for url in [ + "http://proxy:8080/path", + "http://proxy:8080?x=1", + "http://proxy:8080#frag", + ] { + let err = config_from(&[(HTTPS_PROXY, url)]).unwrap_err(); + assert!(err.contains(HTTPS_PROXY), "{url}: {err}"); + } + } + + #[test] + fn unreadable_auth_file_is_fatal() { + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_FILE, "/nonexistent/upstream-proxy-auth"), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + ]) + .unwrap_err(); + assert!(err.contains("auth file"), "{err}"); + } + + #[test] + fn oversized_auth_file_is_fatal() { + // The supervisor uses the same bounded reader as the driver, so an + // oversized credential file is rejected rather than read unbounded. + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), vec![b'a'; 8192]).unwrap(); + let path = file.path().to_str().unwrap().to_string(); + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_FILE, &path), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + ]) + .unwrap_err(); + assert!(err.contains("limit"), "{err}"); + } + + #[test] + fn auth_file_without_insecure_acknowledgement_is_fatal() { + // Basic auth over the plain-TCP proxy connection is readable on the + // network path; sending it requires the explicit opt-in. + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_FILE, "/etc/openshell/auth/upstream-proxy"), + ]) + .unwrap_err(); + assert!(err.contains(PROXY_AUTH_ALLOW_INSECURE), "{err}"); + assert!(err.contains("cleartext"), "{err}"); + } + + #[test] + fn invalid_insecure_acknowledgement_value_is_fatal() { + for value in ["false", "yes", "1", "TRUE"] { + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_FILE, "/etc/openshell/auth/upstream-proxy"), + (PROXY_AUTH_ALLOW_INSECURE, value), + ]) + .unwrap_err(); + assert!(err.contains(PROXY_AUTH_ALLOW_INSECURE), "{value}: {err}"); + } + } + + #[test] + fn insecure_acknowledgement_without_auth_file_is_fatal() { + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + ]) + .unwrap_err(); + assert!(err.contains(PROXY_AUTH_ALLOW_INSECURE), "{err}"); + assert!(err.contains(PROXY_AUTH_FILE), "{err}"); + } + + #[test] + fn malformed_auth_file_credential_is_fatal() { + // Empty, header-injecting, and non-`user:pass` credentials are all + // rejected by the parser shared with the compute driver. + for credential in [" ", "user:pa\r\nss", "userpass", ":pass"] { + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), credential).unwrap(); + let path = file.path().to_str().unwrap().to_string(); + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy.corp.com:8080"), + (PROXY_AUTH_FILE, &path), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + ]) + .unwrap_err(); + assert!(err.contains("auth file"), "{err}"); + assert!( + !err.contains("pa\r\nss"), + "error must not leak the credential: {err}" + ); + } + } + + #[test] + fn auxiliary_settings_without_proxy_are_fatal() { + // An auth file or NO_PROXY list only makes sense relative to a proxy + // boundary the operator believed was in effect. + for (name, value) in [ + (PROXY_AUTH_FILE, "/etc/openshell/auth/upstream-proxy"), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + (NO_PROXY, "*.svc.cluster.local"), + ] { + let err = config_from(&[(name, value)]).unwrap_err(); + assert!(err.contains(name), "{err}"); + assert!(err.contains("no upstream proxy"), "{err}"); + } + } + + #[test] + fn auth_file_credentials_are_applied_to_the_endpoint() { + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), "user:secret\n").unwrap(); + let path = file.path().to_str().unwrap().to_string(); + let cfg = config_ok(&[ + (HTTPS_PROXY, "http://proxy:8080"), + (PROXY_AUTH_FILE, &path), + (PROXY_AUTH_ALLOW_INSECURE, "true"), + ]); + let expected = format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode("user:secret") + ); + let ep = proxy_endpoint(&cfg, "example.com").unwrap(); + assert_eq!(ep.proxy_authorization.as_deref(), Some(expected.as_str())); + } + + #[test] + fn basic_auth_header_encodes_and_rejects_malformed_credentials() { + assert_eq!( + basic_auth_header("user:p@ss").as_deref(), + Ok(format!( + "Basic {}", + base64::engine::general_purpose::STANDARD.encode("user:p@ss") + ) + .as_str()) + ); + assert!(basic_auth_header(" ").is_err()); + assert!(basic_auth_header("user:pa\r\nss").is_err()); + assert!(basic_auth_header("user:pa\nInjected: header").is_err()); + assert!(basic_auth_header("no-separator").is_err()); + } + + #[test] + fn debug_output_hides_credentials() { + let mut cfg = config_ok(&[(HTTPS_PROXY, "http://proxy:8080")]); + cfg.https.proxy_authorization = Some(basic_auth_header("user:secret").unwrap()); + let debug = format!("{cfg:?}"); + assert!(!debug.contains("secret")); + assert!(!cfg.summary().contains("secret")); + } + + #[test] + fn ipv6_proxy_address_parses() { + let cfg = config_ok(&[(HTTPS_PROXY, "http://[fd00::1]:8080")]); + let ep = proxy_endpoint(&cfg, "example.com").unwrap(); + assert_eq!(ep.display_addr(), "fd00::1:8080"); + } + + // -- NO_PROXY matching -- + + fn no_proxy_cfg(no_proxy: &str) -> UpstreamProxyConfig { + config_ok(&[(HTTPS_PROXY, "http://proxy:8080"), (NO_PROXY, no_proxy)]) + } + + /// Hostname-level bypass check at an arbitrary port with no resolved + /// addresses in play. + fn bypasses(cfg: &UpstreamProxyConfig, host: &str) -> bool { + bypasses_port(cfg, host, 443) + } + + fn bypasses_port(cfg: &UpstreamProxyConfig, host: &str, port: u16) -> bool { + matches!(cfg.decision(host, port, &[]), ProxyDecision::Direct(_)) + } + + fn bypasses_with(cfg: &UpstreamProxyConfig, host: &str, resolved: &[SocketAddr]) -> bool { + matches!(cfg.decision(host, 443, resolved), ProxyDecision::Direct(_)) + } + + fn sock(ip: &str, port: u16) -> SocketAddr { + SocketAddr::new(ip.parse().unwrap(), port) + } + + #[test] + fn no_proxy_domain_suffix_matches_host_and_subdomains() { + let cfg = no_proxy_cfg("corp.com,other.example"); + assert!(bypasses(&cfg, "corp.com")); + assert!(bypasses(&cfg, "api.corp.com")); + assert!(!bypasses(&cfg, "notcorp.com")); + assert!(!bypasses(&cfg, "corp.com.evil.io")); + } + + #[test] + fn no_proxy_leading_dot_and_wildcard_prefix_are_equivalent() { + for entry in [".svc.cluster.local", "*.svc.cluster.local"] { + let cfg = no_proxy_cfg(entry); + assert!( + bypasses(&cfg, "kubernetes.default.svc.cluster.local"), + "{entry}" + ); + assert!(!bypasses(&cfg, "example.com"), "{entry}"); + } + } + + #[test] + fn no_proxy_cidr_matches_ip_literals() { + let cfg = no_proxy_cfg("10.96.0.0/12"); + assert!(bypasses(&cfg, "10.96.0.1")); + assert!(bypasses(&cfg, "10.100.20.30")); + assert!(!bypasses(&cfg, "10.200.0.9")); + assert!(!bypasses(&cfg, "93.184.216.34")); + } + + #[test] + fn no_proxy_exact_ip_matches() { + let cfg = no_proxy_cfg("192.168.1.5"); + assert!(bypasses(&cfg, "192.168.1.5")); + assert!(!bypasses(&cfg, "192.168.1.6")); + } + + #[test] + fn no_proxy_wildcard_bypasses_everything() { + let cfg = no_proxy_cfg("*"); + assert!(bypasses(&cfg, "example.com")); + } + + #[test] + fn no_proxy_port_qualifier_limits_bypass_to_that_port() { + // `internal.corp:8443` documents a bypass for one port; broadening + // it to every port would bypass the proxy for traffic the operator + // never excluded. + let cfg = no_proxy_cfg("internal.corp:8443"); + assert!(bypasses_port(&cfg, "internal.corp", 8443)); + assert!(bypasses_port(&cfg, "svc.internal.corp", 8443)); + assert!(!bypasses_port(&cfg, "internal.corp", 443)); + assert!(!bypasses_port(&cfg, "svc.internal.corp", 80)); + } + + #[test] + fn no_proxy_port_qualifier_applies_to_ip_and_cidr_entries() { + let cfg = no_proxy_cfg("192.168.1.5:8443,10.96.0.0/12:6443"); + assert!(bypasses_port(&cfg, "192.168.1.5", 8443)); + assert!(!bypasses_port(&cfg, "192.168.1.5", 443)); + assert!(bypasses_port(&cfg, "10.96.0.1", 6443)); + assert!(!bypasses_port(&cfg, "10.96.0.1", 443)); + } + + #[test] + fn no_proxy_invalid_port_qualifier_is_not_stripped() { + // A trailing qualifier that is not a valid port stays part of the + // pattern instead of silently widening the entry to every port. + let cfg = no_proxy_cfg("internal.corp:99999"); + assert!(!bypasses(&cfg, "internal.corp")); + } + + #[test] + fn no_proxy_cidr_matches_resolved_addresses_of_hostnames() { + // The operator's `10.96.0.0/12` bypass covers cluster-internal + // destinations however they are named; a hostname whose validated + // resolution lands in the range must dial directly. + let cfg = no_proxy_cfg("10.96.0.0/12"); + let inside = sock("10.96.0.7", 443); + match cfg.decision("svc.internal", 443, &[inside]) { + ProxyDecision::Direct(addrs) => assert_eq!(addrs, vec![inside]), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + // Resolution outside the range keeps the proxy. + assert!(matches!( + cfg.decision("example.com", 443, &[sock("93.184.216.34", 443)]), + ProxyDecision::Proxy(_) + )); + } + + #[test] + fn no_proxy_resolved_address_match_limits_direct_dial_to_matching_addrs() { + // Split resolution: only the addresses inside the bypassed range may + // be dialed directly; the others are not covered by the operator's + // exclusion and must not ride along. + let cfg = no_proxy_cfg("10.96.0.0/12"); + let inside = sock("10.96.0.7", 443); + let outside = sock("93.184.216.34", 443); + match cfg.decision("svc.internal", 443, &[outside, inside]) { + ProxyDecision::Direct(addrs) => assert_eq!(addrs, vec![inside]), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + } + + #[test] + fn no_proxy_ip_entry_matches_resolved_addresses() { + let cfg = no_proxy_cfg("10.0.0.5"); + let matching = sock("10.0.0.5", 443); + match cfg.decision("db.internal", 443, &[matching, sock("10.0.0.6", 443)]) { + ProxyDecision::Direct(addrs) => assert_eq!(addrs, vec![matching]), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + } + + #[test] + fn no_proxy_bracketed_ipv6_entry_honors_port_qualifier() { + // The colons of an IPv6 literal must not be misread as a port + // qualifier; only the bracketed form can carry one. + let cfg = no_proxy_cfg("[fd00::1]:8443"); + assert!(bypasses_port(&cfg, "fd00::1", 8443)); + assert!(bypasses_port(&cfg, "[fd00::1]", 8443)); + assert!(!bypasses_port(&cfg, "fd00::1", 443)); + assert!(!bypasses_port(&cfg, "fd00::2", 8443)); + + // A bare IPv6 entry keeps every-port semantics. + let cfg = no_proxy_cfg("fd00::1"); + assert!(bypasses_port(&cfg, "fd00::1", 8443)); + assert!(bypasses_port(&cfg, "fd00::1", 443)); + } + + #[test] + fn no_proxy_ipv6_cidr_matches_resolved_addresses_of_hostnames() { + let cfg = no_proxy_cfg("fd00::/8"); + // An IPv6-literal host inside the range bypasses at hostname level. + assert!(bypasses(&cfg, "fd00::7")); + assert!(!bypasses(&cfg, "2001:db8::1")); + + // A hostname resolving into the range dials directly, restricted to + // the resolved addresses the entry contains. + let inside = sock("fd00::42", 443); + let outside = sock("2001:db8::1", 443); + match cfg.decision("svc.internal", 443, &[outside, inside]) { + ProxyDecision::Direct(addrs) => assert_eq!(addrs, vec![inside]), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + } + + #[test] + fn no_proxy_ipv6_cidr_honors_port_qualifier() { + let cfg = no_proxy_cfg("fd00::/8:6443"); + assert!(bypasses_port(&cfg, "fd00::7", 6443)); + assert!(!bypasses_port(&cfg, "fd00::7", 443)); + match cfg.decision("svc.internal", 6443, &[sock("fd00::42", 6443)]) { + ProxyDecision::Direct(addrs) => assert_eq!(addrs, vec![sock("fd00::42", 6443)]), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + assert!(matches!( + cfg.decision("svc.internal", 443, &[sock("fd00::42", 443)]), + ProxyDecision::Proxy(_) + )); + } + + #[test] + fn no_proxy_hostname_match_authorizes_all_resolved_addresses() { + let cfg = no_proxy_cfg("internal.corp"); + let addrs = [sock("10.0.0.5", 443), sock("93.184.216.34", 443)]; + match cfg.decision("internal.corp", 443, &addrs) { + ProxyDecision::Direct(direct) => assert_eq!(direct, addrs.to_vec()), + ProxyDecision::Proxy(ep) => panic!("expected direct dial, got proxy {ep:?}"), + } + } + + #[test] + fn loopback_bypasses_without_no_proxy() { + // No NO_PROXY at all: loopback still bypasses unconditionally. + let cfg = config_ok(&[(HTTPS_PROXY, "http://proxy:8080")]); + assert!(bypasses_with(&cfg, "localhost", &[sock("127.0.0.1", 443)])); + assert!(bypasses(&cfg, "127.0.0.1")); + assert!(bypasses(&cfg, "::1")); + assert!(!bypasses(&cfg, "example.com")); + } + + /// A sandbox controls its own `/etc/hosts`, and `resolve_socket_addrs` + /// consults it before DNS, so `localhost` can resolve to any address the + /// network policy happens to allow. The name must not authorize a direct + /// dial that escapes the operator proxy — only genuinely loopback + /// resolved addresses may. + #[test] + fn spoofed_localhost_resolution_does_not_bypass_proxy() { + let cfg = config_ok(&[(HTTPS_PROXY, "http://proxy:8080")]); + assert!(!bypasses_with( + &cfg, + "localhost", + &[sock("203.0.113.10", 443)] + )); + // A mixed answer is not partially honored: one non-loopback address + // disqualifies the automatic bypass entirely. + assert!(!bypasses_with( + &cfg, + "localhost", + &[sock("127.0.0.1", 443), sock("203.0.113.10", 443)], + )); + // An empty resolution proves nothing about the destination. + assert!(!bypasses_with(&cfg, "localhost", &[])); + // Genuine loopback resolution still bypasses, including IPv6 and + // dual-stack answers. + assert!(bypasses_with(&cfg, "localhost", &[sock("127.0.0.1", 443)])); + assert!(bypasses_with(&cfg, "localhost", &[sock("::1", 443)])); + assert!(bypasses_with( + &cfg, + "localhost", + &[sock("127.0.0.1", 443), sock("::1", 443)], + )); + } + + /// Losing the automatic bypass does not disable an operator's explicit + /// `NO_PROXY` entry: the operator, not the sandbox, declares that one. + #[test] + fn spoofed_localhost_still_honors_explicit_no_proxy_entry() { + let cfg = no_proxy_cfg("localhost"); + assert!(bypasses_with( + &cfg, + "localhost", + &[sock("203.0.113.10", 443)] + )); + } + + // -- CONNECT handshake -- + + async fn fake_proxy(response: &'static str) -> (SocketAddr, tokio::task::JoinHandle) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let handle = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let request = read_request(&mut socket).await; + socket.write_all(response.as_bytes()).await.unwrap(); + request + }); + (addr, handle) + } + + fn endpoint_for(addr: SocketAddr, auth: Option<&str>) -> ProxyEndpoint { + ProxyEndpoint { + host: addr.ip().to_string(), + port: addr.port(), + proxy_authorization: auth.map(str::to_string), + } + } + + #[tokio::test] + async fn connect_via_targets_validated_ip_by_default() { + // The default CONNECT target is a validated address, so the proxy + // performs no DNS resolution; the hostname only travels inside the + // tunnel (TLS SNI, application Host). + let (addr, handle) = fake_proxy("HTTP/1.1 200 Connection established\r\n\r\n").await; + let endpoint = endpoint_for(addr, Some("Basic dXNlcjpwYXNz")); + let stream = connect_via( + &endpoint, + "api.example.com", + 443, + ConnectTarget::Ip("93.184.216.34".parse().unwrap()), + ) + .await + .unwrap(); + drop(stream); + let request = handle.await.unwrap(); + assert!(request.starts_with("CONNECT 93.184.216.34:443 HTTP/1.1\r\n")); + assert!(request.contains("Host: 93.184.216.34:443\r\n")); + assert!( + !request.contains("api.example.com"), + "hostname must not leak into the CONNECT request in IP mode: {request}" + ); + assert!(request.contains("Proxy-Authorization: Basic dXNlcjpwYXNz\r\n")); + } + + /// Read one request's header block from a fake-proxy socket. + async fn read_request(socket: &mut TcpStream) -> String { + let mut buf = vec![0u8; 4096]; + let mut used = 0; + loop { + let n = socket.read(&mut buf[used..]).await.unwrap(); + used += n; + if n == 0 || buf[..used].windows(4).any(|w| w == b"\r\n\r\n") { + break; + } + } + String::from_utf8_lossy(&buf[..used]).into_owned() + } + + #[tokio::test] + async fn connect_via_validated_falls_back_across_validated_addresses() { + // The direct path tries every validated address; the proxied path + // must too, or a dual-stack destination whose first address is + // unreachable through the proxy fails needlessly. + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let handle = tokio::spawn(async move { + // First attempt: refuse the tunnel. + let (mut refused, _) = listener.accept().await.unwrap(); + let first = read_request(&mut refused).await; + refused + .write_all(b"HTTP/1.1 502 Bad Gateway\r\n\r\n") + .await + .unwrap(); + drop(refused); + // Second attempt: establish it. + let (mut accepted, _) = listener.accept().await.unwrap(); + let second = read_request(&mut accepted).await; + accepted + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + (first, second, accepted) + }); + + let endpoint = endpoint_for(addr, None); + let addrs = [sock("192.0.2.1", 443), sock("192.0.2.2", 443)]; + let stream = connect_via_validated(&endpoint, "api.example.com", 443, &addrs) + .await + .unwrap(); + let (first, second, _accepted) = handle.await.unwrap(); + drop(stream); + assert!( + first.starts_with("CONNECT 192.0.2.1:443 HTTP/1.1\r\n"), + "{first}" + ); + assert!( + second.starts_with("CONNECT 192.0.2.2:443 HTTP/1.1\r\n"), + "{second}" + ); + } + + #[tokio::test] + async fn connect_via_validated_survives_a_hanging_first_attempt() { + // A proxy that accepts the CONNECT but never answers must not + // consume the whole aggregate budget: the per-address cap moves on + // to the next validated address. + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let handle = tokio::spawn(async move { + // First attempt: read the CONNECT, then hang with the socket + // held open until the client times the attempt out. + let (mut hung, _) = listener.accept().await.unwrap(); + let first = read_request(&mut hung).await; + // Second attempt: establish the tunnel. + let (mut accepted, _) = listener.accept().await.unwrap(); + let second = read_request(&mut accepted).await; + accepted + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + drop(hung); + (first, second, accepted) + }); + + let endpoint = endpoint_for(addr, None); + let addrs = [sock("192.0.2.1", 443), sock("192.0.2.2", 443)]; + // Small real-time budget: the first attempt hangs for its fair + // share (half), then the second succeeds within the remainder. + let stream = connect_via_validated_with_budget( + &endpoint, + "api.example.com", + 443, + &addrs, + Duration::from_secs(2), + ) + .await + .unwrap(); + let (first, second, _accepted) = handle.await.unwrap(); + drop(stream); + assert!( + first.starts_with("CONNECT 192.0.2.1:443 HTTP/1.1\r\n"), + "{first}" + ); + assert!( + second.starts_with("CONNECT 192.0.2.2:443 HTTP/1.1\r\n"), + "{second}" + ); + } + + #[tokio::test] + async fn connect_via_validated_reports_failure_across_all_addresses() { + let (addr, _handle) = fake_proxy("HTTP/1.1 502 Bad Gateway\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + // Only the first attempt gets a response; the second finds the + // listener gone, and the aggregate error must say both were tried. + let addrs = [sock("192.0.2.1", 443), sock("192.0.2.2", 443)]; + let err = connect_via_validated(&endpoint, "api.example.com", 443, &addrs) + .await + .unwrap_err(); + assert!( + err.to_string().contains("all 2 validated addresses"), + "{err}" + ); + } + + #[tokio::test] + async fn connect_via_validated_rejects_empty_address_list() { + let endpoint = endpoint_for(sock("127.0.0.1", 3128), None); + let err = connect_via_validated(&endpoint, "api.example.com", 443, &[]) + .await + .unwrap_err(); + assert_eq!(err.kind(), ErrorKind::InvalidInput); + } + + #[tokio::test] + async fn validated_ip_connect_tunnels_to_upstream_tls_verified_by_hostname() { + // End-to-end composition of the SSRF-to-TLS boundary: a validated + // address is CONNECTed to the corporate proxy (never the hostname), + // the proxy tunnels to the destination, and upstream TLS is verified + // against the original hostname carried in SNI. Covers both a + // matching-hostname success and a mismatched-certificate rejection. + use crate::l7::tls; + use std::sync::{Arc, Mutex}; + use tokio::io::copy_bidirectional; + + const SERVER_HOSTNAME: &str = "upstream.example.test"; + + let _ = rustls::crypto::ring::default_provider().install_default(); + + // Trusted CA; the client config trusts it, and the fake upstream + // server presents a leaf for SERVER_HOSTNAME signed by it. + let ca = tls::SandboxCa::generate().unwrap(); + let client_config = tls::build_upstream_client_config(ca.cert_pem()); + let tls_state = Arc::new(tls::ProxyTlsState::new( + tls::CertCache::new(ca), + tls::build_upstream_client_config(""), + )); + + // Fake upstream TLS server: accepts tunneled connections and completes + // a TLS handshake presenting the SERVER_HOSTNAME cert. + let tls_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let tls_addr = tls_listener.local_addr().unwrap(); + tokio::spawn(async move { + while let Ok((conn, _)) = tls_listener.accept().await { + let state = tls_state.clone(); + tokio::spawn(async move { + let _ = tls::tls_terminate_client(conn, &state, SERVER_HOSTNAME).await; + }); + } + }); + + // Fake corporate proxy: records each CONNECT request line and splices + // the tunnel to the upstream TLS server. + let proxy_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let proxy_addr = proxy_listener.local_addr().unwrap(); + let connect_lines: Arc>> = Arc::new(Mutex::new(Vec::new())); + { + let connect_lines = connect_lines.clone(); + tokio::spawn(async move { + while let Ok((mut client, _)) = proxy_listener.accept().await { + let connect_lines = connect_lines.clone(); + tokio::spawn(async move { + let request = read_request(&mut client).await; + connect_lines + .lock() + .unwrap() + .push(request.lines().next().unwrap_or_default().to_string()); + client + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + if let Ok(mut upstream) = TcpStream::connect(tls_addr).await { + let _ = copy_bidirectional(&mut client, &mut upstream).await; + } + }); + } + }); + } + + let endpoint = endpoint_for(proxy_addr, None); + + // Success: CONNECT to the validated IP, TLS verified against the + // requested hostname. + let stream = + connect_via_validated(&endpoint, SERVER_HOSTNAME, tls_addr.port(), &[tls_addr]) + .await + .unwrap(); + tls::tls_connect_upstream(stream, SERVER_HOSTNAME, &client_config) + .await + .expect("TLS must verify for the requested hostname"); + + // The proxy saw the validated IP in the CONNECT authority, never the + // hostname — the tunnel is bound to the SSRF-validated address. + { + let lines = connect_lines.lock().unwrap(); + let line = lines.last().expect("proxy received a CONNECT"); + assert!( + line.starts_with(&format!("CONNECT {}:", tls_addr.ip())), + "CONNECT must target the validated IP: {line}" + ); + assert!( + !line.contains(SERVER_HOSTNAME), + "hostname must not appear in the CONNECT authority: {line}" + ); + } + + // Mismatch: the same validated tunnel, but verifying a different + // hostname must fail — the server cert is only valid for + // SERVER_HOSTNAME, so a rebinding/split-horizon substitution cannot + // pass certificate verification. + let stream = connect_via_validated( + &endpoint, + "wrong.example.test", + tls_addr.port(), + &[tls_addr], + ) + .await + .unwrap(); + assert!( + tls::tls_connect_upstream(stream, "wrong.example.test", &client_config) + .await + .is_err(), + "TLS must reject a certificate that does not match the requested hostname" + ); + } + + #[tokio::test] + async fn connect_via_sends_hostname_on_operator_opt_in() { + let (addr, handle) = fake_proxy("HTTP/1.1 200 Connection established\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + let stream = connect_via(&endpoint, "api.example.com", 443, ConnectTarget::Hostname) + .await + .unwrap(); + drop(stream); + let request = handle.await.unwrap(); + assert!(request.starts_with("CONNECT api.example.com:443 HTTP/1.1\r\n")); + assert!(request.contains("Host: api.example.com:443\r\n")); + } + + #[tokio::test] + async fn connect_via_preserves_payload_after_connect_response() { + // The proxy's 200 response and the first tunneled bytes can arrive + // in a single read; the suffix belongs to the destination stream and + // must be replayed, not discarded. + let (addr, _handle) = + fake_proxy("HTTP/1.1 200 Connection established\r\n\r\nserver-first-payload").await; + let endpoint = endpoint_for(addr, None); + let mut stream = connect_via(&endpoint, "api.example.com", 443, ConnectTarget::Hostname) + .await + .unwrap(); + let mut payload = vec![0u8; "server-first-payload".len()]; + stream.read_exact(&mut payload).await.unwrap(); + assert_eq!(payload, b"server-first-payload"); + } + + #[tokio::test] + async fn prefixed_stream_replays_prefix_before_socket_bytes() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let connect = tokio::spawn(async move { TcpStream::connect(addr).await.unwrap() }); + let (mut accepted, _) = listener.accept().await.unwrap(); + let dialed = connect.await.unwrap(); + + accepted.write_all(b" from-socket").await.unwrap(); + let mut stream = PrefixedStream::new(dialed, b"from-prefix".to_vec()); + let mut out = vec![0u8; "from-prefix from-socket".len()]; + stream.read_exact(&mut out).await.unwrap(); + assert_eq!(out, b"from-prefix from-socket"); + + // Writes pass through to the socket untouched by the prefix. + stream.write_all(b"reply").await.unwrap(); + let mut reply = vec![0u8; 5]; + accepted.read_exact(&mut reply).await.unwrap(); + assert_eq!(reply, b"reply"); + } + + #[tokio::test] + async fn connect_via_rejects_non_200() { + let (addr, _handle) = + fake_proxy("HTTP/1.1 407 Proxy Authentication Required\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + let err = connect_via(&endpoint, "api.example.com", 443, ConnectTarget::Hostname) + .await + .unwrap_err(); + assert!(err.to_string().contains("407"), "{err}"); + } + + #[tokio::test] + async fn connect_via_rejects_malformed_response() { + let (addr, _handle) = fake_proxy("garbage without status\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + let err = connect_via(&endpoint, "api.example.com", 443, ConnectTarget::Hostname) + .await + .unwrap_err(); + assert_eq!(err.kind(), ErrorKind::InvalidData); + } + + #[tokio::test] + async fn connect_via_ipv6_target_is_bracketed() { + // Both modes must bracket IPv6 authorities in the request line. + let (addr, handle) = fake_proxy("HTTP/1.1 200 OK\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + let _ = connect_via( + &endpoint, + "v6.example.com", + 443, + ConnectTarget::Ip("2001:db8::1".parse().unwrap()), + ) + .await + .unwrap(); + let request = handle.await.unwrap(); + assert!(request.starts_with("CONNECT [2001:db8::1]:443 HTTP/1.1\r\n")); + + let (addr, handle) = fake_proxy("HTTP/1.1 200 OK\r\n\r\n").await; + let endpoint = endpoint_for(addr, None); + let _ = connect_via(&endpoint, "2001:db8::1", 443, ConnectTarget::Hostname) + .await + .unwrap(); + let request = handle.await.unwrap(); + assert!(request.starts_with("CONNECT [2001:db8::1]:443 HTTP/1.1\r\n")); + } + + #[test] + fn connect_by_hostname_requires_exact_true() { + // Default is the validated-IP binding. + let cfg = config_ok(&[(HTTPS_PROXY, "http://proxy:8080")]); + assert!(!cfg.connect_by_hostname()); + + let cfg = config_ok(&[ + (HTTPS_PROXY, "http://proxy:8080"), + (PROXY_CONNECT_BY_HOSTNAME, "true"), + ]); + assert!(cfg.connect_by_hostname()); + + // Anything other than the exact opt-in value the driver writes is a + // misconfiguration, not a silent fallback to either mode. + for value in ["false", "yes", "1", "TRUE"] { + let err = config_from(&[ + (HTTPS_PROXY, "http://proxy:8080"), + (PROXY_CONNECT_BY_HOSTNAME, value), + ]) + .unwrap_err(); + assert!(err.contains(PROXY_CONNECT_BY_HOSTNAME), "{value}: {err}"); + } + } + + #[test] + fn connect_by_hostname_without_proxy_is_fatal() { + let err = config_from(&[(PROXY_CONNECT_BY_HOSTNAME, "true")]).unwrap_err(); + assert!(err.contains(PROXY_CONNECT_BY_HOSTNAME), "{err}"); + assert!(err.contains("no upstream proxy"), "{err}"); + } +} diff --git a/crates/openshell-supervisor-process/src/process.rs b/crates/openshell-supervisor-process/src/process.rs index 72effa98f8..322dcadd86 100644 --- a/crates/openshell-supervisor-process/src/process.rs +++ b/crates/openshell-supervisor-process/src/process.rs @@ -1168,10 +1168,14 @@ fn rewrite_group_at(path: &Path, gid: &str) -> Result<()> { /// Symlinks are skipped (not followed) to prevent privilege escalation via /// malicious container images. The TOCTOU window is not exploitable because /// no untrusted process is running yet. +/// +/// The root path is chowned unconditionally — EROFS there is a hard error +/// (a read-only `/sandbox` is a misconfiguration). For children, `EROFS` +/// causes the walker to skip that path and its entire subtree — descending +/// into a read-only mount we do not control would be a TOCTOU risk +/// (CWE-367/CWE-59). Siblings of the read-only path are still visited. #[cfg(unix)] fn chown_sandbox_home(root: &Path, uid: Option, gid: Option) -> Result<()> { - use nix::unistd::chown; - let meta = std::fs::symlink_metadata(root).into_diagnostic()?; if meta.file_type().is_symlink() { return Err(miette::miette!( @@ -1180,23 +1184,63 @@ fn chown_sandbox_home(root: &Path, uid: Option, gid: Option) -> Result )); } - chown(root, uid, gid).into_diagnostic()?; + nix::unistd::chown(root, uid, gid).into_diagnostic()?; - if meta.is_dir() - && let Ok(entries) = std::fs::read_dir(root) - { - for entry in entries { - let entry = entry.into_diagnostic()?; - let path = entry.path(); - if path - .symlink_metadata() - .is_ok_and(|m| m.file_type().is_symlink()) - { - debug!(path = %path.display(), "Skipping symlink during sandbox home chown"); - continue; + if meta.is_dir() { + chown_children(root, uid, gid, &nix::unistd::chown)?; + } + + Ok(()) +} + +/// Walk directory children and chown each entry, skipping symlinks and +/// EROFS subtrees. Called after the parent has already been chowned. +#[cfg(unix)] +fn chown_children( + dir: &Path, + uid: Option, + gid: Option, + do_chown: &impl Fn(&Path, Option, Option) -> nix::Result<()>, +) -> Result<()> { + match std::fs::read_dir(dir) { + Ok(entries) => { + for entry in entries { + let entry = entry.into_diagnostic()?; + let child = entry.path(); + chown_recursive(&child, uid, gid, do_chown)?; } - chown_sandbox_home(&path, uid, gid)?; } + Err(e) => { + debug!(path = %dir.display(), error = %e, "Cannot list directory during sandbox home chown"); + } + } + Ok(()) +} + +#[cfg(unix)] +fn chown_recursive( + path: &Path, + uid: Option, + gid: Option, + do_chown: &impl Fn(&Path, Option, Option) -> nix::Result<()>, +) -> Result<()> { + let meta = std::fs::symlink_metadata(path).into_diagnostic()?; + + if meta.file_type().is_symlink() { + debug!(path = %path.display(), "Skipping symlink during sandbox home chown"); + return Ok(()); + } + + if let Err(e) = do_chown(path, uid, gid) { + if e == nix::errno::Errno::EROFS { + debug!(path = %path.display(), "Skipping read-only path during sandbox home chown"); + return Ok(()); + } + return Err(e).into_diagnostic(); + } + + if meta.is_dir() { + chown_children(path, uid, gid, do_chown)?; } Ok(()) @@ -2115,6 +2159,89 @@ mod tests { .expect("should skip symlink children without error"); } + #[cfg(unix)] + #[test] + fn chown_recursive_skips_erofs_subtree_but_continues_siblings() { + use std::sync::{Arc, Mutex}; + + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("sandbox"); + std::fs::create_dir(&root).unwrap(); + + let readonly_dir = root.join("ro-mount"); + std::fs::create_dir(&readonly_dir).unwrap(); + std::fs::write(readonly_dir.join("child-under-ro.txt"), "data").unwrap(); + + std::fs::write(root.join("writable-sibling.txt"), "data").unwrap(); + + let uid = Some(nix::unistd::geteuid()); + let gid = Some(nix::unistd::getegid()); + + let chowned: Arc>> = Arc::new(Mutex::new(Vec::new())); + let chowned_ref = Arc::clone(&chowned); + + let readonly_dir_clone = readonly_dir.clone(); + let fake_chown = + move |path: &Path, _uid: Option, _gid: Option| -> nix::Result<()> { + if path == readonly_dir_clone { + return Err(nix::errno::Errno::EROFS); + } + chowned_ref.lock().unwrap().push(path.to_path_buf()); + Ok(()) + }; + + chown_children(&root, uid, gid, &fake_chown).expect("EROFS should be handled gracefully"); + + let chowned = chowned.lock().unwrap(); + assert!( + !chowned.contains(&readonly_dir.join("child-under-ro.txt")), + "children under EROFS directory must NOT be descended into" + ); + assert!( + chowned.contains(&root.join("writable-sibling.txt")), + "writable sibling should still be chowned" + ); + } + + #[cfg(unix)] + #[test] + fn chown_recursive_propagates_non_erofs_errors() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("sandbox"); + std::fs::create_dir(&root).unwrap(); + + let uid = Some(nix::unistd::geteuid()); + let gid = Some(nix::unistd::getegid()); + + let fake_chown = |_path: &Path, _uid: Option, _gid: Option| -> nix::Result<()> { + Err(nix::errno::Errno::EPERM) + }; + + let result = chown_recursive(&root, uid, gid, &fake_chown); + assert!(result.is_err(), "non-EROFS errors should propagate"); + } + + #[cfg(unix)] + #[test] + fn chown_children_skips_all_erofs_children_gracefully() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("sandbox"); + std::fs::create_dir(&root).unwrap(); + std::fs::create_dir(root.join("a")).unwrap(); + std::fs::write(root.join("b.txt"), "data").unwrap(); + + let uid = Some(nix::unistd::geteuid()); + let gid = Some(nix::unistd::getegid()); + + let always_erofs = |_path: &Path, + _uid: Option, + _gid: Option| + -> nix::Result<()> { Err(nix::errno::Errno::EROFS) }; + + chown_children(&root, uid, gid, &always_erofs) + .expect("EROFS on all children should be skipped gracefully"); + } + #[cfg(unix)] #[test] fn rewrite_passwd_modifies_existing_sandbox_entry() { diff --git a/crates/openshell-supervisor-process/src/run.rs b/crates/openshell-supervisor-process/src/run.rs index 5a06d97a8f..3cb29d1e6b 100644 --- a/crates/openshell-supervisor-process/src/run.rs +++ b/crates/openshell-supervisor-process/src/run.rs @@ -24,6 +24,7 @@ use openshell_ocsf::{ #[cfg(target_os = "linux")] use crate::netns::NetworkNamespace; use openshell_core::policy::{NetworkMode, SandboxPolicy}; +use openshell_core::proposals::AgentProposals; use openshell_core::provider_credentials::ProviderCredentialState; #[cfg(target_os = "linux")] @@ -64,6 +65,7 @@ pub async fn run_process( provider_credentials: ProviderCredentialState, provider_env: std::collections::HashMap, ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>, + agent_proposals: AgentProposals, #[cfg(target_os = "linux")] netns: Option<&NetworkNamespace>, #[cfg(target_os = "linux")] bypass_denial_tx: Option< tokio::sync::mpsc::UnboundedSender, @@ -99,7 +101,7 @@ pub async fn run_process( // proposals flag is on at startup, rather than waiting for the policy // poll loop's first tick. In offline/file-mode there is no gateway, so // the flag stays at its default (false) and no skill is installed. - install_initial_agent_skill(sandbox_id, openshell_endpoint).await; + install_initial_agent_skill(sandbox_id, openshell_endpoint, &agent_proposals).await; // Install the supervisor seccomp prelude before spawning any workload-side // tasks. By this point the orchestrator has finished privileged startup @@ -416,17 +418,12 @@ fn ssh_proxy_url_for_policy( /// /// Best-effort: any failure (no gateway, RPC error, install failure) is /// logged but does not fail sandbox startup. -async fn install_initial_agent_skill(sandbox_id: Option<&str>, openshell_endpoint: Option<&str>) { +async fn install_initial_agent_skill( + sandbox_id: Option<&str>, + openshell_endpoint: Option<&str>, + agent_proposals: &AgentProposals, +) { use openshell_core::proto::setting_value; - use std::sync::atomic::Ordering; - - let Some(flag) = openshell_core::proposals::AGENT_PROPOSALS_ENABLED.get() else { - // The orchestrator is responsible for setting the OnceLock before - // calling run_process. If it isn't set, behave as if the flag is - // off and skip the install. - tracing::debug!("AGENT_PROPOSALS_ENABLED not initialized; skipping skill install"); - return; - }; if let (Some(id), Some(endpoint)) = (sandbox_id, openshell_endpoint) && let Ok(client) = @@ -443,10 +440,10 @@ async fn install_initial_agent_skill(sandbox_id: Option<&str>, openshell_endpoin _ => None, }) .unwrap_or(false); - flag.store(initial, Ordering::Relaxed); + agent_proposals.set_enabled(initial); } - if openshell_core::proposals::agent_proposals_enabled() { + if agent_proposals.enabled() { match crate::skills::install_static_skills() { Ok(installed) => info!( path = %installed.policy_advisor.display(), diff --git a/crates/openshell-tui/Cargo.toml b/crates/openshell-tui/Cargo.toml index caa2ec1ab4..c1c64d83cb 100644 --- a/crates/openshell-tui/Cargo.toml +++ b/crates/openshell-tui/Cargo.toml @@ -26,6 +26,7 @@ miette = { workspace = true } owo-colors = { workspace = true } serde = { workspace = true } url = { workspace = true } +indexmap = { workspace = true } [lints] workspace = true diff --git a/crates/openshell-tui/src/app.rs b/crates/openshell-tui/src/app.rs index 6db3ac14f2..db7eef1d32 100644 --- a/crates/openshell-tui/src/app.rs +++ b/crates/openshell-tui/src/app.rs @@ -5,6 +5,7 @@ use std::collections::HashMap; use std::time::{Duration, Instant}; use crossterm::event::{KeyCode, KeyEvent, KeyModifiers}; +use indexmap::IndexMap; use openshell_bootstrap::GatewayMetadataSource; use openshell_core::auth::EdgeAuthInterceptor; use openshell_core::proto::open_shell_client::OpenShellClient; @@ -247,8 +248,9 @@ pub type CreateFormData = ( ); /// Which field is focused in the create sandbox modal. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub enum CreateFormField { + #[default] Name, Image, Command, @@ -290,9 +292,10 @@ pub struct ProviderEntry { } /// Tracks which phase the create sandbox modal is in. -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone, Default, PartialEq, Eq)] pub enum CreatePhase { /// Filling out the form. + #[default] Form, /// Creating the sandbox (background task running). Creating, @@ -302,6 +305,7 @@ pub enum CreatePhase { pub const MIN_CREATING_DISPLAY: Duration = Duration::from_secs(4); /// State for the create sandbox modal form. +#[derive(Default)] pub struct CreateSandboxForm { pub focused_field: CreateFormField, pub name: String, @@ -326,9 +330,10 @@ pub struct CreateSandboxForm { // Create provider form // --------------------------------------------------------------------------- -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone, Default, PartialEq, Eq)] pub enum CreateProviderPhase { /// Pick provider type from the known list. + #[default] SelectType, /// Choose: autodetect from env or enter key manually. ChooseMethod, @@ -339,8 +344,9 @@ pub enum CreateProviderPhase { } /// Which field is focused in the provider key entry form. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub enum ProviderKeyField { + #[default] Name, /// Focused credential row for known types (index via `cred_cursor`). Credential, @@ -348,9 +354,16 @@ pub enum ProviderKeyField { EnvVarName, /// Custom env var value (generic / no-known-env-vars types only). GenericValue, + /// Browsing/deleting existing config entries (Up/Down/Ctrl+D). + ConfigList, + /// Config key name input. + ConfigKeyName, + /// Config key value input. + ConfigKeyValue, Submit, } +#[derive(Default)] pub struct CreateProviderForm { pub phase: CreateProviderPhase, /// Known provider type slugs. @@ -364,6 +377,14 @@ pub struct CreateProviderForm { pub credentials: Vec<(String, String)>, /// Which credential row is focused. pub cred_cursor: usize, + /// Provider config key-value pairs (e.g. `ANTHROPIC_BASE_URL`). + pub config: IndexMap, + /// Which existing config entry is selected (for deletion). + pub config_cursor: usize, + /// Config key being entered. + pub config_key_input: String, + /// Config value being entered. + pub config_value_input: String, /// For generic / types with no known env vars: custom env var name. pub generic_env_name: String, /// For generic / types with no known env vars: custom value. @@ -411,12 +432,12 @@ pub struct ProviderDetailView { } #[derive(Clone)] -pub struct ProviderV2Entry { +pub struct ProviderListEntry { pub provider: openshell_core::proto::Provider, pub profile: Option, } -impl ProviderV2Entry { +impl ProviderListEntry { pub fn name(&self) -> &str { provider_name(&self.provider) } @@ -498,9 +519,41 @@ pub struct UpdateProviderForm { pub provider_type: String, pub credential_key: String, pub new_value: String, + pub config: IndexMap, + pub original_config: IndexMap, + pub config_key_input: String, + pub config_value_input: String, + pub config_cursor: usize, + pub deleted_keys: Vec, + pub focus: UpdateProviderField, pub status: Option, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UpdateProviderField { + CredentialValue, + ConfigEntry, + ConfigKey, + ConfigValue, + Submit, +} + +// --------------------------------------------------------------------------- +// Shared config helpers +// --------------------------------------------------------------------------- + +fn flush_config_input( + config: &mut IndexMap, + key_input: &mut String, + value_input: &mut String, +) -> bool { + if !key_input.is_empty() && !value_input.is_empty() { + config.insert(std::mem::take(key_input), std::mem::take(value_input)); + return true; + } + false +} + // --------------------------------------------------------------------------- // App state // --------------------------------------------------------------------------- @@ -538,7 +591,7 @@ pub struct App { // Provider list pub providers_v2_enabled: bool, - pub provider_entries: Vec, + pub provider_entries: Vec, pub provider_names: Vec, pub provider_types: Vec, pub provider_cred_keys: Vec, @@ -2167,17 +2220,8 @@ impl App { .collect(); self.create_form = Some(CreateSandboxForm { - focused_field: CreateFormField::Name, - name: String::new(), - image: String::new(), - command: String::new(), providers, - provider_cursor: 0, - ports: String::new(), - status: None, - phase: CreatePhase::Form, - anim_start: None, - create_result: None, + ..CreateSandboxForm::default() }); } @@ -2278,22 +2322,8 @@ impl App { let types: Vec = known.into_iter().map(String::from).collect(); self.create_provider_form = Some(CreateProviderForm { - phase: CreateProviderPhase::SelectType, types, - type_cursor: 0, - method_cursor: 0, - name: String::new(), - credentials: Vec::new(), - cred_cursor: 0, - generic_env_name: String::new(), - generic_value: String::new(), - key_field: ProviderKeyField::Name, - is_generic: false, - status: None, - warning: None, - anim_start: None, - create_result: None, - discovered_credentials: None, + ..CreateProviderForm::default() }); } @@ -2393,34 +2423,119 @@ impl App { form.name.clear(); form.credentials.clear(); form.cred_cursor = 0; + form.config.clear(); + form.config_cursor = 0; + form.config_key_input.clear(); + form.config_value_input.clear(); form.generic_env_name.clear(); form.generic_value.clear(); } KeyCode::Tab => { if form.is_generic { - // Name → EnvVarName → GenericValue → Submit → Name - form.key_field = match form.key_field { - ProviderKeyField::Name => ProviderKeyField::EnvVarName, - ProviderKeyField::EnvVarName => ProviderKeyField::GenericValue, - ProviderKeyField::GenericValue => ProviderKeyField::Submit, - _ => ProviderKeyField::Name, - }; + // Name → EnvVarName → GenericValue → ConfigList → ConfigKeyName → ConfigKeyValue → Submit → Name + match form.key_field { + ProviderKeyField::Name => { + form.key_field = ProviderKeyField::EnvVarName; + } + ProviderKeyField::EnvVarName => { + form.key_field = ProviderKeyField::GenericValue; + } + ProviderKeyField::GenericValue => { + if form.config.is_empty() { + form.key_field = ProviderKeyField::ConfigKeyName; + } else { + form.key_field = ProviderKeyField::ConfigList; + form.config_cursor = 0_usize; + } + } + ProviderKeyField::ConfigList => { + if form.config_cursor < form.config.len().saturating_sub(1) { + form.config_cursor += 1; + } else { + form.key_field = ProviderKeyField::ConfigKeyName; + } + } + ProviderKeyField::ConfigKeyName => { + form.key_field = ProviderKeyField::ConfigKeyValue; + } + ProviderKeyField::ConfigKeyValue => { + if flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ) { + form.key_field = ProviderKeyField::ConfigKeyName; + form.config_cursor = form.config.len().saturating_sub(1_usize); + } else if form.config_key_input.is_empty() + && form.config_value_input.is_empty() + { + form.key_field = ProviderKeyField::Submit; + } else { + form.status = Some( + "Both key and value required to add config entry." + .to_string(), + ); + form.key_field = ProviderKeyField::ConfigKeyName; + } + } + _ => { + form.key_field = ProviderKeyField::Name; + } + } } else { - // Name → Credential[0..N-1] → Submit → Name + // Name → Credential[0..N-1] → [ConfigList →] ConfigKeyName → ConfigKeyValue → Submit → Name match form.key_field { ProviderKeyField::Name => { if form.credentials.is_empty() { - form.key_field = ProviderKeyField::Submit; + if form.config.is_empty() { + form.key_field = ProviderKeyField::ConfigKeyName; + } else { + form.key_field = ProviderKeyField::ConfigList; + form.config_cursor = 0_usize; + } } else { form.key_field = ProviderKeyField::Credential; - form.cred_cursor = 0; + form.cred_cursor = 0_usize; } } ProviderKeyField::Credential => { if form.cred_cursor < form.credentials.len().saturating_sub(1) { form.cred_cursor += 1; + } else if !form.config.is_empty() { + form.key_field = ProviderKeyField::ConfigList; + form.config_cursor = 0_usize; } else { + form.key_field = ProviderKeyField::ConfigKeyName; + } + } + ProviderKeyField::ConfigList => { + if form.config_cursor < form.config.len().saturating_sub(1) { + form.config_cursor += 1_usize; + } else { + form.key_field = ProviderKeyField::ConfigKeyName; + } + } + ProviderKeyField::ConfigKeyName => { + form.key_field = ProviderKeyField::ConfigKeyValue; + } + ProviderKeyField::ConfigKeyValue => { + if flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ) { + form.key_field = ProviderKeyField::ConfigKeyName; + form.config_cursor = form.config.len().saturating_sub(1_usize); + } else if form.config_key_input.is_empty() + && form.config_value_input.is_empty() + { form.key_field = ProviderKeyField::Submit; + } else { + form.status = Some( + "Both key and value required to add config entry." + .to_string(), + ); + form.key_field = ProviderKeyField::ConfigKeyName; } } _ => { @@ -2431,12 +2546,38 @@ impl App { } KeyCode::BackTab => { if form.is_generic { - form.key_field = match form.key_field { - ProviderKeyField::EnvVarName => ProviderKeyField::Name, - ProviderKeyField::GenericValue => ProviderKeyField::EnvVarName, - ProviderKeyField::Submit => ProviderKeyField::GenericValue, - _ => ProviderKeyField::Submit, - }; + match form.key_field { + ProviderKeyField::EnvVarName => { + form.key_field = ProviderKeyField::Name; + } + ProviderKeyField::GenericValue => { + form.key_field = ProviderKeyField::EnvVarName; + } + ProviderKeyField::ConfigList => { + if form.config_cursor > 0 { + form.config_cursor -= 1_usize; + } else { + form.key_field = ProviderKeyField::GenericValue; + } + } + ProviderKeyField::ConfigKeyName => { + if form.config.is_empty() { + form.key_field = ProviderKeyField::GenericValue; + } else { + form.config_cursor = form.config.len().saturating_sub(1); + form.key_field = ProviderKeyField::ConfigList; + } + } + ProviderKeyField::ConfigKeyValue => { + form.key_field = ProviderKeyField::ConfigKeyName; + } + ProviderKeyField::Submit => { + form.key_field = ProviderKeyField::ConfigKeyValue; + } + _ => { + form.key_field = ProviderKeyField::Submit; + } + } } else { match form.key_field { ProviderKeyField::Credential => { @@ -2446,14 +2587,33 @@ impl App { form.key_field = ProviderKeyField::Name; } } - ProviderKeyField::Submit => { - if form.credentials.is_empty() { + ProviderKeyField::ConfigList => { + if form.config_cursor > 0 { + form.config_cursor -= 1; + } else if form.credentials.is_empty() { + form.key_field = ProviderKeyField::Name; + } else { + form.key_field = ProviderKeyField::Credential; + form.cred_cursor = form.credentials.len().saturating_sub(1); + } + } + ProviderKeyField::ConfigKeyName => { + if !form.config.is_empty() { + form.config_cursor = form.config.len().saturating_sub(1); + form.key_field = ProviderKeyField::ConfigList; + } else if form.credentials.is_empty() { form.key_field = ProviderKeyField::Name; } else { form.key_field = ProviderKeyField::Credential; form.cred_cursor = form.credentials.len().saturating_sub(1); } } + ProviderKeyField::ConfigKeyValue => { + form.key_field = ProviderKeyField::ConfigKeyName; + } + ProviderKeyField::Submit => { + form.key_field = ProviderKeyField::ConfigKeyValue; + } _ => { form.key_field = ProviderKeyField::Submit; } @@ -2467,6 +2627,60 @@ impl App { Self::handle_text_input(value, key); } } + ProviderKeyField::ConfigList => match key.code { + KeyCode::Up => { + form.config_cursor = form.config_cursor.saturating_sub(1); + } + KeyCode::Down if !form.config.is_empty() => { + form.config_cursor = + (form.config_cursor + 1).min(form.config.len() - 1); + } + KeyCode::Char('d') + if key.modifiers.contains(KeyModifiers::CONTROL) + && !form.config.is_empty() => + { + let key_to_remove = form + .config + .keys() + .nth(form.config_cursor) + .cloned() + .unwrap_or_default(); + form.config.shift_remove(&key_to_remove); + form.config_cursor = + form.config_cursor.min(form.config.len().saturating_sub(1)); + if form.config.is_empty() { + form.key_field = ProviderKeyField::ConfigKeyName; + } + } + _ => {} + }, + ProviderKeyField::ConfigKeyName => match key.code { + KeyCode::Enter => { + flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ); + } + _ => { + Self::handle_text_input(&mut form.config_key_input, key); + } + }, + ProviderKeyField::ConfigKeyValue => match key.code { + KeyCode::Enter => { + if flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ) { + form.key_field = ProviderKeyField::ConfigKeyName; + form.config_cursor = form.config.len().saturating_sub(1_usize); + } + } + _ => { + Self::handle_text_input(&mut form.config_value_input, key); + } + }, ProviderKeyField::EnvVarName => { Self::handle_text_input(&mut form.generic_env_name, key); } @@ -2475,6 +2689,20 @@ impl App { } ProviderKeyField::Submit => { if key.code == KeyCode::Enter { + flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ); + if !form.config_key_input.is_empty() + || !form.config_value_input.is_empty() + { + form.status = Some( + "Both key and value are required to add config entry." + .to_string(), + ); + return; + } // Validate and build credentials map. let mut creds = HashMap::new(); if form.is_generic { @@ -2588,6 +2816,17 @@ impl App { .get(self.provider_selected) .cloned() .unwrap_or_default(); + let existing_config = self + .provider_entries + .get(self.provider_selected) + .map(|e| { + e.provider + .config + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .collect::>() + }) + .unwrap_or_default(); // If we don't know the credential key, derive from registry. let key = if cred_key.is_empty() { @@ -2605,6 +2844,13 @@ impl App { provider_type: ptype, credential_key: key, new_value: String::new(), + config: existing_config.clone(), + original_config: existing_config, + config_key_input: String::new(), + config_value_input: String::new(), + config_cursor: 0, + deleted_keys: Vec::new(), + focus: UpdateProviderField::CredentialValue, status: None, }); } @@ -2618,18 +2864,153 @@ impl App { KeyCode::Esc => { self.update_provider_form = None; } - KeyCode::Enter => { - if form.new_value.is_empty() { - form.status = Some("Value is required.".to_string()); - return; + KeyCode::Tab => match form.focus { + UpdateProviderField::CredentialValue => { + if form.config.is_empty() { + form.focus = UpdateProviderField::ConfigKey; + } else { + form.focus = UpdateProviderField::ConfigEntry; + form.config_cursor = 0_usize; + } } - self.pending_provider_update = true; - } - KeyCode::Char(c) => form.new_value.push(c), - KeyCode::Backspace => { - form.new_value.pop(); - } - _ => {} + UpdateProviderField::ConfigEntry => { + if form.config_cursor < form.config.len().saturating_sub(1) { + form.config_cursor += 1_usize; + } else { + form.focus = UpdateProviderField::ConfigKey; + } + } + UpdateProviderField::ConfigKey => { + form.focus = UpdateProviderField::ConfigValue; + } + UpdateProviderField::ConfigValue => { + if flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ) { + form.focus = UpdateProviderField::ConfigKey; + form.config_cursor = form.config.len().saturating_sub(1_usize); + } else if form.config_key_input.is_empty() && form.config_value_input.is_empty() + { + form.focus = UpdateProviderField::Submit; + } else { + form.status = + Some("Both key and value required to add config entry.".to_string()); + form.focus = UpdateProviderField::ConfigKey; + } + } + UpdateProviderField::Submit => { + form.focus = UpdateProviderField::CredentialValue; + } + }, + KeyCode::BackTab => match form.focus { + UpdateProviderField::CredentialValue => { + form.focus = UpdateProviderField::Submit; + } + UpdateProviderField::ConfigEntry => { + if form.config_cursor > 0 { + form.config_cursor -= 1_usize; + } else { + form.focus = UpdateProviderField::CredentialValue; + } + } + UpdateProviderField::ConfigKey => { + if form.config.is_empty() { + form.focus = UpdateProviderField::CredentialValue; + } else { + form.focus = UpdateProviderField::ConfigEntry; + form.config_cursor = form.config.len().saturating_sub(1); + } + } + UpdateProviderField::ConfigValue => { + form.focus = UpdateProviderField::ConfigKey; + } + UpdateProviderField::Submit => { + form.focus = UpdateProviderField::ConfigValue; + } + }, + _ => match form.focus { + UpdateProviderField::CredentialValue => { + Self::handle_text_input(&mut form.new_value, key); + } + UpdateProviderField::ConfigEntry => match key.code { + KeyCode::Up => { + form.config_cursor = form.config_cursor.saturating_sub(1); + } + KeyCode::Down if !form.config.is_empty() => { + form.config_cursor = (form.config_cursor + 1).min(form.config.len() - 1); + } + KeyCode::Char('d') + if key.modifiers.contains(KeyModifiers::CONTROL) + && !form.config.is_empty() => + { + let key_to_remove = form + .config + .keys() + .nth(form.config_cursor) + .cloned() + .unwrap_or_default(); + form.deleted_keys.push(key_to_remove.clone()); + form.config.shift_remove(&key_to_remove); + form.config_cursor = + form.config_cursor.min(form.config.len().saturating_sub(1)); + if form.config.is_empty() { + form.focus = UpdateProviderField::ConfigKey; + } + } + _ => {} + }, + UpdateProviderField::ConfigKey => match key.code { + KeyCode::Enter => { + flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ); + } + _ => { + Self::handle_text_input(&mut form.config_key_input, key); + } + }, + UpdateProviderField::ConfigValue => match key.code { + KeyCode::Enter => { + if flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ) { + form.focus = UpdateProviderField::ConfigKey; + form.config_cursor = form.config.len().saturating_sub(1_usize); + } + } + _ => { + Self::handle_text_input(&mut form.config_value_input, key); + } + }, + UpdateProviderField::Submit => { + if key.code == KeyCode::Enter { + flush_config_input( + &mut form.config, + &mut form.config_key_input, + &mut form.config_value_input, + ); + if !form.config_key_input.is_empty() || !form.config_value_input.is_empty() + { + form.status = Some( + "Both key and value are required to add config entry.".to_string(), + ); + return; + } + if form.new_value.is_empty() && form.config == form.original_config { + form.status = + Some("Credential value or config keys required.".to_string()); + return; + } + self.pending_provider_update = true; + } + } + }, } } @@ -2786,7 +3167,12 @@ impl App { }, ); - let mut config_lines = provider.config.keys().cloned().collect::>(); + let mut config_lines = provider + .config + .iter() + .map(|(k, v)| format!("{k}={v}")) + .collect::>(); + config_lines.sort(); if config_lines.is_empty() { config_lines.push("".to_string()); @@ -3171,4 +3557,246 @@ mod tests { let result = workspaces.get(selected).unwrap_or(¤t); assert_eq!(*result, "default"); } + + #[test] + fn flush_config_input_inserts_when_both_present() { + let mut config = IndexMap::new(); + let mut key = "FOO".to_string(); + let mut val = "bar".to_string(); + assert!(flush_config_input(&mut config, &mut key, &mut val)); + assert_eq!(config.get("FOO"), Some(&"bar".to_string())); + assert!(key.is_empty()); + assert!(val.is_empty()); + } + + #[test] + fn flush_config_input_noop_when_key_empty() { + let mut config = IndexMap::new(); + let mut key = String::new(); + let mut val = "bar".to_string(); + assert!(!flush_config_input(&mut config, &mut key, &mut val)); + assert!(config.is_empty()); + assert_eq!(val, "bar"); + } + + #[test] + fn flush_config_input_noop_when_value_empty() { + let mut config = IndexMap::new(); + let mut key = "FOO".to_string(); + let mut val = String::new(); + assert!(!flush_config_input(&mut config, &mut key, &mut val)); + assert!(config.is_empty()); + assert_eq!(key, "FOO"); + } + + // -- config deletion tombstones ------------------------------------ + + #[test] + fn delete_config_entry_records_tombstone() { + let config = IndexMap::from([("FOO".into(), "1".into()), ("BAR".into(), "2".into())]); + + let mut form = UpdateProviderForm { + provider_name: "p".into(), + provider_type: "t".into(), + credential_key: "k".into(), + new_value: String::new(), + original_config: config.clone(), + config, + config_key_input: String::new(), + config_value_input: String::new(), + config_cursor: 0, + focus: UpdateProviderField::ConfigKey, + status: None, + deleted_keys: Vec::new(), + }; + + let key_to_remove = form.config.keys().next().cloned().unwrap(); + form.deleted_keys.push(key_to_remove.clone()); + form.config.shift_remove(&key_to_remove); + + assert!(!form.config.contains_key("FOO")); + assert!(form.deleted_keys.contains(&"FOO".to_owned())); + } + + #[test] + fn delete_last_config_entry_allows_submit() { + let form = UpdateProviderForm { + provider_name: "p".into(), + provider_type: "t".into(), + credential_key: "k".into(), + new_value: String::new(), + config: IndexMap::new(), + original_config: IndexMap::from([("FOO".into(), "1".into())]), + config_key_input: String::new(), + config_value_input: String::new(), + config_cursor: 0, + focus: UpdateProviderField::Submit, + status: None, + deleted_keys: vec!["FOO".into()], + }; + + assert!( + !(form.new_value.is_empty() && form.config.is_empty() && form.deleted_keys.is_empty()) + ); + } + + // -- cursor-relative scroll window --------------------------------- + + #[test] + fn scroll_offset_zero_when_within_window() { + let (total, cursor, window) = (4_usize, 3_usize, 6_usize); + let offset = if total > window { + cursor + .saturating_sub(window - 2_usize) + .min(total.saturating_sub(window)) + } else { + 0_usize + }; + + assert_eq!(offset, 0_usize); + } + + #[test] + fn scroll_offset_follows_cursor_past_window() { + let (total, cursor, window) = (10_usize, 8_usize, 6_usize); + let offset = if total > window { + cursor + .saturating_sub(window - 2) + .min(total.saturating_sub(window)) + } else { + 0 + }; + assert_eq!(offset, 4); + assert!(cursor >= offset && cursor < offset + window); + } + + // -- pending input flush on submit --------------------------------- + + #[test] + fn pending_config_input_flushed_on_submit() { + let mut config = IndexMap::new(); + let mut key_input = "MY_KEY".to_string(); + let mut val_input = "my_val".to_string(); + flush_config_input(&mut config, &mut key_input, &mut val_input); + assert_eq!(config.get("MY_KEY"), Some(&"my_val".to_string())); + assert!(key_input.is_empty()); + assert!(val_input.is_empty()); + } + + // -- delta-only update request ------------------------------------- + + #[test] + fn update_request_contains_only_config_delta() { + let original_config: IndexMap = IndexMap::from([ + ("A".to_string(), "1".to_string()), + ("B".to_string(), "2".to_string()), + ]); + let config: IndexMap = IndexMap::from([ + ("A".to_string(), "1".to_string()), + ("B".to_string(), "changed".to_string()), + ]); + + let delta: HashMap = config + .iter() + .filter(|(k, v)| original_config.get(*k) != Some(*v)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + + assert!(delta.contains_key("B"), "changed key must be in delta"); + assert!( + !delta.contains_key("A"), + "unchanged key must not be in delta" + ); + } + + #[test] + fn deleted_key_tombstoned_readded_key_not_tombstoned() { + let original_config: IndexMap = IndexMap::from([ + ("DEL".to_string(), "old".to_string()), + ("READD".to_string(), "orig".to_string()), + ("KEEP".to_string(), "keep".to_string()), + ]); + // DEL was removed, READD was deleted then re-added with new value, KEEP unchanged. + let config: IndexMap = IndexMap::from([ + ("READD".to_string(), "new".to_string()), + ("KEEP".to_string(), "keep".to_string()), + ]); + + let mut request = config + .iter() + .filter(|(k, v)| original_config.get(*k) != Some(*v)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect::>(); + original_config + .keys() + .filter(|k| !config.contains_key(*k)) + .for_each(|key| { + request.insert(key.clone(), String::new()); + }); + + assert_eq!( + request.get("DEL"), + Some(&String::new()), + "DEL must be tombstoned" + ); + assert_eq!( + request.get("READD"), + Some(&"new".to_string()), + "READD must have new value, not tombstone" + ); + assert!( + !request.contains_key("KEEP"), + "KEEP unchanged must not be in delta" + ); + } + + // -- partial config input on submit -------------------------------- + + #[test] + fn submit_guard_triggers_when_key_filled_value_empty() { + let mut config = IndexMap::new(); + let mut key_input = "FOO".to_string(); + let mut val_input = String::new(); + + let flushed = flush_config_input(&mut config, &mut key_input, &mut val_input); + + assert!(!flushed, "flush must fail when value is empty"); + assert!( + !key_input.is_empty() || !val_input.is_empty(), + "submit guard must detect partial input" + ); + assert!(config.is_empty(), "config must not be modified"); + } + + #[test] + fn submit_guard_triggers_when_value_filled_key_empty() { + let mut config = IndexMap::new(); + let mut key_input = String::new(); + let mut val_input = "bar".to_string(); + + let flushed = flush_config_input(&mut config, &mut key_input, &mut val_input); + + assert!(!flushed, "flush must fail when key is empty"); + assert!( + !key_input.is_empty() || !val_input.is_empty(), + "submit guard must detect partial input" + ); + assert!(config.is_empty(), "config must not be modified"); + } + + #[test] + fn submit_guard_clear_when_both_filled() { + let mut config = IndexMap::new(); + let mut key_input = "FOO".to_string(); + let mut val_input = "bar".to_string(); + + let flushed = flush_config_input(&mut config, &mut key_input, &mut val_input); + + assert!(flushed, "flush must succeed when both fields filled"); + assert!( + key_input.is_empty() && val_input.is_empty(), + "submit guard must not trigger after successful flush" + ); + assert_eq!(config.get("FOO"), Some(&"bar".to_string())); + } } diff --git a/crates/openshell-tui/src/lib.rs b/crates/openshell-tui/src/lib.rs index 0d9d9fe613..b3937e2b90 100644 --- a/crates/openshell-tui/src/lib.rs +++ b/crates/openshell-tui/src/lib.rs @@ -1647,6 +1647,11 @@ fn spawn_create_provider(app: &App, tx: mpsc::UnboundedSender) { }; let credentials = form.discovered_credentials.clone().unwrap_or_default(); let workspace = app.current_workspace.clone(); + let config: HashMap = form + .config + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); tokio::spawn(async move { // Try with the chosen name, retry with suffix on collision. @@ -1671,7 +1676,7 @@ fn spawn_create_provider(app: &App, tx: mpsc::UnboundedSender) { }), r#type: ptype.clone(), credentials: credentials.clone(), - config: HashMap::default(), + config: config.clone(), credential_expires_at_ms: HashMap::default(), profile_workspace: workspace.clone(), }), @@ -1751,10 +1756,24 @@ fn spawn_update_provider(app: &App, tx: mpsc::UnboundedSender) { let cred_key = form.credential_key.clone(); let new_value = form.new_value.clone(); let workspace = app.selected_provider_workspace(); + let mut config: HashMap = form + .config + .iter() + .filter(|(k, v)| form.original_config.get(*k) != Some(*v)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + form.original_config + .keys() + .filter(|k| !form.config.contains_key(*k)) + .for_each(|key| { + config.insert(key.clone(), String::new()); + }); tokio::spawn(async move { let mut credentials = HashMap::new(); - credentials.insert(cred_key, new_value); + if !new_value.is_empty() { + credentials.insert(cred_key, new_value); + } let req = openshell_core::proto::UpdateProviderRequest { provider: Some(openshell_core::proto::Provider { @@ -1770,7 +1789,7 @@ fn spawn_update_provider(app: &App, tx: mpsc::UnboundedSender) { }), r#type: ptype, credentials, - config: HashMap::default(), + config, credential_expires_at_ms: HashMap::default(), profile_workspace: String::new(), }), @@ -2059,20 +2078,16 @@ async fn refresh_providers(app: &mut App) { }; app.provider_count = providers.len(); - app.provider_entries = if app.providers_v2_enabled { - providers - .iter() - .cloned() - .map(|provider| app::ProviderV2Entry { - profile: profiles - .get(&(provider.profile_workspace.clone(), provider.r#type.clone())) - .cloned(), - provider, - }) - .collect() - } else { - Vec::new() - }; + app.provider_entries = providers + .iter() + .cloned() + .map(|provider| app::ProviderListEntry { + profile: profiles + .get(&(provider.profile_workspace.clone(), provider.r#type.clone())) + .cloned(), + provider, + }) + .collect(); app.provider_names = providers .iter() .map(|p| app::provider_name(p).to_string()) diff --git a/crates/openshell-tui/src/ui/create_provider.rs b/crates/openshell-tui/src/ui/create_provider.rs index a0896aa46a..cb39bff341 100644 --- a/crates/openshell-tui/src/ui/create_provider.rs +++ b/crates/openshell-tui/src/ui/create_provider.rs @@ -6,10 +6,14 @@ use ratatui::layout::{Constraint, Direction, Layout, Rect}; use ratatui::text::{Line, Span}; use ratatui::widgets::{Block, Borders, Clear, Padding, Paragraph}; -use crate::app::{App, CreateProviderPhase, ProviderKeyField}; +use crate::app::{App, CreateProviderPhase, ProviderKeyField, UpdateProviderField}; + +use indexmap::IndexMap; use super::centered_rect; +const MAX_VISIBLE_CONFIG: usize = 6; + /// Draw the create provider modal overlay. pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect) { let t = &app.theme; @@ -201,14 +205,18 @@ fn draw_enter_key( let has_warning = form.warning.is_some(); let warning_rows: u16 = if has_warning { 2 } else { 0 }; // warning + spacer + #[allow(clippy::cast_possible_truncation)] + let config_rows = (form.config.len() as u16).min(MAX_VISIBLE_CONFIG as u16) + 3; #[allow(clippy::cast_possible_truncation)] let content_height = if form.is_generic { - // type(1) + name(2) + spacer(1) + env_name(2) + value(2) + spacer(1) + submit(1) + status(1) + hint(1) - warning_rows + 12 + // type(1) + name(2) + spacer(1) + env_name(2) + value(2) + spacer(1) + // + config_section + spacer(1) + submit(1) + status(1) + hint(1) + warning_rows + 1 + 2 + 1 + 2 + 2 + 1 + config_rows + 1 + 1 + 1 + 1 } else { let num_creds = form.credentials.len().clamp(1, 8) as u16; - // type(1) + name(2) + spacer(1) + creds + spacer(1) + submit(1) + status(1) + hint(1) - warning_rows + 1 + 2 + 1 + num_creds + 1 + 1 + 1 + 1 + // type(1) + name(2) + spacer(1) + creds + spacer(1) + // + config_section + spacer(1) + submit(1) + status(1) + hint(1) + warning_rows + 1 + 2 + 1 + num_creds + 1 + config_rows + 1 + 1 + 1 + 1 }; let modal_height = (content_height + 4).min(area.height.saturating_sub(2)); let popup_area = centered_rect(modal_width, modal_height, area); @@ -241,6 +249,17 @@ fn draw_enter_key( let num_creds = form.credentials.len().clamp(1, 8) as u16; constraints.push(Constraint::Length(num_creds)); // credential rows } + + constraints.push(Constraint::Length(1)); // spacer before config + constraints.push(Constraint::Length(1)); // config keys label + #[allow(clippy::cast_possible_truncation)] + let num_config = form.config.len().min(MAX_VISIBLE_CONFIG) as u16; + if num_config > 0 { + constraints.push(Constraint::Length(num_config)); // existing config entries + } + constraints.push(Constraint::Length(1)); // config key input + constraints.push(Constraint::Length(1)); // config value input + constraints.push(Constraint::Length(1)); // spacer constraints.push(Constraint::Length(1)); // submit constraints.push(Constraint::Length(1)); // status @@ -361,7 +380,68 @@ fn draw_enter_key( } idx += 1; - // Spacer. + // Spacer before config. + idx += 1; + + // Config Keys label. + let config_section_focused = matches!( + form.key_field, + ProviderKeyField::ConfigList + | ProviderKeyField::ConfigKeyName + | ProviderKeyField::ConfigKeyValue + ); + let header_style = if config_section_focused { + t.accent_bold + } else { + t.muted + }; + frame.render_widget( + Paragraph::new(Line::from(Span::styled("Config Keys:", header_style))), + chunks[idx], + ); + idx += 1; + + // Existing config entries. + if !form.config.is_empty() { + let config_entry_active = form.key_field == ProviderKeyField::ConfigList; + render_config_entries( + frame, + &form.config, + form.config_cursor, + config_entry_active, + chunks[idx], + t, + ); + idx += 1; + } + + // Config key input. + let editing_key = form.key_field == ProviderKeyField::ConfigKeyName; + render_config_input_field( + frame, + "Key", + &form.config_key_input, + editing_key, + "key", + chunks[idx], + t, + ); + idx += 1; + + // Config value input. + let editing_val = form.key_field == ProviderKeyField::ConfigKeyValue; + render_config_input_field( + frame, + "Val", + &form.config_value_input, + editing_val, + "value", + chunks[idx], + t, + ); + idx += 1; + + // Spacer before submit. idx += 1; // Submit button. @@ -383,20 +463,7 @@ fn draw_enter_key( idx += 1; // Status. - if let Some(ref status) = form.status { - let style = if status.contains("required") - || status.contains("failed") - || status.contains("Failed") - { - t.status_err - } else { - t.status_ok - }; - frame.render_widget( - Paragraph::new(Line::from(Span::styled(format!(" {status}"), style))), - chunks[idx], - ); - } + render_status(frame, form.status.as_deref(), chunks[idx], t); idx += 1; // Hint. @@ -405,6 +472,8 @@ fn draw_enter_key( Span::styled(" Next ", t.muted), Span::styled("[S-Tab]", t.key_hint), Span::styled(" Prev ", t.muted), + Span::styled("[C-d]", t.key_hint), + Span::styled(" Delete ", t.muted), Span::styled("[Enter]", t.key_hint), Span::styled(" Submit ", t.muted), Span::styled("[Esc]", t.key_hint), @@ -669,9 +738,14 @@ pub fn draw_update(frame: &mut Frame<'_>, app: &App, area: Rect) { return; }; - let modal_width = 60u16.min(area.width.saturating_sub(4)); - // name(1) + type(1) + spacer(1) + key_label(1) + value(1) + cursor_hint(1) + spacer(1) + status(1) + hint(1) - let content_height = 9; + let modal_width = 64u16.min(area.width.saturating_sub(4)); + + #[allow(clippy::cast_possible_truncation)] + let num_config = form.config.len().min(MAX_VISIBLE_CONFIG) as u16; + let config_rows = num_config + 3; // existing entries + label(1) + key input(1) + value input(1) + // name(1) + type(1) + spacer(1) + key_label(1) + value(1) + spacer(1) + // + config_section + spacer(1) + submit(1) + status(1) + hint(1) + let content_height: u16 = 1 + 1 + 1 + 1 + 1 + 1 + config_rows + 1 + 1 + 1 + 1; let modal_height = (content_height + 4).min(area.height.saturating_sub(2)); let popup_area = centered_rect(modal_width, modal_height, area); @@ -686,91 +760,184 @@ pub fn draw_update(frame: &mut Frame<'_>, app: &App, area: Rect) { let inner = block.inner(popup_area); frame.render_widget(block, popup_area); + let mut constraints = vec![ + Constraint::Length(1), // name + Constraint::Length(1), // type + Constraint::Length(1), // spacer + Constraint::Length(1), // key label + Constraint::Length(1), // value input + Constraint::Length(1), // spacer before config + Constraint::Length(1), // config keys label + ]; + if num_config > 0 { + constraints.push(Constraint::Length(num_config)); // existing config entries + } + constraints.extend([ + Constraint::Length(1), // config key input + Constraint::Length(1), // config value input + Constraint::Length(1), // spacer before submit + Constraint::Length(1), // submit + Constraint::Length(1), // status + Constraint::Length(1), // hint + Constraint::Min(0), + ]); + let chunks = Layout::default() .direction(Direction::Vertical) - .constraints([ - Constraint::Length(1), // name - Constraint::Length(1), // type - Constraint::Length(1), // spacer - Constraint::Length(1), // key label - Constraint::Length(1), // value input - Constraint::Length(1), // cursor hint - Constraint::Length(1), // spacer - Constraint::Length(1), // status - Constraint::Length(1), // hint - Constraint::Min(0), - ]) + .constraints(constraints) .split(inner); + let mut idx = 0; + + // Name. frame.render_widget( Paragraph::new(Line::from(vec![ Span::styled("Name: ", t.muted), Span::styled(&form.provider_name, t.heading), ])), - chunks[0], + chunks[idx], ); + idx += 1; + // Type. frame.render_widget( Paragraph::new(Line::from(vec![ Span::styled("Type: ", t.muted), Span::styled(&form.provider_type, t.text), ])), - chunks[1], + chunks[idx], ); + idx += 1; + + // Spacer. + idx += 1; + // Credential key label. + let cred_focused = form.focus == UpdateProviderField::CredentialValue; let key_label = if form.credential_key.is_empty() { - "New value:" + "New value" } else { &form.credential_key }; frame.render_widget( Paragraph::new(Line::from(Span::styled( format!("{key_label}:"), - t.accent_bold, + if cred_focused { t.accent_bold } else { t.muted }, ))), - chunks[3], + chunks[idx], ); + idx += 1; - // Mask the input value as dots. + // Credential value input (masked). let masked: String = "*".repeat(form.new_value.len()); - frame.render_widget( - Paragraph::new(Line::from(vec![ - Span::styled(format!(" {masked}"), t.accent), - Span::styled("_", t.accent), - ])), - chunks[4], - ); + let cred_style = if cred_focused { t.accent } else { t.muted }; + let mut cred_spans = vec![Span::styled(format!(" {masked}"), cred_style)]; + if cred_focused { + cred_spans.push(Span::styled("_", t.accent)); + } + frame.render_widget(Paragraph::new(Line::from(cred_spans)), chunks[idx]); + idx += 1; + + // Spacer before config. + idx += 1; + // Config Keys label. + let config_section_focused = matches!( + form.focus, + UpdateProviderField::ConfigEntry + | UpdateProviderField::ConfigKey + | UpdateProviderField::ConfigValue + ); + let header_style = if config_section_focused { + t.accent_bold + } else { + t.muted + }; frame.render_widget( - Paragraph::new(Line::from(Span::styled( - " Type the new credential value", - t.muted, - ))), - chunks[5], + Paragraph::new(Line::from(Span::styled("Config Keys:", header_style))), + chunks[idx], ); + idx += 1; - if let Some(ref status) = form.status { - let style = if status.contains("required") - || status.contains("failed") - || status.contains("Failed") - { - t.status_err - } else { - t.status_ok - }; - frame.render_widget( - Paragraph::new(Line::from(Span::styled(format!(" {status}"), style))), - chunks[7], + // Existing config entries. + if !form.config.is_empty() { + let config_entry_active = form.focus == UpdateProviderField::ConfigEntry; + render_config_entries( + frame, + &form.config, + form.config_cursor, + config_entry_active, + chunks[idx], + t, ); + idx += 1; } + // Config key input. + let editing_key = form.focus == UpdateProviderField::ConfigKey; + render_config_input_field( + frame, + "Key", + &form.config_key_input, + editing_key, + "key", + chunks[idx], + t, + ); + idx += 1; + + // Config value input. + let editing_val = form.focus == UpdateProviderField::ConfigValue; + render_config_input_field( + frame, + "Val", + &form.config_value_input, + editing_val, + "value", + chunks[idx], + t, + ); + idx += 1; + + // Spacer before submit. + idx += 1; + + // Submit button. + let submit_focused = form.focus == UpdateProviderField::Submit; + let submit_style = if submit_focused { + t.accent_bold + } else { + t.muted + }; + let submit_label = if submit_focused { + " > Update Provider" + } else { + " Update Provider" + }; + frame.render_widget( + Paragraph::new(Line::from(Span::styled(submit_label, submit_style))), + chunks[idx], + ); + idx += 1; + + // Status. + render_status(frame, form.status.as_deref(), chunks[idx], t); + idx += 1; + + // Hint. let hint = Line::from(vec![ + Span::styled("[Tab]", t.key_hint), + Span::styled(" Next ", t.muted), + Span::styled("[S-Tab]", t.key_hint), + Span::styled(" Prev ", t.muted), + Span::styled("[C-d]", t.key_hint), + Span::styled(" Delete ", t.muted), Span::styled("[Enter]", t.key_hint), - Span::styled(" Update ", t.muted), + Span::styled(" Submit ", t.muted), Span::styled("[Esc]", t.key_hint), Span::styled(" Cancel", t.muted), ]); - frame.render_widget(Paragraph::new(hint), chunks[8]); + frame.render_widget(Paragraph::new(hint), chunks[idx]); } // --------------------------------------------------------------------------- @@ -813,3 +980,117 @@ fn draw_secret_field( }; frame.render_widget(Paragraph::new(display), chunks[1]); } + +fn render_config_entries( + frame: &mut Frame<'_>, + config: &IndexMap, + config_cursor: usize, + config_focused: bool, + area: Rect, + theme: &crate::theme::Theme, +) { + let total = config.len(); + let scroll_offset = if total > MAX_VISIBLE_CONFIG { + config_cursor + .saturating_sub(MAX_VISIBLE_CONFIG - 2_usize) + .min(total.saturating_sub(MAX_VISIBLE_CONFIG)) + } else { + 0_usize + }; + let take_count = MAX_VISIBLE_CONFIG.min(total.saturating_sub(scroll_offset)); + let overflow_below = scroll_offset + take_count < total; + let take_count = if overflow_below { + take_count.saturating_sub(1_usize) + } else { + take_count + }; + + let mut config_lines: Vec> = config + .iter() + .enumerate() + .skip(scroll_offset) + .take(take_count) + .map(|(i, (key, value))| { + let is_selected = config_focused && i == config_cursor; + let style = if is_selected { + theme.accent_bold + } else { + theme.text + }; + Line::from(vec![ + Span::styled(format!(" {key}="), style), + Span::styled( + value.as_str(), + if is_selected { + theme.accent + } else { + theme.muted + }, + ), + ]) + }) + .collect(); + if overflow_below { + let remaining = total - scroll_offset - take_count; + config_lines.push(Line::from(Span::styled( + format!(" \u{2026}and {remaining} more"), + theme.muted, + ))); + } + frame.render_widget(Paragraph::new(config_lines), area); +} + +fn render_config_input_field( + frame: &mut Frame<'_>, + label: &str, + input: &str, + editing: bool, + placeholder: &str, + area: Rect, + theme: &crate::theme::Theme, +) { + let t = theme; + let display = if input.is_empty() { + if editing { + "_".to_string() + } else { + placeholder.to_string() + } + } else { + let mut s = input.to_string(); + if editing { + s.push('_'); + } + s + }; + frame.render_widget( + Paragraph::new(Line::from(vec![ + Span::styled(format!(" {label}: "), t.muted), + Span::styled(display, if editing { t.accent } else { t.muted }), + ])), + area, + ); +} + +fn render_status( + frame: &mut Frame<'_>, + status: Option<&str>, + area: Rect, + theme: &crate::theme::Theme, +) { + let t = theme; + if let Some(status) = status { + let style = if status.contains("required") + || status.contains("failed") + || status.contains("Failed") + { + t.status_err + } else { + t.status_ok + }; + frame.render_widget( + Paragraph::new(Line::from(Span::styled(format!(" {status}"), style))), + area, + ); + } +} diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml index 9f6926142d..4d82964189 100644 --- a/deploy/docker/docker-compose.yml +++ b/deploy/docker/docker-compose.yml @@ -19,9 +19,9 @@ # ANTHROPIC_API_KEY=sk-ant-... \ # openshell provider create --type anthropic --from-existing # -# 4. Create a sandboxed agent — Claude Code or OpenClaw: +# 4. Create a sandboxed agent — Claude Code: # openshell sandbox create -- claude -# openshell sandbox create --from openclaw +# For OpenClaw, use NemoClaw: https://github.com/NVIDIA/NemoClaw # # Sandbox containers are managed by the gateway, not by this Compose file. # Each `openshell sandbox create` call launches a fresh container; the gateway diff --git a/docs/brand/assets/favicon-16.png b/docs/brand/assets/favicon-16.png new file mode 100644 index 0000000000..afc0171a0c Binary files /dev/null and b/docs/brand/assets/favicon-16.png differ diff --git a/docs/brand/assets/favicon-180.png b/docs/brand/assets/favicon-180.png new file mode 100644 index 0000000000..d1f9f14c8e Binary files /dev/null and b/docs/brand/assets/favicon-180.png differ diff --git a/docs/brand/assets/favicon-32.png b/docs/brand/assets/favicon-32.png new file mode 100644 index 0000000000..7dcdd5fed7 Binary files /dev/null and b/docs/brand/assets/favicon-32.png differ diff --git a/docs/brand/assets/favicon-48.png b/docs/brand/assets/favicon-48.png new file mode 100644 index 0000000000..a8fd4dd5e2 Binary files /dev/null and b/docs/brand/assets/favicon-48.png differ diff --git a/docs/brand/assets/favicon-512.png b/docs/brand/assets/favicon-512.png new file mode 100644 index 0000000000..03b24809d6 Binary files /dev/null and b/docs/brand/assets/favicon-512.png differ diff --git a/docs/brand/assets/favicon.svg b/docs/brand/assets/favicon.svg new file mode 100644 index 0000000000..0dd7272ca9 --- /dev/null +++ b/docs/brand/assets/favicon.svg @@ -0,0 +1,7 @@ + + + + + + + \ No newline at end of file diff --git a/docs/brand/assets/openshell-avatar-1024.png b/docs/brand/assets/openshell-avatar-1024.png new file mode 100644 index 0000000000..dc21bb9d35 Binary files /dev/null and b/docs/brand/assets/openshell-avatar-1024.png differ diff --git a/docs/brand/assets/openshell-avatar-512.png b/docs/brand/assets/openshell-avatar-512.png new file mode 100644 index 0000000000..19f8384138 Binary files /dev/null and b/docs/brand/assets/openshell-avatar-512.png differ diff --git a/docs/brand/assets/openshell-avatar-dark-1024.png b/docs/brand/assets/openshell-avatar-dark-1024.png new file mode 100644 index 0000000000..ed2dde158c Binary files /dev/null and b/docs/brand/assets/openshell-avatar-dark-1024.png differ diff --git a/docs/brand/assets/openshell-avatar-dark-512.png b/docs/brand/assets/openshell-avatar-dark-512.png new file mode 100644 index 0000000000..a08b0638ab Binary files /dev/null and b/docs/brand/assets/openshell-avatar-dark-512.png differ diff --git a/docs/brand/assets/openshell-lockup-horizontal-dark.png b/docs/brand/assets/openshell-lockup-horizontal-dark.png new file mode 100644 index 0000000000..05b2c87ba4 Binary files /dev/null and b/docs/brand/assets/openshell-lockup-horizontal-dark.png differ diff --git a/docs/brand/assets/openshell-lockup-horizontal-reversed.svg b/docs/brand/assets/openshell-lockup-horizontal-reversed.svg new file mode 100644 index 0000000000..b45bd439cc --- /dev/null +++ b/docs/brand/assets/openshell-lockup-horizontal-reversed.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/brand/assets/openshell-lockup-horizontal.png b/docs/brand/assets/openshell-lockup-horizontal.png new file mode 100644 index 0000000000..2096614159 Binary files /dev/null and b/docs/brand/assets/openshell-lockup-horizontal.png differ diff --git a/docs/brand/assets/openshell-lockup-horizontal.svg b/docs/brand/assets/openshell-lockup-horizontal.svg new file mode 100644 index 0000000000..daec178c99 --- /dev/null +++ b/docs/brand/assets/openshell-lockup-horizontal.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/brand/assets/openshell-lockup-stacked-dark.png b/docs/brand/assets/openshell-lockup-stacked-dark.png new file mode 100644 index 0000000000..16e9268f98 Binary files /dev/null and b/docs/brand/assets/openshell-lockup-stacked-dark.png differ diff --git a/docs/brand/assets/openshell-lockup-stacked-reversed.svg b/docs/brand/assets/openshell-lockup-stacked-reversed.svg new file mode 100644 index 0000000000..1ad4511d3a --- /dev/null +++ b/docs/brand/assets/openshell-lockup-stacked-reversed.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/brand/assets/openshell-lockup-stacked.png b/docs/brand/assets/openshell-lockup-stacked.png new file mode 100644 index 0000000000..6e7de90985 Binary files /dev/null and b/docs/brand/assets/openshell-lockup-stacked.png differ diff --git a/docs/brand/assets/openshell-lockup-stacked.svg b/docs/brand/assets/openshell-lockup-stacked.svg new file mode 100644 index 0000000000..416346c41b --- /dev/null +++ b/docs/brand/assets/openshell-lockup-stacked.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/brand/assets/openshell-mark-mono-black.svg b/docs/brand/assets/openshell-mark-mono-black.svg new file mode 100644 index 0000000000..f43412d479 --- /dev/null +++ b/docs/brand/assets/openshell-mark-mono-black.svg @@ -0,0 +1,12 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/brand/assets/openshell-mark-mono-white.svg b/docs/brand/assets/openshell-mark-mono-white.svg new file mode 100644 index 0000000000..c1b5ad2ff3 --- /dev/null +++ b/docs/brand/assets/openshell-mark-mono-white.svg @@ -0,0 +1,12 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/brand/assets/openshell-mark-reversed.svg b/docs/brand/assets/openshell-mark-reversed.svg new file mode 100644 index 0000000000..29111f9cd0 --- /dev/null +++ b/docs/brand/assets/openshell-mark-reversed.svg @@ -0,0 +1,7 @@ + + + + + + + \ No newline at end of file diff --git a/docs/brand/assets/openshell-mark.svg b/docs/brand/assets/openshell-mark.svg new file mode 100644 index 0000000000..0dd7272ca9 --- /dev/null +++ b/docs/brand/assets/openshell-mark.svg @@ -0,0 +1,7 @@ + + + + + + + \ No newline at end of file diff --git a/docs/extensibility/gateway-interceptors.mdx b/docs/extensibility/gateway-interceptors.mdx new file mode 100644 index 0000000000..9fe50ef5a7 --- /dev/null +++ b/docs/extensibility/gateway-interceptors.mdx @@ -0,0 +1,167 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: "Gateway Interceptors" +sidebar-title: "Gateway Interceptors" +description: "Extend OpenShell gateway operations with deployment-specific governance and business logic." +keywords: "Generative AI, Cybersecurity, AI Agents, Gateway Interceptors, Extensibility, Governance" +--- +Gateway interceptors let operators add deployment-specific governance to OpenShell control-plane operations without modifying the gateway. An external gRPC service can modify or validate selected API writes before the gateway handles them, then observe successful responses after commit. + +See the [governance interceptor example](https://github.com/NVIDIA/OpenShell/tree/main/examples/governance-interceptor) for a complete service that vends provider profiles, applies a signed policy to new sandboxes, and rejects attempts to weaken that policy. + +## Choose Gateway Interceptors + +Use a gateway interceptor when an external service needs to govern gateway API operations. For example, an interceptor can: + +- Apply an approved policy to every new sandbox. +- Reject provider or policy changes that violate organizational rules. +- Enforce tenant quotas or naming conventions. +- Observe committed operations for an audit or inventory service. +- Vend an authoritative or composed provider profile catalog. + +Gateway interceptors do not replace gateway persistence, authentication, authorization, policy safety checks, or driver validation. The gateway remains the system of record and validates an operation after interceptor modification. + +## How Interception Works + +The gateway runs interceptors after authentication and before dispatching a request to its handler: + +`authenticate → decode and omit secrets → modify_operation → validate → gateway handler → post_commit` + +| Phase | Input | Capabilities | +| ------------------ | ------------------------------------------- | ------------------------------------------------ | +| `modify_operation` | Proposed request | Allow, deny, or return RFC 6902 JSON patches. | +| `validate` | Modified request and optional current state | Allow or deny. | +| `post_commit` | Successful gateway response | Observe the response and attach log annotations. | + +Only explicitly allowlisted unary mutation RPCs are interceptable. The gateway converts an operation to its protobuf JSON representation before evaluation. It applies patches atomically, validates the result against the RPC's protobuf schema, and converts the operation back to protobuf before the handler receives it. + +The gateway runs built-in operation and driver validation after `modify_operation`. A patched operation cannot bypass gateway-owned invariants. + +`post_commit` is observational. It cannot deny or modify an operation that the gateway has already committed. + +## Implement an Interceptor Service + +An interceptor implements the `openshell.gateway_interceptor.v1.GatewayInterceptor` gRPC service defined in [`proto/gateway_interceptor.proto`](https://github.com/NVIDIA/OpenShell/blob/main/proto/gateway_interceptor.proto): + +- `Describe` declares the service's bindings and capabilities. +- `Evaluate` handles one selected operation phase. +- `SnapshotProviderProfiles` optionally returns a provider profile catalog. + +Each `InterceptorEvaluation` identifies the configured interceptor, manifest binding, public OpenShell service and method, authenticated principal, and active phase. The phase determines whether the payload contains a proposed operation, optional current state, or committed response. + +The interceptor returns an `InterceptorResult` with an allow or deny decision, an optional denial status and reason, JSON patches during `modify_operation`, and non-secret log annotations. + +## Declare and Authorize Bindings + +The interceptor declares bindings in its `InterceptorManifest` returned by `Describe`. Each binding selects one public OpenShell RPC and one or more phases. The gateway validates these declarations at startup against its compiled protobuf descriptors and explicit interceptable RPC allowlist. + +The operator chooses how the manifest and gateway configuration combine with `binding_policy`: + +| Policy | Behavior | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| `dynamic` | Enables valid manifest bindings. Gateway configuration may narrow or disable them. This is the compatibility default and emits a startup warning. | +| `allowlist` | Enables only operator-configured RPCs and phases. Extra manifest bindings are ignored and logged. | +| `exact` | Requires the configured RPCs and phases to match the manifest exactly. | + +Use `allowlist` or `exact` when interceptor authority is part of a security boundary. These modes select bindings by public RPC rather than manifest binding ID, so renaming a binding does not change its authority. + +## Register an Interceptor Service + +Start the interceptor before the gateway, then register it in gateway TOML: + +```toml +[[openshell.gateway.interceptors]] +name = "policy-governance" +grpc_endpoint = "http://127.0.0.1:18081" +order = 10 +failure_policy = "fail_closed" +binding_policy = "allowlist" +timeout = "500ms" +max_response_bytes = 1048576 +max_patches = 32 + +[[openshell.gateway.interceptors.bindings]] +rpc = "openshell.v1.OpenShell/CreateSandbox" +phases = ["modify_operation", "validate"] + +[[openshell.gateway.interceptors.bindings]] +rpc = "openshell.v1.OpenShell/UpdateConfig" +phases = ["validate"] +``` + +The gateway supports `http://`, `https://`, and `unix://` interceptor endpoints. It calls `Describe` and builds an immutable execution plan during startup. An unavailable service, invalid manifest, or unauthorized configured binding prevents the gateway from starting. + +Registration is static. Restart the gateway after adding, removing, or changing an interceptor. See [Gateway Configuration](/reference/gateway-config#gateway-interceptors) for the complete field reference. + +## Select RPCs and Phases + +Start with the invariant the interceptor must preserve, then identify every gateway RPC that can establish or weaken it. For example, an interceptor that owns sandbox policy can use `modify_operation` on `CreateSandbox` to apply an approved initial policy and `validate` on `UpdateConfig` to reject unauthorized changes. + +The gateway maintains the canonical [interceptable route allowlist](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-gateway-interceptors/src/routes.rs). Read the interceptor manifest and gateway startup diagnostics when selecting bindings. Unknown, streaming, read-only, and non-allowlisted RPCs cannot be intercepted. + +Choose a phase by intent: + +- Use `modify_operation` to apply defaults or controlled changes. +- Use `validate` to enforce a rule without changing the request. +- Use `post_commit` to notify or audit an external system after success. + +## Mutate Operations Safely + +Operations and committed responses use protobuf JSON field names and shapes. Only `modify_operation` accepts RFC 6902 JSON patches. + +The gateway applies all patches returned by one binding as an atomic candidate. It then encodes that candidate as the RPC's protobuf request type and decodes it back to canonical protobuf JSON. If any patch or the resulting operation is invalid, the gateway discards the complete candidate and applies the binding's failure policy. Later bindings see only schema-valid operations that the handler can receive. + +Fields marked secret in the protobuf schema are recursively omitted from interceptor requests and committed responses. An interceptor cannot patch an omitted field, use one as a patch source, or replace a containing object. Gateway handlers retain the complete operation and continue to receive secret fields that were omitted from the interceptor view. + +## Configure Failure Behavior + +Failure policy controls what happens when the gateway cannot obtain or apply a valid result. Failures include timeouts, transport errors, invalid responses, response-size violations, invalid phase behavior, and patch-limit violations. + +| Policy | Behavior | +| ------------- | -------------------------------------------------------------------------------------------------- | +| `fail_closed` | Rejects the API operation before handler dispatch. | +| `fail_open` | Skips the failed result, continues with the previous valid operation, and emits warning telemetry. | + +A valid deny result during `modify_operation` or `validate` always rejects the operation. It is not an interceptor failure and does not follow the failure policy. + +Bindings that include `post_commit` must resolve to `fail_open`. The gateway rejects fail-closed post-commit configuration at startup because an observer cannot revoke an already committed response. A post-commit observation or evaluation failure is logged and counted without replacing the successful gateway response. + +Use `fail_open` only when bypassing an unavailable or invalid interceptor preserves the intended governance boundary. + +## Vend Provider Profiles + +An interceptor can advertise `provider_profiles = true` in its manifest and implement `SnapshotProviderProfiles`. The RPC returns a `ProviderProfileSnapshot`. Add that interceptor to `provider_profile_sources` to include its profiles in the gateway's effective catalog. + +Select only the interceptor to make its catalog authoritative: + +```toml +[openshell.gateway] +provider_profile_sources = [ + { type = "interceptor", name = "provider-governance" }, +] +``` + +Include `{ type = "builtin" }` or `{ type = "user" }` entries to compose interceptor profiles with the built-in or user-managed sources. Duplicate normalized profile IDs fail instead of overriding one source with another. + +The gateway validates snapshot structure and provider profile semantics. It treats the configured interceptor as a trusted source and does not verify interceptor-defined signature, hash, or key annotations. + +## Operate and Observe Interceptors + +Plan interceptor deployment around these boundaries: + +- Start every configured service before the gateway. +- Restart the gateway after changing registrations or bindings. +- Keep fail-closed services available whenever the gateway accepts writes. +- Treat the endpoint and its operator configuration as part of the gateway trust boundary. +- Do not include secrets in denial reasons or log annotations. + +The gateway emits structured evaluation logs containing the interceptor name, binding ID, RPC, phase, decision, patch count, and interceptor-provided log annotations. Metrics record evaluation decisions, latency, patch application, fail-open and fail-closed outcomes, and post-commit observation failures. + +## Current Limitations + +- Only explicitly allowlisted unary write RPCs are interceptable. New gateway RPCs are non-interceptable until added to the allowlist. +- `current_state` is available only in the `validate` contract. The gateway does not yet populate it with method-specific state. +- Registration changes require a gateway restart. +- Custom TLS roots, client authentication, service health checks, and runtime registration are not available. +- Interceptors cannot receive or mutate protobuf fields marked secret. diff --git a/docs/get-started/tutorials/docker-compose.mdx b/docs/get-started/tutorials/docker-compose.mdx index 645b5ddc98..4037c02995 100644 --- a/docs/get-started/tutorials/docker-compose.mdx +++ b/docs/get-started/tutorials/docker-compose.mdx @@ -4,7 +4,7 @@ title: "Run the Gateway with Docker Compose" sidebar-title: "Docker Compose Setup" slug: "get-started/tutorials/docker-compose" -description: "Run the OpenShell gateway as a Docker Compose service and create agent sandboxes including OpenClaw." +description: "Run the OpenShell gateway as a Docker Compose service and create agent sandboxes." keywords: "Generative AI, Docker Compose, Gateway, Sandbox, OpenClaw, Docker, Installation" --- @@ -15,7 +15,7 @@ After completing this tutorial you have: - An OpenShell gateway running as a Compose service. - The `openshell` CLI registered against that gateway. - An AI provider configured with your API key. -- A running OpenClaw sandbox. +- A running agent sandbox. ## Prerequisites @@ -134,9 +134,6 @@ Sandbox images are pulled automatically on first use, but the initial pull can t ```shell # Base image — includes Claude Code, OpenCode, Codex, and Copilot docker pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest - -# OpenClaw image -docker pull ghcr.io/nvidia/openshell-community/sandboxes/openclaw:latest ``` ## Create a sandbox @@ -144,11 +141,9 @@ docker pull ghcr.io/nvidia/openshell-community/sandboxes/openclaw:latest -```shell -openshell sandbox create --from openclaw -``` +OpenClaw runs inside OpenShell through [NemoClaw](https://github.com/NVIDIA/NemoClaw), which manages the sandbox image, inference routing, and security policies. -OpenClaw launches directly. The first run pulls the image if it is not cached. +Follow the [NemoClaw Quickstart](https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart/) to set up an OpenClaw sandbox with managed inference. diff --git a/docs/observability/logging.mdx b/docs/observability/logging.mdx index 7a13ffea0b..1ab0b5bfed 100644 --- a/docs/observability/logging.mdx +++ b/docs/observability/logging.mdx @@ -192,10 +192,13 @@ A denied CONNECT returns `403 Forbidden`: ```json { "error": "policy_denied", - "detail": "CONNECT api.example.com:443 not permitted by policy" + "detail": "CONNECT api.example.com:443 not permitted by policy", + "reason": "binary '/usr/bin/node' not allowed in policy 'allow_api' (ancestors: [/usr/local/bin/claude])" } ``` +The `reason` field is included when the policy engine provides a specific denial reason (for example, which binary or rule caused the rejection). It is omitted when no additional detail is available. + An upstream that the proxy cannot reach returns `502 Bad Gateway`: ```json @@ -205,7 +208,7 @@ An upstream that the proxy cannot reach returns `502 Bad Gateway`: } ``` -The `error` field is a short machine-readable code (`policy_denied`, `middleware_denied`, `middleware_failed`, `ssrf_denied`, `upstream_unreachable`). The `detail` field is a human-readable explanation suitable for display in an agent transcript. +The `error` field is a short machine-readable code (`policy_denied`, `middleware_denied`, `middleware_failed`, `ssrf_denied`, `upstream_unreachable`). The `detail` field is a human-readable explanation suitable for display in an agent transcript. The optional `reason` field, when present, provides the specific denial cause from the policy engine (for example, which binary was not allowed or which rule was missing). For L7 REST policy denials, the body also includes structured policy fields such as `method`, `path`, `rule_missing`, and `next_steps`. When policy advisor is enabled, it also includes `agent_guidance`, a short plain-language instruction telling the agent to read `/etc/openshell/skills/policy_advisor.md`, propose the narrowest rule through `http://policy.local/v1/proposals`, wait for `policy_reloaded: true`, and retry. A middleware denial instead identifies the policy-local config in `middleware` and can include a validated `reason_code`. A fail-closed runtime failure uses `middleware_failed` with platform-owned text. Both middleware responses omit `rule_missing`, `next_steps`, and `agent_guidance` because no policy rule is missing. diff --git a/docs/reference/gateway-auth.mdx b/docs/reference/gateway-auth.mdx index 94aca659ee..b278f0f403 100644 --- a/docs/reference/gateway-auth.mdx +++ b/docs/reference/gateway-auth.mdx @@ -20,6 +20,12 @@ When any CLI command needs to talk to the gateway, it resolves the target throug The CLI loads gateway metadata from `~/.config/openshell/gateways//metadata.json` first and falls back to `/etc/openshell/gateways//metadata.json` when no user entry exists. +## Authentication Status + +`openshell status` reports gateway reachability and authentication separately. The public health RPC determines whether the gateway is connected and supplies its version. The CLI then calls the existing protected gateway-info capability query to verify the configured credentials. An authorization denial still confirms that the gateway authenticated the caller. + +For example, an expired bearer token can produce `Status: Connected` and `Authentication: Failed`. This means the gateway is healthy but protected commands cannot use the current credentials. Run `openshell gateway login ` to re-authenticate. When a gateway predates the gateway-info capability query, the CLI reports `Authentication: Unverified` rather than treating a successful public health check as proof of authentication. + ## Authentication Modes The CLI uses one of these authentication modes depending on the gateway's configuration. @@ -119,7 +125,7 @@ When you register or log in to an OIDC gateway, the CLI uses the Authorization C The connection flow: 1. The CLI loads the stored OIDC token bundle. -2. If the access token is expired and a refresh token is available, the CLI refreshes it. +2. If the access token is expired and a refresh token is available, the CLI refreshes it with the OIDC scopes saved in the gateway metadata. 3. The CLI connects to the gateway and attaches `authorization: Bearer ` metadata to each gRPC request. 4. The gateway validates the JWT signature, issuer, audience, expiration, and key ID against the issuer's JWKS. 5. The gateway extracts roles and optional scopes from the configured claim paths. diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 3dde7643d5..75ac6f7382 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -362,6 +362,8 @@ compute_drivers = ["podman"] [openshell.drivers.podman] # Rootless socket path. For root Podman use /run/podman/podman.sock. +# Omit to auto-detect: the driver probes for a responsive Podman socket and +# fails to start if none respond. socket_path = "/run/user/1000/podman/podman.sock" default_image = "ghcr.io/nvidia/openshell/sandbox:latest" image_pull_policy = "missing" # always | missing | never | newer @@ -389,6 +391,66 @@ sandbox_pids_limit = 2048 # but increase process churn (each check spawns a conmon subprocess). # Set to 0 to disable health checks entirely. Default: 10. health_check_interval_secs = 10 +# Corporate forward proxy for sandbox egress. When set, the in-container +# supervisor chains policy-approved TLS tunnels through this proxy with HTTP +# CONNECT instead of dialing destinations directly. Plain-HTTP requests are +# not proxied and always dial the destination directly. Only http:// proxy +# URLs in explicit http://host:port form are supported: the scheme and port +# are both required, and a URL carrying a path, query, or fragment is +# rejected rather than silently truncated. +# NO_PROXY entries (hostnames, domain suffixes, IPs, CIDRs, each with an +# optional :port qualifier) are dialed directly. A port-qualified entry only +# bypasses that destination port. IP and CIDR entries also match hostnames +# through their validated DNS resolution; such a match dials directly only +# the resolved addresses inside the entry. This is an operator-owned egress +# boundary: +# sandbox and template environment cannot override it, and the conventional +# HTTPS_PROXY/HTTP_PROXY/NO_PROXY variables a sandbox sets do not affect it. +# +# The CONNECT request sent to the proxy targets a validated resolved IP, +# not the hostname, so the proxy performs no DNS resolution of its own and +# the tunnel stays bound to the address that passed the sandbox's SSRF and +# allowed_ips validation. The hostname still travels inside the tunnel (TLS +# SNI, application Host), so destination servers behave normally. In +# split-horizon networks, point the gateway host at the corporate resolver +# so internal names validate to their internal addresses. If the proxy's +# ACLs filter on hostnames and reject IP CONNECT targets, set +# proxy_connect_by_hostname = true as a last resort: the proxy then +# resolves the name itself, so a name that resolves differently at the +# proxy (split-horizon DNS, rebinding) can reach destinations the sandbox +# policy never approved, and the proxy's own ACLs become the effective +# egress control for proxied TLS. +# +# Configuration is fail-closed: an invalid proxy URL is rejected at gateway +# startup, setting no_proxy, proxy_auth_file, or proxy_connect_by_hostname +# without a proxy URL is rejected as well, and a set-but-invalid value +# reaching a sandbox (for example an unreadable or malformed auth file) is +# fatal to that sandbox's supervisor instead of silently falling back to +# direct or unauthenticated egress. +# +# Credentials must NOT be embedded in the URL (an inline user:pass@ is +# rejected at startup, since it would be stored here and exposed in container +# metadata). Instead point proxy_auth_file at a file containing "user:pass"; +# the gateway delivers it to the supervisor through a root-only secret mount. +# The credential must use the user:pass form (non-empty user, no control +# characters); the same validation runs at sandbox-create time and in the +# supervisor, so a credential accepted here is never rejected in-container. +# Keep gateway.toml and the auth file owner-readable only (mode 0600). +# +# WARNING: the supervisor sends the credential as a Proxy-Authorization: +# Basic header over the plain-TCP connection to the http:// proxy. Basic +# auth is base64, not encryption: anyone on the network path between the +# sandbox host and the proxy can recover the credential. Because of that, +# proxy_auth_file requires the explicit acknowledgement +# proxy_auth_allow_insecure = true; without it the configuration is +# rejected at gateway startup. Only opt in when the path to the proxy is a +# trusted network segment. +# https_proxy = "http://proxy.corp.com:8080" +# no_proxy = "*.svc.cluster.local,10.0.0.0/8" +# proxy_auth_file = "/etc/openshell/secrets/proxy-auth" +# proxy_auth_allow_insecure = true +# Last resort for hostname-filtering proxy ACLs; see the warning above. +# proxy_connect_by_hostname = true ``` ### MicroVM diff --git a/docs/reference/policy-schema.mdx b/docs/reference/policy-schema.mdx index 721f16109d..d585517d13 100644 --- a/docs/reference/policy-schema.mdx +++ b/docs/reference/policy-schema.mdx @@ -179,6 +179,18 @@ Each endpoint defines a reachable destination and optional inspection rules. | `mcp.allow_all_known_mcp_methods` | bool | No | Defaults to `false`. When `true`, enables the endpoint MCP method profile: omitted `rules` allow all MCP-family methods and all tools before `deny_rules`, and omitted rule `method` uses that profile. When unset or `false`, explicit MCP method rules are required; rules with `tool` or `params.name` must set `method: tools/call`. | | `json_rpc` | object | No | JSON-RPC endpoint options. For `protocol: json-rpc`, `json_rpc.max_body_bytes` sets the maximum JSON-RPC-over-HTTP request body bytes buffered for inspection. Defaults to `65536`. | +**Validation constraints:** + +- `access` and `rules` are mutually exclusive; setting both is rejected. +- When `protocol` is set, at least one of `access` or `rules` is required for `rest`, `websocket`, `graphql`, and `sql`. +- `mcp` and `json-rpc` reject `access` presets; use explicit `rules`. +- `json-rpc` requires explicit `rules` with `allow.method`. +- `mcp` requires `rules` unless `mcp.allow_all_known_mcp_methods: true`. +- `deny_rules` require `protocol`. For non-MCP protocols, `deny_rules` also require `rules` or `access` to define the base allow set. MCP `deny_rules` may omit both when `mcp.allow_all_known_mcp_methods: true` supplies the base allow set. +- `rules: []` (empty list) is rejected; use `access: full` or remove `rules`. +- Non-empty `rules` must contain at least one effective allow clause; rules where every entry lacks an allow are rejected as deny-all. +- `deny_rules: []` (empty list) is rejected; remove it if no denials are needed. + Credential rewrite recognizes the canonical `openshell:resolve:env:KEY` placeholder form and whole-token provider-shaped aliases such as `provider-OPENSHELL-RESOLVE-ENV-API_TOKEN` when the referenced environment key exists in the configured provider credentials. #### Access Levels diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index a0167ae72a..e0eec81285 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -176,12 +176,14 @@ checks do not make host bind mounts safe. [Podman](https://podman.io/)-backed sandboxes run as rootless containers on the gateway host. Use Podman for Linux workstation workflows that avoid a rootful Docker daemon. -The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. +The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. When `socket_path` is not set, the driver probes for a responsive Podman socket and fails to start if none respond. For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md). Select Podman with `compute_drivers = ["podman"]` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `sandbox_ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`. +For proxy-required networks, the Podman driver also accepts the corporate egress proxy keys `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, and `proxy_connect_by_hostname`. The supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior. + On macOS with `podman machine`, the driver uses gvproxy's host-loopback IP, `192.168.127.254`, for sandbox host aliases by default. Set `host_gateway_ip` only when your Podman machine uses a non-standard host-loopback address. On Linux, an empty `host_gateway_ip` keeps Podman's `host-gateway` resolver behavior. ### Podman Driver Config Mounts diff --git a/docs/sandboxes/manage-gateways.mdx b/docs/sandboxes/manage-gateways.mdx index 443a65a750..ff1d0ccd3e 100644 --- a/docs/sandboxes/manage-gateways.mdx +++ b/docs/sandboxes/manage-gateways.mdx @@ -113,13 +113,25 @@ openshell status -g staging ## Inspect Gateway Status -Use `openshell status` for a live gateway check. It verifies that the CLI can -reach the running gateway, then reports health and the gateway version: +Use `openshell status` for a live gateway check. It reports gateway +reachability, authentication, and the gateway version independently: ```shell openshell status ``` +`Status: Connected` means the public health endpoint responded. +`Authentication: Authenticated` confirms the configured credentials also +passed the gateway authentication layer. If authentication fails while status +remains connected, run `openshell gateway login ` to refresh the stored +credentials. + +For automation or scripting, use `--output json` or `--output yaml` to get machine-readable output: + +```shell +openshell status --output json +``` + Use `openshell gateway info` when you need elevated runtime details such as initialized compute drivers and driver-reported capability versions: diff --git a/docs/sandboxes/manage-sandboxes.mdx b/docs/sandboxes/manage-sandboxes.mdx index 05418f8f7d..f3f36ecc9e 100644 --- a/docs/sandboxes/manage-sandboxes.mdx +++ b/docs/sandboxes/manage-sandboxes.mdx @@ -20,6 +20,12 @@ Create a sandbox with a single command. For example, to create a sandbox with Cl openshell sandbox create -- claude ``` +For automation, use `--output json` or `--output yaml` to get machine-readable sandbox metadata after creation: + +```shell +openshell sandbox create --output json +``` + Every sandbox requires a gateway. Register or select one before running sandbox commands: ```shell @@ -309,6 +315,12 @@ Get detailed information about a specific sandbox. The output lists **Policy sou openshell sandbox get my-sandbox ``` +For automation, use `--output json` or `--output yaml` to get machine-readable sandbox details: + +```shell +openshell sandbox get my-sandbox --output json +``` + Print only that policy YAML for scripting (same effective policy, no metadata): ```shell diff --git a/docs/sandboxes/policy-advisor.mdx b/docs/sandboxes/policy-advisor.mdx index 6139aafb38..76c5e1f45b 100644 --- a/docs/sandboxes/policy-advisor.mdx +++ b/docs/sandboxes/policy-advisor.mdx @@ -10,7 +10,7 @@ position: 6 Policy advisor lets a running sandboxed agent ask for a narrow network policy change after OpenShell denies a request. The agent submits a draft through `policy.local`, a developer approves or rejects it from outside the sandbox, and approved network policy hot-reloads into the same sandbox. -Policy advisor preserves OpenShell's default-deny posture. The structured rule is the approval contract, and the agent's rationale is supporting context. By default every proposal lands in the draft inbox for human review. Opt-in [auto mode](#approval-modes) lets the gateway approve provably safe proposals — those whose [prover delta](#what-auto-approval-checks) is empty — without a reviewer in the loop; proposals with any prover finding still require human approval. +Policy advisor preserves OpenShell's default-deny posture. The structured rule is the approval contract, and the agent's rationale is supporting context. By default every accepted proposal lands in the draft inbox for human review. Opt-in [auto mode](#approval-modes) approves a proposal without a reviewer only when its [prover delta](#what-auto-approval-checks) is empty and the current draft rule produces no security notes. A prover finding or security note keeps the proposal pending for human review. ## Enable Policy Advisor @@ -49,16 +49,16 @@ Set the value before creating a sandbox when you want the first denied request t ## Approval Modes -Every proposal — mechanistic or agent-authored — is routed through the [policy prover](#what-auto-approval-checks). The `proposal_approval_mode` setting decides what happens when the prover finds nothing to flag. +Every proposal, mechanistic or agent-authored, is routed through the [policy prover](#what-auto-approval-checks). The gateway also recalculates security notes from the current draft rule before auto-approval. The `proposal_approval_mode` setting decides whether proposals that pass both checks require human review. | Mode | When unset / `manual` | `auto` | |---|---|---| -| Empty prover delta | Lands in the draft inbox for human review. | Approved automatically; the sandbox hot-reloads the new rule and the agent retries. | -| Any prover finding | Lands in the draft inbox. | Lands in the draft inbox — auto-approval is gated on an empty delta. | +| Empty prover delta and no security notes | Lands in the draft inbox for human review. | Approved automatically. The sandbox hot-reloads the new rule and the agent retries. | +| Any prover finding or security note | Lands in the draft inbox. | Remains pending for human review. | `manual` is the default. Auto mode is an explicit opt-in; OpenShell's default-deny posture is preserved unless you choose otherwise. -Enable auto mode at gateway scope when you want every sandbox on this gateway to auto-approve safe proposals: +Enable auto mode at gateway scope when you want every sandbox on this gateway to auto-approve eligible proposals: ```shell openshell settings set --global \ @@ -103,7 +103,7 @@ The loop has seven steps: 3. The agent reads the policy advisor skill, inspects the current policy, and optionally reads recent denial log lines. 4. The agent submits one or more `addRule` proposals to `http://policy.local/v1/proposals`. 5. The gateway stores accepted proposals as pending draft chunks for the sandbox and runs the [policy prover](#what-auto-approval-checks) against the proposed delta. -6. Under `auto` mode, proposals with an empty prover delta are approved immediately and skipped past human review. Under `manual` mode (the default), every proposal — and under `auto` mode, every proposal with a prover finding — lands in the draft inbox for a developer to approve or reject. +6. Before auto-approval, the gateway reloads the current stored rule, recalculates its prover verdict, and regenerates its security notes. Under `auto` mode, it approves the proposal only when both the prover delta and security notes are empty. Any finding or security note keeps the proposal pending. Under `manual` mode, every accepted proposal lands in the draft inbox for a developer to approve or reject. 7. The agent waits on `/v1/proposals/{chunk_id}/wait` until a decision is available. Approved proposals hot-reload into the sandbox; rejected proposals return `rejection_reason` and `validation_result` so the agent can revise. When a proposal is approved, `/wait` reports `policy_reloaded: true` only after the local sandbox policy covers the approved rule. At that point the agent can retry the original denied action once. If a proposal is rejected, `/wait` returns `rejection_reason` and `validation_result` so the agent can revise or stop. `validation_result` carries the categorical prover findings — `link_local_reach`, `l7_bypass_credentialed`, `credential_reach_expansion`, `capability_expansion` — so the agent can narrow the next attempt to the specific concern the prover flagged. @@ -160,18 +160,26 @@ The current `policy.local` JSON shape covers L4 endpoints and REST method or pat Policy advisor proposals do not add `allowed_ips` automatically. If an advisor-proposed hostname resolves to an internal or private address, OpenShell's SSRF protections still block the connection until a developer explicitly adds the required `allowed_ips` entry. Exact hostname trust for user-declared policy endpoints does not apply to advisor-generated proposal binaries. +Private RFC 1918, CGNAT, IPv6 ULA, and other special-use destinations classified as internal produce advisory security notes when they appear as literal endpoint IPs or in `allowed_ips`. CIDR intersections are included, and hostless `allowed_ips` rules receive an additional warning because they can match any hostname resolving into the configured range. + +Always-blocked destinations are not advisory. Loopback, link-local, and unspecified IPs or CIDRs, plus `localhost` and known metadata endpoint hostnames, are excluded from security notes. Submit and edit can store such a draft, but approval fails when merge validation prevents it from entering the active policy. Runtime SSRF protections continue to enforce the same boundary. + ## What Auto-Approval Checks -The policy prover runs against every proposal — mechanistic and agent-authored alike — and asks four formal questions about the proposed change. Each "yes" is one categorical finding. Any finding blocks auto-approval; only an empty delta is eligible. +Auto-approval requires all three conditions: the effective mode is `auto`, the prover delta is empty, and recalculating security notes from the current stored rule produces none. + +The policy prover runs against mechanistic and agent-authored proposals alike and asks four formal questions about the proposed change. Each "yes" is one categorical finding. Any finding blocks auto-approval. An empty delta is necessary but not sufficient. | Category | Triggered when | |---|---| -| `link_local_reach` | The proposal reaches a host in `169.254.0.0/16`, `fe80::/10`, or a known metadata hostname such as `metadata.google.internal` (cloud-metadata territory, which serves credentials regardless of sandbox state). Unconditional. | +| `link_local_reach` | A rule reaches `169.254.0.0/16`, `fe80::/10`, or a known metadata hostname. | | `l7_bypass_credentialed` | A binary using a wire protocol the L7 proxy cannot inspect (`git-remote-https`, `ssh`, `nc`) gains reach to a host where a credential is in scope. | | `credential_reach_expansion` | A binary gains credentialed reach to a `(host, port)` it could not reach before. | | `capability_expansion` | On a `(binary, host, port)` that already had credentialed reach, the proposal adds a new HTTP method. The finding cites the specific method. | -Findings are categorical — there is no severity tier. The reviewer reads the category and the structured evidence to decide. When the prover delta is empty, the proposal is provably safe under the model and auto-approval (if enabled) can fire. +Findings are categorical. There is no severity tier. The reviewer reads the category and the structured evidence to decide. + +Before auto-approval, the gateway reloads the draft chunk, recomputes its prover verdict, and regenerates security notes from its current rule. These checks apply after edits and deduplicated resubmissions: an edited stored rule controls the decision even when the duplicate incoming proposal has an empty prover delta. The recalculated verdict is used for this decision but is not written back, so a later `validation_result` read can still show the submit-time verdict. Failures leave the chunk pending. Security notes flag concerns such as internal or private destinations and `allowed_ips`, wildcard hosts, hostless `allowed_ips`, ephemeral ports, and well-known database or service ports. Draft reads and bulk approval also regenerate notes from the stored rule. Any prover finding or security note keeps the chunk pending. The full reasoning model lives in [`crates/openshell-prover/README.md`](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-prover/README.md). Provider profiles composed in via [Providers v2](/sandboxes/providers-v2) are part of the effective policy the prover reasons over. @@ -183,7 +191,7 @@ Review pending chunks from the host: openshell rule get --status pending ``` -Under `auto` mode, only proposals the prover flagged appear here; empty-delta proposals are already approved and visible under `--status approved` with the auto-approval audit fields described in [Approval Modes](#approval-modes). Under `manual` mode, every proposal — regardless of prover verdict — shows up as pending. +Under `auto` mode, proposals with a prover finding or any recalculated security note remain pending for human review. Proposals that pass both checks are visible under `--status approved` with the auto-approval audit fields described in [Approval Modes](#approval-modes). Under `manual` mode, every accepted proposal shows up as pending regardless of the prover verdict or security notes. The output shows the chunk ID, status, rationale, binary, and endpoint summary. For L7 proposals, the endpoint summary includes the protocol, method, and path: diff --git a/docs/sandboxes/providers-v2.mdx b/docs/sandboxes/providers-v2.mdx index 69b4cfb307..fbb8e404fa 100644 --- a/docs/sandboxes/providers-v2.mdx +++ b/docs/sandboxes/providers-v2.mdx @@ -238,7 +238,6 @@ endpoints: path: /v1/** protocol: rest tls: "" - access: read-write enforcement: enforce allowed_ips: [] ports: [] diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index e4d99dc45c..b36a32203f 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -62,6 +62,11 @@ name = "podman_gateway_resume" path = "tests/podman_gateway_resume.rs" required-features = ["e2e-podman"] +[[test]] +name = "podman_corporate_proxy" +path = "tests/podman_corporate_proxy.rs" +required-features = ["e2e-podman"] + [[test]] name = "vm_gateway_resume" path = "tests/vm_gateway_resume.rs" diff --git a/e2e/rust/src/harness/container.rs b/e2e/rust/src/harness/container.rs index e4511e00b9..764ee6d0f6 100644 --- a/e2e/rust/src/harness/container.rs +++ b/e2e/rust/src/harness/container.rs @@ -182,6 +182,149 @@ impl ContainerHttpServer { } } +/// A generic support container running on the shared e2e container network. +/// +/// Unlike [`ContainerHttpServer`], this helper requires the network mode used +/// by the Docker/Podman gateway lanes (`OPENSHELL_E2E_NETWORK_NAME`), probes +/// readiness with a plain TCP connect instead of an HTTP GET (so it can host +/// non-HTTP fixtures such as forward proxies and TLS servers), and exposes the +/// container's logs and network IP for test assertions. +pub struct SupportContainer { + /// Network alias other containers on the e2e network reach this one by. + pub alias: String, + container_id: String, + network: String, + engine: ContainerEngine, +} + +impl SupportContainer { + /// Start a `python3 -c