feat(helm): cert-manager external issuer + OpenShift passthrough Route

[jhjaggars]

Lets cert-manager issue the gateway's server TLS certificate from a real Issuer/ClusterIssuer (for example an ACME issuer) instead of only the chart's built-in self-signed CA, and adds an OpenShift Route template with TLS passthrough so the gateway can be exposed externally with a publicly-trusted certificate while it keeps terminating its own TLS/mTLS. Also fixes a related sandbox-supervisor TLS trust bug this uncovered.
Closes #2466

• certManager.serverIssuerRef / certManager.clientIssuerRef (new values): override the issuerRef on the server/client Certificate resources, falling back to the chart's own self-signed CA issuer when unset (fully backward compatible).
• Server Certificate's SANs/commonName are scoped to only the operator-configured external hostnames when serverIssuerRef is set, since ACME issuers reject internal-only names (*.svc.cluster.local, localhost, loopback IPs) and require commonName to also be a SAN.
• New templates/route.yaml: an OpenShift Route with tls.termination: passthrough, gated by openshiftRoute.enabled.
• crates/openshell-core: the sandbox supervisor's gRPC client only trusted the CA configured via OPENSHELL_TLS_CA. Once the server cert comes from a different (public) issuer than the client cert, every supervisor connection failed the TLS handshake with UnknownCA. Fixed by also trusting the public root stores (native + webpki) in addition to the configured CA — this is additive, not a replacement, so existing self-signed deployments are unaffected.
• Docs: production section in docs/kubernetes/openshift.mdx, new section in docs/kubernetes/managing-certificates.mdx documenting the new fields and why the client (mTLS) certificate should stay on the chart's internal CA even when the server certificate moves to a public issuer.
• New CI render-coverage values file (ci/values-openshift-route-cert-manager.yaml) and Helm unittest coverage (tests/route_test.yaml, tests/cert_manager_pki_test.yaml).
• Synced .agents/skills/debug-openshell-cluster/SKILL.md with troubleshooting steps for the new failure modes.
  Validated end-to-end against a live OpenShift (ROSA) cluster: passthrough Route serving a real Let's Encrypt certificate issued via a Route53 DNS-01 ClusterIssuer, verified with a standard TLS client (no custom CA trust required), OIDC-authenticated CLI access via Keycloak, and a sandbox pod's supervisor successfully mTLS-connecting back to the gateway with the fixed trust store.
• mise run pre-commit passes
• Unit tests added/updated (Helm unittest coverage for the Route template and issuerRef overrides; existing statefulset_client_ca_test.yaml coverage re-verified with no regressions)
• E2E tests added/updated (if applicable) — validated manually against a live OpenShift cluster instead; see PR description above
• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mrunalp]

/ok to test cf02477

[mrunalp]

Blocking

1. Native roots are controlled by the sandbox image

crates/openshell-core/src/grpc_client.rs:176-180

The supervisor now calls both with_native_roots() and with_webpki_roots(). For Docker and Podman, the supervisor binary runs inside the user-selected sandbox image. The image's CA bundle is therefore not an operator-controlled trust source. rustls-native-certs also honors SSL_CERT_FILE and SSL_CERT_DIR, and both drivers pass user-provided template/spec environment variables into the supervisor.

If an attacker can also influence DNS or routing for the gateway connection, they can install their own CA, present a certificate for the gateway hostname, and receive the sandbox JWT that the supervisor sends on its RPCs. That can expose sandbox-scoped data and let the fake gateway return attacker-selected policy during supervisor startup.

There is also a compatibility regression: tonic 0.14 returns NativeCertsNotFound when the native store is empty before it considers the configured CA or WebPKI roots. A minimal BYOC image can therefore fail to start even though OPENSHELL_TLS_CA is valid.

Please remove with_native_roots() here and use the explicitly configured CA plus the compiled-in WebPKI roots. If enterprise/private server issuers need support, add an operator-mounted server CA bundle rather than trusting the sandbox image's native store. It would also be prudent to reserve or scrub SSL_CERT_FILE, SSL_CERT_DIR, and OPENSHELL_GATEWAY_TLS_SERVER_NAME from supervisor input.

2. serverIssuerRef renders a broken default client-CA configuration

deploy/helm/openshell/values.yaml:351-369
deploy/helm/openshell/templates/_gateway-workload.tpl:147-157

Setting the advertised serverIssuerRef and an external SAN while leaving the other defaults unchanged keeps clientCaFromServerTlsSecret=true. The gateway then tries to mount ca.crt from the public server certificate Secret.

ACME Secrets commonly do not contain ca.crt, leaving the gateway pod in MountVolume.SetUp failed. If an issuer does populate it, it is still not the CA that signed the chart-issued supervisor client certificate.

The chart should fail rendering when serverIssuerRef.name is combined with clientCaFromServerTlsSecret=true, or derive the correct client CA source automatically. Please add a negative Helm test for the default interaction.

3. clientIssuerRef is not independently usable as documented

deploy/helm/openshell/templates/cert-manager-pki.yaml:138-147
docs/kubernetes/managing-certificates.mdx:109-111

Changing only clientIssuerRef leaves the gateway trusting the built-in server CA while the supervisor receives the custom client issuer's CA. The gateway does not trust the new client certificate, and the supervisor no longer trusts the default server certificate. Every supervisor TLS connection consequently fails.

Either remove this option for now or model the server-verification and client-verification trust bundles separately and validate the complete configuration. kind should also be required or default to cert-manager's standard Issuer; silently defaulting to ClusterIssuer is surprising for an interface that advertises both.

4. The documented OIDC production command cannot connect

docs/kubernetes/openshift.mdx:102-135
crates/openshell-server/src/cli.rs:256-301

The Helm command never sets server.oidc.issuer, but the next step registers the CLI using --oidc-issuer. That flag configures only the CLI. Without server-side OIDC, the gateway requires a client certificate and rejects the OIDC-only CLI during the TLS handshake.

Please add server.oidc.issuer (and the expected audience) to the Helm command, or make completing the Access Control configuration an explicit prerequisite before registration.

5. The passthrough Route accepts a plaintext backend

deploy/helm/openshell/templates/route.yaml:4-26

openshiftRoute.enabled=true can currently be combined with server.disableTls=true, including by adding the Route settings to the existing OpenShift quickstart. Helm succeeds, but HAProxy forwards TLS handshake bytes to a plaintext gateway, resulting in resets or hangs.

The template should fail when passthrough routing is enabled while gateway TLS is disabled, with a test covering that combination.