feat: minimal changes to run OpenShell in an existing Kubernetes cluster

[EItanya]

Problem Statement

OpenShell already has most of what it needs to run in a user-provided Kubernetes cluster: the Helm chart at deploy/helm/openshell deploys the gateway, sandboxes already run as k8s pods via openshell-driver-kubernetes, and the driver's kube client at crates/openshell-driver-kubernetes/src/driver.rs:121 calls kube::Config::incluster() before falling back to kube::Config::infer() — so when the gateway runs as a pod it picks up its ServiceAccount automatically. The Helm chart already creates a ServiceAccount + ClusterRole/RoleBinding under deploy/helm/openshell/templates/, and gateway add http://... (crates/openshell-cli/src/main.rs:858-900) already supports registering a plaintext gateway by URL with no mTLS.

What blocks running the same chart on an arbitrary cluster is one structural assumption: every sandbox pod mounts /opt/openshell/bin as a read-only hostPath to pick up the openshell-sandbox supervisor binary, which is baked into the k3s cluster image at build time. See crates/openshell-driver-kubernetes/src/driver.rs:657-718 and deploy/docker/Dockerfile.images:334. The design comment at driver.rs:658 explicitly states the supervisor is never baked into user sandbox images so it stays version-locked with the gateway. This works in single-node k3s-in-Docker because we own the node filesystem. It breaks on real k8s — multi-node arch mismatches, managed-cluster PSS/admission policies blocking hostPath, immutable node AMIs.

This issue scopes the minimum server-side changes to lift the hostPath dependency and make the existing chart cleanly installable on any cluster. It is part of M1 in the upstream roadmap (#1012).

Proposed Design

1. Replace hostPath supervisor side-load with init-container + emptyDir

Ship a small openshell-supervisor:<gateway-tag> OCI image containing only /openshell-sandbox. Modify apply_supervisor_sideload in crates/openshell-driver-kubernetes/src/driver.rs so pod templates get:

• An emptyDir volume named openshell-supervisor-bin (replacing the current hostPath).
• An init container openshell-supervisor-loader that runs cp /openshell-sandbox /supervisor/openshell-sandbox against the volume.
• The existing read-only volume mount on the agent container at /opt/openshell/bin and the existing command override.

Helm values gain supervisor.image.repository and supervisor.image.tag, defaulted to track the gateway release. The cluster image's /opt/openshell/bin baking in Dockerfile.images and the supervisor presence check in deploy/docker/cluster-healthcheck.sh:55-61 can then be removed — same model works for both k3s-in-Docker and external clusters.

Cost: ~50–200 ms extra cold-start on first pull (image cached after that), one extra container in pod spec. Standard k8s pattern for shipping a binary into someone else's container without baking it into their image.

The Kubernetes 1.31 volumes[].image source is a cleaner future option but not yet ubiquitous on managed clusters; keep it on the roadmap as an opt-in fast path, not the default.

2. RBAC audit for the gateway ServiceAccount

The chart's existing ClusterRole/Role (deploy/helm/openshell/templates/clusterrole.yaml, role.yaml) was authored against single-node k3s. Audit the verbs/resources actually used by openshell-driver-kubernetes and the server against a real (multi-node, non-k3s) cluster:

• Sandbox CRD verbs in the sandbox namespace.
• pods, pods/exec, pods/log, pods/status in the sandbox namespace.
• secrets in the sandbox namespace (for client mTLS materials mounted into sandboxes).
• events in the sandbox namespace (the platform-event watcher).
• Cluster-scoped reads on nodes and node.k8s.io/runtimeclasses (already present for GPU admission).

Tighten any over-broad rules and confirm there are no implicit cluster-admin-equivalent expectations. The audit's output is the documented minimum-RBAC manifest published with the chart.

3. Documentation

A new architecture/gateway-kubernetes.md (or extension of gateway-single-node.md) that covers:

• helm install openshell deploy/helm/openshell -n openshell --create-namespace -f values.yaml
• Recommended values for an MVP install: server.disableTls=true, server.disableGatewayAuth=true, server.sshGatewayHost/sshGatewayPort set to the externally reachable address.
• Reaching the gateway: kubectl port-forward for dev, NodePort/LoadBalancer/Ingress for shared use.
• Registering on the client: openshell gateway add http://<reachable-host>:<port>.
• Minimum RBAC manifest from (2).
• mTLS hardening called out as a follow-up, not part of MVP.

Scope boundary

In scope for this issue: the three items above. They are sufficient to take the existing Helm chart and install it on any cluster as plaintext-MVP without functional regressions.

Explicitly not in this issue:

• mTLS bundle distribution for cluster-deployed gateways (M2 hardening — separate issue).
• Helm chart publishing as an OCI artifact on ghcr.io (separate M1 follow-up).
• k3s + skaffold/tilt local dev loop (separate M1 follow-up).
• Horizontal scaling, ingress recipes, RuntimeClass/kata/gvisor integration (M2).
• Operator, snapshotting, sandbox-supervisor rolling upgrades (M3).
• Any new CLI commands or GatewayMetadata shape changes — gateway add http://... already covers the plaintext registration path.

Alternatives Considered

Bake the supervisor into a required base image. Forces every BYOC sandbox image to FROM ghcr.io/nvidia/openshell/sandbox-base. Removes runtime injection but reintroduces version-skew risk — exactly what the comment at driver.rs:658 was written to avoid. Rejected.

DaemonSet that drops the supervisor binary onto each node's hostPath. Preserves the existing model on multi-node, but still requires hostPath write privileges (worse from a PSS standpoint than what we have today) and doesn't help on managed/restricted clusters. Rejected.

External kubeconfig in the server config. Initially considered as a way to let openshell-server manage a remote cluster. Unnecessary: when the server runs as a pod inside the same cluster, kube::Config::incluster() already picks up the ServiceAccount automatically. The driver's existing fallback chain (incluster() → infer()) handles both in-cluster and out-of-cluster development without new wiring.

New Kubernetes bootstrap backend in openshell-bootstrap. Considered an openshell gateway install subcommand that wraps helm install and watches Service status for the endpoint. Dropped: helm install is the install UX, and gateway add http://... already covers the client side. No bootstrap code is needed for an in-cluster install — it's an operator-driven flow, not a CLI-driven one.

New GatewayKind::Kubernetes { context, namespace } enum on GatewayMetadata. Dropped for the same reason: the CLI doesn't need to know its target is in k8s. It just needs a URL and (eventually) credentials.

Agent Investigation

Files inspected:

• crates/openshell-driver-kubernetes/src/driver.rs:121 — confirmed kube::Config::incluster() is tried first, then kube::Config::infer(). The in-cluster ServiceAccount path is already wired.
• crates/openshell-driver-kubernetes/src/driver.rs:657-718 — confirmed apply_supervisor_sideload is the only injection point for the supervisor and is the sole consumer of the hostPath mount via supervisor_volume() at line 672.
• deploy/docker/Dockerfile.images:334 and deploy/docker/cluster-healthcheck.sh:55-61 — confirmed the supervisor binary is baked at /opt/openshell/bin/openshell-sandbox in the cluster image at build time and is the only producer of the side-loaded binary.
• deploy/helm/openshell/values.yaml:99-104 — confirmed server.disableTls and server.disableGatewayAuth already exist; an MVP install can run plaintext today with no chart changes.
• crates/openshell-cli/src/main.rs:858-900 — confirmed openshell gateway add <endpoint> already registers a gateway by URL and treats http://... as plaintext with no mTLS extraction. No new CLI surface is required for the MVP.
• deploy/helm/openshell/templates/clusterrole.yaml, role.yaml, clusterrolebinding.yaml, rolebinding.yaml, serviceaccount.yaml — confirmed RBAC objects exist; their verb/resource set should be audited against a real cluster (item 2).

The Helm chart, the sandbox CRD pipeline, the in-cluster kube client, and the CLI's plaintext-add path are already cluster-ready. The supervisor hostPath side-load is the single blocking change.

Related

• Roadmap: High-availability Kubernetes Support #1012 (M1 — "OpenShell server accepts a kube config and is decoupled from any k3s")

[drew]

Thanks for the detailed feature request. This is largely how we're thinking about Kubernetes support as well.

Replace hostPath supervisor side-load with init-container + emptyDir

We're thinking something similar. We've started by publishing an image here https://github.com/NVIDIA/OpenShell/pkgs/container/openshell%2Fsupervisor, and are side loading this with our podman/docker/microvm drivers. We'll plan to do something similar for Kubernetes.

Add a Kubernetes bootstrap backend

We'll likely be deprecating the bootstrapping pattern. For Kubernetes based deployments we'll publish a Helm chart. For single node installation we'll provide homebrew/rpm/deb/snap packages for installing OpenShell.

Metadata and CLI changes

Each cluster with likely have some control plane auth strategy with single node deployments continuing to use mtls and Kubernetes based deployments using keycloak and other idp solutions.

We'll be publishing a roadmap issue for this soon.

[EItanya]

Makes sense, I have a lot of experience in this area and would love to help

[rrhubenov]

Looking at the roadmap issue it seems there's a recent development of leveraging image volumes for this case. I find that to be a cleaner approach.

Only downside is that image volumes are introduced in k8s 1.31 behind a feature gate and are GA since 1.36. Thus this won't work for every cluster.

[EItanya]

GA since 1.36 is a pretty large hurdle, that's the most recent. Most folks who are actually running k8s are definitely not on latest. I personally lean towards not letting perfect be the enemy of good here and start with something which is more supported and optionally allow for the more ideal path.

[TaylorMutch]

This implements the side-load pattern with init container and emptyDir #1154