Index of every page on this site: llms.txt
All skills
nvidia avatar

/helm-dev-environment

@c24fcd2
by NVIDIA Corporationnvidia/openshell22k stars
2,144
Run it yourself

Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.

Use this Skill: https://skilld.dev/gh/nvidia/openshell/helm-dev-environment

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈138 tokens for metadata: the name and description. ≈9.5k when used: this file.

Description uses 6.9% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈551 characters, or 6.9%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).


Prerequisites

  • Docker Desktop (macOS) or Docker Engine (Linux) running
  • mise install completed (provides k3d, kubectl, skaffold, helm)

Startup

1. Create the cluster

mise run helm:k3s:create

Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file. When the named cluster already exists, the task starts any stopped containers and refreshes same-named kubeconfig entries so a recreated load balancer's current API port takes effect. Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace collector and UI in the observability namespace, and preloads the default sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time.

Multi-worktree support: the cluster name is derived from the last component of the current git branch (e.g. branch chore/1234-local-dev/octocat → cluster openshell-dev-octocat). Each worktree therefore gets its own isolated cluster and its own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

Host port Target Used by
8080 Port 80 via k3d load balancer Envoy Gateway LoadBalancer service (values-gateway.yaml)

Override with env vars before running helm:k3s:create:

  • HELM_K3S_LB_HOST_PORT (default: 8080)
  • HELM_K3S_PRELOAD_SANDBOX_IMAGE (default: nvcr.io/nvidia/base/ubuntu:24.04; set to an empty value to skip)
  • HELM_K3S_COLLECTOR_IMAGE (default: mcr.microsoft.com/dotnet/aspire-dashboard:latest)
  • HELM_K3S_COLLECTOR_HEALTH_TIMEOUT (default: 120 seconds)

Gateway metrics

The chart enables its dedicated Prometheus endpoint by default. Set metrics.enabled=false to omit the listener and its container and Service ports without changing service.metricsPort (default 9090). The port must remain between 1 and 65535 even when disabled. Legacy metricsPort: 0 or false values fail rendering; migrate them to metrics.enabled=false and a valid port in the same upgrade. metrics.tls.enabled=true requires metrics.enabled=true. This setting does not disable OpenTelemetry export.

Shared supervisor plumbing

Set gatewayConfig.openshell.drivers.kubernetes.supervisor_topology using a values file (the table key is literally openshell.drivers.kubernetes):

gatewayConfig:
  openshell.drivers.kubernetes:
    supervisor_topology: gateway

The gateway chart creates a shared supervisor Deployment (supervisor.replicas, default one), Service, NetworkPolicy, and an empty <release>-supervisor-token Secret in the release namespace, regardless of shared/managed/operator workspace mode. The gateway mints a supervisor token from its sandbox-JWT signing key, writes it into that Secret (the gateway Role may get/patch only that Secret), and renews it at half its lifetime; the supervisor mounts the Secret and presents the token on the private openshell.supervisor.host.v1 assignment stream, which accepts nothing else. The topology requires gateway TLS and the signing key (server.disableTls is rejected; the gateway refuses to start without gateway_jwt). New sandbox generations run as sandbox instances in that process. The chart starts --mode=shared, sets --gateway-endpoint to the gateway gRPC URL, mounts the token at --token-file, and binds --address (5501) to a plain TCP health listener for the tcpSocket probes; nothing dials the supervisor but sandbox runtimes (5500) and the kubelet. Build the gateway, supervisor, and sandbox images from the same branch: all three must support the routed runtime connection and the assignment stream. workspace is reserved and rejected. Omit the field to disable it. mise run e2e:kubernetes:shared-supervisor runs the focused e2e (ci/values-supervisor.yaml). The task prepends that overlay to any colon-separated OPENSHELL_E2E_KUBE_EXTRA_VALUES list, so layer a workspace-mode overlay to cover managed namespaces:

OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-workspace-managed.yaml \
OPENSHELL_E2E_KUBE_IMAGE_PULL_SECRET=e2e-regcred \
  mise run e2e:kubernetes:shared-supervisor

mise run e2e:kubernetes:shared-supervisor-spiffe installs pinned SPIRE charts (ci/values-spire-broker-stack.yaml) and the openshell.gateway.spiffe_broker overlay (ci/values-supervisor-spiffe.yaml). It exercises provider exchange and abrupt supervisor takeover over TLS, using Sandbox CR UID registrations and enforced Kubernetes authorization. Workload generation, Pod, process, and stable SPIFFE identity survive; the new owner acquires a fresh provider token and the broker rejects the old bearer. The fixture's 30-second workload X.509-SVIDs exercise acquisition after credential rotation. The gateway Pod must pass Restricted Pod Security admission. The task owns its SPIRE install and cluster.

The task creates three nodes (HELM_K3S_AGENT_COUNT=2 adds two k3d agents) and requires three SPIRE agents. Its cluster-scoped server_spiffe_id_prefix accepts the concrete node identities, and a node alias distributes Sandbox registrations to all eligible agents. The test selects each Broker backend in turn and forces a new generation's acquisition, then replaces one agent and checks acquisition without re-registering the sandbox. Exact-ID mode remains available for a fixed endpoint. The installed SPIRE version and operator must expose the experimental Broker API.

For remote clusters, set OPENSHELL_E2E_HOST_GATEWAY_IP to a host reachable from Pods. Optional OPENSHELL_E2E_SPIFFE_STS_PORT and OPENSHELL_E2E_SPIFFE_TARGET_PORT fix the test's host listener ports so an operator can forward them through a reverse tunnel. Keep the tunnel available through the test and remove it afterward. Use an explicit kubeconfig and context to avoid repository environment defaults targeting a local cluster. OPENSHELL_E2E_SPIFFE_DRIVER_CONFIG_JSON can pass a driver override to test sandbox creation, such as a node selector for a Landlock-capable pool. Enable allow_driver_config in the test gateway when using this override. GKE COS nodes may lack Landlock; use compatible Ubuntu nodes for sandbox workloads. For remote abrupt-failover coverage, set OPENSHELL_E2E_SUPERVISOR_PROCESS_CONTROL=1. Before creating sandboxes, the test adds an unprivileged process-control sidecar and enables the supervisor Pod's shared process namespace. The sidecar sends SIGKILL to the supervisor as the same UID; it needs no host PID namespace, capabilities, or runtime socket. The default k3d/kind test continues to kill through the node container runtime.

Shared supervisor high availability

mise run e2e:kubernetes:shared-supervisor-spiffe-node-local validates direct node-local Broker TCP routing. The chart injects OPENSHELL_NODE_IP from status.hostIP for spiffe_broker.node_local_port; it creates no Broker Service. The task uses an external PostgreSQL fixture and a gateway Deployment, moves the gateway across the SPIRE nodes, restricts Broker egress to its current node, and forces fresh token acquisition on each node and after local agent replacement. The shared-supervisor failover, credential rotation, and identity fencing checks also run. Keep gateway/supervisor/sandbox images from the same branch.

The shared supervisor is always a pool of supervisor.replicas members (default 1), or an HPA-managed pool when supervisor.autoscaling.enabled=true. ci/values-supervisor-autoscaling.yaml enables the memory HPA (default 512Mi average per Pod, two to four members, one removal every two minutes after five minutes of stabilization). It requires the resource metrics API, usually Metrics Server. The HPA omits the Deployment's fixed replica count and ignores supervisor.replicas; top-level autoscaling still controls only the gateway. Do not combine the HPA overlay with the manual 1→2→1 scaling E2E: that test expects Helm to own replicas. Changing HPA bounds or target must leave the gateway config and supervisor Pod template unchanged, as checked by deploy/helm/test-supervisor-pool-scaling.sh. With the gateway topology the chart renders the gateway's [openshell.gateway.supervisor_pool] tuning table (only recovery_timeout_secs from supervisor.recoveryTimeoutSeconds, default 60; the driver's shared_supervisor table stays unchanged), a RollingUpdate Deployment (maxSurge: 1, maxUnavailable: 0) with a 60-second grace period, no affinity unless supervisor.affinity is set (top-level affinity applies only to the gateway), the pool identity env vars (OPENSHELL_SUPERVISOR_POOL_MEMBER_ID and _ADVERTISE_IP from the Pod name and IP), and a maxUnavailable: 1 PodDisruptionBudget unless supervisor.podDisruptionBudget.enabled=false (default true; toggling it changes only the PodDisruptionBudget). Each supervisor resource has its own template: supervisor.yaml (Deployment), supervisor-service.yaml, supervisor-networkpolicy.yaml, supervisor-token-secret.yaml, supervisor-pdb.yaml, and supervisor-hpa.yaml. ci/values-supervisor-ha.yaml selects two replicas and a 60-second recovery timeout. The topology needs gateway TLS: the e2e harness keeps it on with ci/values-tls-port-forward.yaml and OPENSHELL_E2E_KUBE_TLS=1, which reaches the gateway over mTLS through its port-forward. mise run e2e:kubernetes:shared-supervisor-ha sets both, creates a task-scoped k3d cluster, kills the supervisor Pod that owns a sandbox, and checks that the sandbox keeps its Pod, generation, main process, and in-memory state.

The rendered gateway configuration and supervisor Pod template do not depend on supervisor.replicas; deploy/helm/test-supervisor-pool-scaling.sh (part of mise run helm:test) renders a pool at 1, 2, and 3 replicas and fails if anything but the replica count differs. mise run e2e:kubernetes:shared-supervisor-scaling installs a one-member pool (ci/values-supervisor-ha-scaling.yaml), then the test runs helm upgrade --reuse-values --set supervisor.replicas=2 and back to 1 against the chart the harness installed (OPENSHELL_E2E_KUBE_CHART), probing exec and policy-mediated networking throughout. It checks that scaling up moves nothing, that the member removed by scaling down hands its sandboxes to the other (cause=leaving), that no workload, gateway Pod, or remaining member restarts, and that a new sandbox runs after each step. It upgrades its release, so it runs on a release of its own; the two pool tests also share a nextest test group that never runs them at once. Against an existing cluster whose kubeconfig the shell environment may override, set OPENSHELL_E2E_KUBECONFIG (passed as --kubeconfig to kubectl and helm).

On k3d, deploy the pool with the Skaffold profile, which layers the overlay and turns gateway TLS back on over ci/values-skaffold.yaml:

cd deploy/helm/openshell
KUBECONFIG=../../../kubeconfig skaffold run -p shared-supervisor-high-availability
cd ../../..

The plaintext registration from helm:skaffold:run does not apply. Keep mise run helm:k3s:forward running and set up the client certificate as in "With mTLS enabled" under Connecting through the forwarding task. Both members share the single k3d node, so exercise Pod failures:

export KUBECONFIG=kubeconfig OPENSHELL_GATEWAY_ENDPOINT=https://localhost:8090
openshell sandbox create --detach --name ha-probe -- sh -c 'echo $$ > /sandbox/main.pid; exec sleep infinity'
openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid
openshell sandbox get ha-probe  # note the sandbox ID
# Supervisor logs name the instance by its route, not its sandbox ID.
kubectl get sandboxes.agents.x-k8s.io -A -l openshell.ai/sandbox-name=ha-probe \
  -o jsonpath='{.items[0].metadata.annotations.openshell\.ai/sandbox-runtime-supervisor-session}'
# The member whose latest admission, removal, or release line names the route owns it.
kubectl -n openshell logs -l app.kubernetes.io/name=openshell-supervisor --prefix --tail=-1 \
  | grep -E '(admitted assigned|removing withdrawn|relinquishing a) sandbox instance.* <route>$'
# Force-delete the owner, the closest local stand-in for a crash.
kubectl -n openshell delete pod <owner-pod> --grace-period=0 --force
kubectl -n openshell logs -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
  -c openshell-gateway --prefix --tail=200 | grep 'taking over a shared sandbox instance'
kubectl -n openshell logs -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload \
  --tail=200 | grep -E 'sandbox-supervisor-|Supervisor takeover confirmed'
openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid  # same PID

A plain kubectl delete pod exercises the graceful leave instead: the member announces it is leaving and the gateway hands its instances over at once. A member that stops without announcing it, as on node loss, is declared dead only after the 15-second membership TTL. kubectl -n openshell rollout restart deployment/openshell-supervisor replaces every member one at a time. The PID must survive each case, while an open sandbox connect or exec stream breaks and must reconnect. A sandbox-supervisor-recovery-expired finding in the workload log means no replacement confirmed within the recovery timeout, so the runtime terminated the workload.

For RFC 0017 app-listener validation, enable the listener and supply the app TLS Secret on the shared Deployment, plus a separate NetworkPolicy allowing the test client/proxy to reach its app port. Runtime-port access does not grant app access. Discovery returns the acknowledged owner's Pod IP and app port; no headless app Service is needed. Exercise HTTP/WebSocket reconnection through graceful downscale and abrupt loss, preserving workload generation/PID/in-memory state and the public hostname. Verify stale-owner rejection independently of the replacement's success. Shared serving leases expire within fifteen seconds of issuance and do not renew from configuration alone. Do not claim seamless socket migration or infer external ingress readiness from an advertisement.

2. Deploy OpenShell

Iterative dev (rebuilds on file changes, recommended during active development):

mise run helm:skaffold:dev

One-shot deploy (build once and leave running):

mise run helm:skaffold:run

Resource admission defaults to enabled and caller driver config to disabled. Driver-config scenarios need an explicit allowDriverConfig opt-in; external attachments also need administrator-controlled approval labels in the target namespace. GPU attachments and operator-selected image-pull Secrets are exempt from labels. Managed workspace image-pull Secrets are copied from the configured source in the gateway namespace; do not grant approval to the gateway database PVC or disable admission to make tests pass.

The Skaffold flow builds distinct gateway, sandbox, and supervisor images and deploys the OpenShell Helm chart. The Kubernetes driver creates a capability-free workload Pod and a directly managed capability-free supervisor Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell workload Pod. New sandbox generations use runtime-initiated control: the sandbox runtime dials its supervisor through os-supervisor-<id>. The legacy os-boundary-<id> Service keeps selecting the workload for old gateways during rolling upgrades. New gateways create both Services and bind the control-mode annotation to its runtime generation. The driver adds a per-sandbox os-control-<id> NetworkPolicy for that path only. Existing running generations keep their connection until replaced; stop/start and gateway recovery that recreates both Pods upgrade legacy generations automatically. Use compatible gateway, supervisor, and sandbox runtime images together; there is no legacy rollout flag. The pkiInitJob hook (a pre-install Job that runs openshell-gateway generate-certs) generates gateway and CLI TLS secrets on first install. Supervisor Pods project only ca.crt and authenticate gateway RPCs with sandbox bearer tokens. User client certificates and private keys remain outside supervisor and workload Pods. The default Skaffold values export gateway and Kubernetes-driver traces to the collector service installed by helm:k3s:create. Envoy Gateway is opt-in; see the Optional Add-ons section.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or the unified local forwarding task:

mise run helm:k3s:forward

The task forwards OTLP/gRPC to http://127.0.0.1:4317 and the trace UI to http://127.0.0.1:18888. When Skaffold has deployed a Kubernetes gateway, it also forwards the gateway to http://127.0.0.1:8090; otherwise it continues with the collector ports only. A successful plaintext helm:skaffold:run registers the gateway under the worktree-specific k3d cluster name and selects it as the active gateway. Keep the forwarding task running while using those endpoints.

Viewing local traces

The gateway exports OTLP/gRPC to http://openshell-collector.observability.svc.cluster.local:4317 through ci/values-local-tracing.yaml, which only skaffold.yaml layers. The Kubernetes E2E wrapper omits it, so E2E clusters export no traces. Forward OTLP/gRPC and the trace UI to the host:

mise run helm:k3s:forward

Open http://127.0.0.1:18888 and exercise the gateway to inspect gateway and Kubernetes compute-driver spans under their distinct service names, along with Agent Sandbox controller reconciliation spans linked through the Sandbox trace-context annotation. The same command exposes OTLP/gRPC on http://127.0.0.1:4317 and, when deployed, the Kubernetes gateway on http://127.0.0.1:8090. The local gateway, gateway:docker, gateway:podman, and gateway:vm tasks enable trace export, OCSF log export, and the agent trace relay only when OPENSHELL_DEV_OTLP_ENDPOINT is set, for example OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317. Spans an agent exports from inside a sandbox arrive through the supervisor's relay with openshell.telemetry.source=agent and share openshell.sandbox.id with that sandbox's supervisor spans.

The Skaffold profile for HA reverse-proxy development is available from deploy/helm/openshell/:

# Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route.
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability

The high-availability profile expects a Secret named openshell-ha-pg in the openshell namespace with a uri key. For local manual testing, either create your own PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in e2e/kubernetes/postgres-fixture.yaml.

For the high-availability profile, return to the repository root and apply the GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed Envoy Gateway:

KUBECONFIG=kubeconfig mise run helm:gateway:apply

The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for OpenShell's GRPCRoute. Keep that policy in deploy/kube/manifests/envoy-gateway-openshell.yaml, not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay streams during gateway rollouts and scale events.

TLS behaviour

ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run plaintext by default. Override server.disableTls=false to exercise TLS/mTLS.

Mode server.disableTls Gateway scheme
Skaffold dev (default) true http://
TLS enabled false (or omitted) https://

Connecting through the forwarding task

Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so the forwarding task uses local port 8090 for the gateway. In a second terminal, confirm that the gateway is registered and active:

openshell gateway list

Plaintext (default Skaffold deploy):

openshell sandbox list

With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:

mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key

The server cert SANs include localhost and 127.0.0.1, so hostname verification passes over a port-forward without any extra flags:

openshell sandbox list --gateway-endpoint https://localhost:8090

Teardown

Remove the Helm releases (keep cluster)

mise run helm:skaffold:delete

Delete the cluster entirely

mise run helm:k3s:delete

This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.


Optional Add-ons

Some add-ons can be enabled by uncommenting values in skaffold.yaml, but prefer the dedicated Skaffold profiles when they exist. Profiles avoid leaving local manual edits in the worktree.

Envoy Gateway (Gateway API / GRPCRoute)

Use the high-availability Skaffold profile for HA reverse-proxy testing. The profile intentionally includes Envoy Gateway so multi-replica behavior is exercised through the same Gateway API path used by reverse-proxy deployments:

cd deploy/helm/openshell
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
cd ../../..
KUBECONFIG=kubeconfig mise run helm:gateway:apply

values-gateway.yaml creates the openshell Gateway (listener on port 80, class eg) and an openshell GRPCRoute in the openshell namespace. The high-availability profile installs the Envoy Gateway Helm chart and layers both values-high-availability.yaml and values-gateway.yaml onto the OpenShell release.

deploy/kube/manifests/envoy-gateway-openshell.yaml creates:

  • GatewayClass/eg
  • BackendTrafficPolicy/openshell-grpc-timeouts

The Envoy Gateway proxy Service is usually exposed through the k3d load balancer at http://127.0.0.1:8080. If the cluster was created with a different HELM_K3S_LB_HOST_PORT, use that host port instead.

For manual tests against an existing cluster, prefer forwarding the Envoy proxy Service rather than svc/openshell. That keeps client traffic on the same path as a real reverse proxy while gateway pods rotate behind it:

KUBECONFIG=kubeconfig kubectl get svc -A \
  -l gateway.envoyproxy.io/owning-gateway-name=openshell
KUBECONFIG=kubeconfig kubectl -n <envoy-service-namespace> port-forward \
  svc/<envoy-service-name> 8080:80
openshell gateway add http://127.0.0.1:8080 --name openshell --local

When running e2e tests manually through Envoy, register gateway metadata (as above) instead of relying only on OPENSHELL_GATEWAY_ENDPOINT; some tests call openshell gateway info and expect metadata for the active gateway.

Kubernetes E2E Notes

Use mise run e2e:kubernetes for the standard Helm-backed Kubernetes suite. Set OPENSHELL_E2E_KUBE_TEST=kubernetes_image_update to scope the Rust suite to combined image, command, environment, and metadata updates with retained sandbox, SSH, and PVC identities. The next start verifies the new argv and environment. This target is included in both existing Agent Sandbox API-version lanes. Docker separately supports image-only updates on stopped, opt-in persistent /sandbox workspaces; this Kubernetes lane does not qualify that path. Docker requires locally cached images, a matching numeric target USER UID:GID, and no image-declared volumes. Do not treat a skipped Docker lane as live image-replacement coverage.

The separate docker-updates testkit suite qualifies the full Docker path through an owned mTLS gateway, including SSH exec, image replacement, restart, credential-publication repair and deletion. Follow tests/suites/features/docker-updates/README.md; the wrapper's OPENSHELL_E2E_DOCKER_IMAGE_UPDATES=1 opts in only its fresh gateway and refuses an existing endpoint. On Docker Desktop without host networking, select the documented bridge override rather than changing engine settings.

Image updates default to disabled outside test fixtures. The e2e wrapper opts in only for a fresh gateway fixture that it owns; set OPENSHELL_E2E_KUBE_IMAGE_UPDATES=0 to prevent that opt-in. Do not propagate activation to an existing shared gateway or enable it during a mixed-version rollout. For manual testing, follow the retirement and activation checklist in docs/kubernetes/high-availability.mdx before setting gatewayConfig: {"openshell.gateway": {enable_sandbox_updates: true}}. The gateway process equivalents are --enable-sandbox-updates true and OPENSHELL_ENABLE_SANDBOX_UPDATES=true. Retire every old process and Job, settle outstanding driver work, prevent old binaries returning, and do not treat PostgreSQL connection markers or membership TTL expiry as retirement proof. SQLite testing supports one gateway process only. Image-update clients must call the separate SandboxUpdates service on the existing gateway listener, not add a method to the baseline OpenShell service or inspect new public SandboxStatus image fields. Resume a pending operation with SandboxUpdates.UpdateSandbox, the current resource version, and no selected fields, following the documented request-ID semantics.

For the opt-in application listener, build/import matching branch images and run mise run e2e:kubernetes:app-listener; see e2e/kubernetes/app-listener/README.md for image, topology, delivery-mode and takeover inputs. The task owns an isolated Helm release/namespace, temporary TLS material, in-cluster clients, teardown and evidence under results/app-listener/. It checks policy denial, Pod-IP ingress without app Services, shared-owner takeover, gateway-outage expiry and disabled-listener gateway compatibility, without discovery or a proxy. CI runs four cases under test:e2e and includes them in Core E2E result. Its reusable kind setup installs pinned Calico only for this lane; other jobs retain kindnet. Local k3s already enforces policy. The lower-level verify.py can target a predeployed release. The kube e2e wrapper creates only one port-forward, to svc/openshell; it no longer forwards the unauthenticated health listener or runs a /readyz e2e target. /readyz remains covered by server unit/integration tests.

Use mise run e2e:kubernetes:ha-rebalancing for full-suite HA coverage. The task creates an external PostgreSQL fixture, installs Envoy Gateway, applies deploy/kube/manifests/envoy-gateway-openshell.yaml, enables the chart GRPCRoute, and runs the full Kubernetes e2e suite, including kubernetes_ha_rebalancing. That coverage validates sandbox create/watch and exec through the Envoy proxy while gateway replicas scale up, scale down, and rotate. It also keeps a long-running sandbox alive and runs upload/download operations while gateway pods roll, so file sync exercises the same relay retry path as interactive sessions.

If you reuse an existing Skaffold cluster for the full kube suite, make sure the chart has server.hostGatewayIP set so sandbox pods can resolve host.openshell.internal back to the test host. The e2e wrapper detects this on chart installs; manual reuse may require:

HOST_GATEWAY_IP="${OPENSHELL_E2E_HOST_GATEWAY_IP:?set host gateway IP}"
KUBECONFIG=kubeconfig helm upgrade openshell deploy/helm/openshell \
  --namespace openshell --reuse-values \
  --set "server.hostGatewayIP=${HOST_GATEWAY_IP}" \
  --wait --timeout 5m

Use the IP that pods in that cluster use to reach listeners on the test host.

BackendTLSPolicy (end-to-end TLS)

To enable end-to-end TLS between the Gateway proxy and the gateway pod, add BackendTLSPolicy values to the Helm install:

helm upgrade --install openshell deploy/helm/openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.backendTLSPolicy.enabled=true \
  --set server.tls.enableMtls=false \
  ...

This requires server.tls.enableMtls=false because ingress proxies cannot present client certificates to the backend. The certgen hook creates a backend CA ConfigMap from the server Secret's ca.crt key. With cert-manager, a separate post-install Job polls for the cert-manager-issued certificate (up to pkiInitJob.timeoutSeconds); with built-in PKI the ConfigMap is created in the same pre-install hook. The ConfigMap is reconciled on every upgrade so CA rotations propagate automatically.

Key Helm values:

  • grpcRoute.backendTLSPolicy.enabled: create the BackendTLSPolicy resource
  • grpcRoute.backendTLSPolicy.caCertificateConfigMapName: override ConfigMap name
  • grpcRoute.backendTLSPolicy.hostname: override backend validation hostname
  • server.tls.enableMtls: must be false for BackendTLSPolicy
  • pkiInitJob.timeoutSeconds: polling duration for cert-manager mode
  • pkiInitJob.failOnTimeout: fail install if cert-manager times out

Keycloak OIDC

Initial setup — rerun it whenever you want to rotate the development CA:

mise run keycloak:k8s:setup

This deploys Keycloak (quay.io/keycloak/keycloak:24.0) into the keycloak namespace, imports the openshell realm from scripts/keycloak-realm.json, generates a short-lived development TLS certificate, and publishes its trust anchor as the openshell-keycloak-ca ConfigMap in the OpenShell namespace. The command prints a port-forward command for acquiring tokens from the CLI. Rerunning setup rotates the development certificate and trust anchor; redeploy the gateway afterward so it reloads the mounted CA bundle. The chart renders the mount path as [openshell.gateway.oidc] ca_bundle; the gateway adds that issuer CA to native roots for OIDC discovery and JWKS requests without changing trust for other HTTPS clients. The mounted bundle must be a regular file no larger than 1 MiB; the gateway rejects other file types and oversized bundles during startup.

Then activate OIDC in the OpenShell Helm chart:

  1. Uncomment #- ci/values-keycloak.yaml in skaffold.yaml
  2. Redeploy: mise run helm:skaffold:run

To remove Keycloak:

mise run keycloak:k8s:teardown

SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate SPIFFE JWT-SVIDs for dynamic provider token grants:

  1. Uncomment the spire-crds and spire releases in deploy/helm/openshell/skaffold.yaml
  2. Uncomment #- ci/values-spire.yaml in the OpenShell release values files
  3. Redeploy: mise run helm:skaffold:run

ci/values-spire-stack.yaml configures the local SPIRE trust domain as openshell.local and adds a ClusterSPIFFEID that maps sandbox pod annotations to spiffe://openshell.local/openshell/sandbox/<sandbox-id>. OpenShell mounts the SPIFFE CSI Workload API socket at /spiffe-workload-api/spire-agent.sock only into supervisor Pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected Kubernetes compute driver validates the projected token before the gateway returns the current generation-bound session JWT. The driver also returns the runtime identity recorded during provisioning. Restart preserves its namespace and Sandbox CR UID, rejects ambiguous label matches, and rotates only the supervisor Pod UID; bootstrap fails when the live identity does not match the durable sandbox record.

Vault Credential Driver

The credential-driver-vault Skaffold profile applies ci/values-credential-driver-vault.yaml. Its external OpenBao/Vault backend must expose HTTPS at the configured service DNS name and publish the issuing CA certificate as the ca.crt key in the openbao-ca ConfigMap. Local e2e uses OpenBao dev TLS and an openbao-0 DNS alias matching its generated certificate. The Helm value server.credentialDrivers.vault.caConfigMapName mounts that key into the gateway and renders the driver's ca_bundle setting. Non-loopback HTTP addresses fail gateway startup, and hostname verification requires the service DNS name in the server certificate SANs.

cd deploy/helm/openshell
skaffold run -p credential-driver-vault
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200

Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):

mise run helm:k3s:stop
mise run helm:k3s:start

Check cluster status:

mise run helm:k3s:status

Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or Skaffold inputs:

mise run helm:lint

If Helm reports missing chart dependencies, remove the specific stale subchart archive or directory named by the error from deploy/helm/openshell/charts/, then rerun the lint task.

For example, when lint reports chart metadata is missing these dependencies: postgresql, remove stale PostgreSQL chart artifacts:

rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint

The charts/ directory is ignored and regenerated by helm dependency build for dependencies still declared in Chart.yaml.


Key Files

Path Purpose
deploy/helm/openshell/skaffold.yaml Skaffold config — images, Helm releases, values overlays
deploy/helm/openshell/values.yaml Default Helm values
deploy/helm/openshell/ci/values-skaffold.yaml Dev overrides (image pull policy, TLS disabled for local Skaffold)
deploy/helm/openshell/ci/values-local-tracing.yaml OTLP export to the helm:k3s:create collector (Skaffold only)
deploy/helm/openshell/ci/values-cert-manager.yaml cert-manager PKI overlay (opt-in; disables pkiInitJob)
deploy/helm/openshell/ci/values-gateway.yaml Envoy Gateway GRPCRoute + Gateway overlay
deploy/helm/openshell/ci/values-high-availability.yaml HA test overlay (replicaCount: 2 with external PostgreSQL Secret)
deploy/helm/openshell/ci/values-supervisor.yaml, ci/values-supervisor-ha.yaml, ci/values-supervisor-ha-scaling.yaml Shared supervisor pool overlays: one member, two members (Skaffold profile shared-supervisor-high-availability), or one member to scale to two and back
deploy/helm/openshell/ci/values-tls-port-forward.yaml Keeps gateway TLS on over the Skaffold values for e2e topologies that need it (OPENSHELL_E2E_KUBE_TLS=1)
deploy/helm/openshell/ci/values-autoscaling.yaml Render-only overlay for the optional gateway HorizontalPodAutoscaler (helm lint and helm-unittest)
deploy/helm/openshell/ci/values-keycloak.yaml Keycloak OIDC overlay
deploy/helm/openshell/ci/values-spire.yaml SPIFFE/SPIRE provider token grant overlay
deploy/helm/openshell/ci/values-spire-stack.yaml SPIRE hardened chart values for local dev
deploy/helm/openshell/ci/values-tls-disabled.yaml Lint-only: TLS + auth disabled (reverse-proxy edge termination)
deploy/helm/openshell/ci/values-credential-driver-vault.yaml Vault credential-driver validation overlay with HTTPS and private-CA trust
deploy/kube/manifests/envoy-gateway-openshell.yaml GatewayClass and BackendTrafficPolicy for Envoy Gateway (mise run helm:gateway:apply)
tasks/scripts/helm-k3s-local.sh k3d cluster create/delete/start/stop/status
tasks/scripts/keycloak-k8s-setup.sh Keycloak deploy, realm import, and development TLS trust anchor

Sandbox update regression lane

Run mise run e2e:kubernetes:updates for the testkit update suite against both supported Agent Sandbox APIs with the combined Helm chart and its bundled workspace resources. It verifies stopped-only updates, retained CR/PVC identity and contents, command/environment changes, and request replay. OPENSHELL_E2E_UPDATE_ORIGINAL_IMAGE and OPENSHELL_E2E_UPDATE_TARGET_IMAGE can select mirrored Alpine 3.21 and 3.22 manifests. Defaults use the public registry cache; keep immutable digests when overriding them.

App listener fixture

The opt-in deploy/helm/openshell/ci/values-app-listener.yaml enables the supervisor listener at port 9443 without discovery or a proxy. Provision the operator TLS Secret and an ingress-client NetworkPolicy first. Follow e2e/kubernetes/app-listener/README.md for direct in-cluster verification, shared-member selection, and the HA follow-up checks. Keep the fixture disabled in default profiles. Dedicated and shared supervisors expose app ports on their Pod IPs without app Services. Select the member owning the sandbox; automatic owner selection belongs to the later discovery API.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts6mo3 checks · Risk SAFE
  • Gen Agent Trust Hub6mo

    This skill provides a set of instructions and automation tasks for managing a local Kubernetes development environment using standard tools like k3d, Helm, and Skaffold. It automates cluster setup, application deployment, and local authentication configuration. All operations and external dependencies (such as Keycloak from quay.io) are consistent with legitimate development workflows and well-known services.

  • Socket6mo

    No alerts

  • Snyk6mo

    Risk: LOW · No issues

Signed by skilld at c24fcd2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
metadata
{
  "internal": true
}

README badge

README badge for nvidia/openshell/helm-dev-environment