---
name: helm-dev-environment
description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.
metadata:
  internal: true
---

# Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell.
The stack is: **k3d** (Docker-backed k3s) for the cluster, **Skaffold** for image builds and Helm deploys, and the **OpenShell Helm chart** (`deploy/helm/openshell/`).

---

## Prerequisites

- Docker Desktop (macOS) or Docker Engine (Linux) running
- `mise install` completed (provides `k3d`, `kubectl`, `skaffold`, `helm`)

---

## Startup

### 1. Create the cluster

```bash
mise run helm:k3s:create
```

Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
When the named cluster already exists, the task starts any stopped containers and refreshes
same-named kubeconfig entries so a recreated load balancer's current API port takes effect.
Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION`
in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox`
releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace
collector and UI in the `observability` namespace,
and preloads the default sandbox image into k3d so the first sandbox create
does not wait on a large registry pull. Traefik is disabled at cluster creation time.

**Multi-worktree support:** the cluster name is derived from the last component of the
current git branch (e.g. branch `chore/1234-local-dev/octocat` → cluster
`openshell-dev-octocat`). Each worktree therefore gets its own isolated cluster and its
own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name
or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

| Host port | Target | Used by |
|-----------|--------|---------|
| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) |

Override with env vars before running `helm:k3s:create`:
- `HELM_K3S_LB_HOST_PORT` (default: `8080`)
- `HELM_K3S_PRELOAD_SANDBOX_IMAGE` (default:
  `nvcr.io/nvidia/base/ubuntu:24.04`; set to an empty value to skip)
- `HELM_K3S_COLLECTOR_IMAGE` (default:
  `mcr.microsoft.com/dotnet/aspire-dashboard:latest`)
- `HELM_K3S_COLLECTOR_HEALTH_TIMEOUT` (default: `120` seconds)

### Gateway metrics

The chart enables its dedicated Prometheus endpoint by default. Set
`metrics.enabled=false` to omit the listener and its container and Service
ports without changing `service.metricsPort` (default `9090`). The port must
remain between 1 and 65535 even when disabled. Legacy `metricsPort: 0` or
`false` values fail rendering; migrate them to `metrics.enabled=false` and a
valid port in the same upgrade. `metrics.tls.enabled=true` requires
`metrics.enabled=true`. This setting does not disable OpenTelemetry export.

### Shared supervisor plumbing

Set `gatewayConfig.openshell.drivers.kubernetes.supervisor_topology`
using a values file (the table key is literally `openshell.drivers.kubernetes`):

```yaml
gatewayConfig:
  openshell.drivers.kubernetes:
    supervisor_topology: gateway
```

The gateway chart creates a shared supervisor Deployment (`supervisor.replicas`,
default one), Service, NetworkPolicy, and an empty
`<release>-supervisor-token` Secret in
the release namespace, regardless of shared/managed/operator workspace mode.
The gateway mints a supervisor token from its sandbox-JWT signing key, writes
it into that Secret (the gateway Role may `get`/`patch` only that Secret), and
renews it at half its lifetime; the supervisor mounts the Secret and presents
the token on the private `openshell.supervisor.host.v1` assignment stream,
which accepts nothing else. The topology requires gateway TLS and the signing
key (`server.disableTls` is rejected; the gateway refuses to start without
`gateway_jwt`). New sandbox generations run as sandbox instances in that
process. The chart starts `--mode=shared`, sets `--gateway-endpoint` to the
gateway gRPC URL, mounts the token at `--token-file`, and binds `--address`
(5501) to a plain TCP health listener for the `tcpSocket` probes; nothing
dials the supervisor but sandbox runtimes (5500) and the kubelet. Build the gateway, supervisor, and sandbox
images from the same branch: all three must support the routed runtime
connection and the assignment stream. `workspace` is reserved and rejected.
Omit the field to disable it. `mise run
e2e:kubernetes:shared-supervisor` runs the focused e2e
(`ci/values-supervisor.yaml`). The task prepends that overlay to any
colon-separated `OPENSHELL_E2E_KUBE_EXTRA_VALUES` list, so layer a
workspace-mode overlay to cover managed namespaces:

```bash
OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-workspace-managed.yaml \
OPENSHELL_E2E_KUBE_IMAGE_PULL_SECRET=e2e-regcred \
  mise run e2e:kubernetes:shared-supervisor
```

`mise run e2e:kubernetes:shared-supervisor-spiffe` installs pinned SPIRE charts
(`ci/values-spire-broker-stack.yaml`) and the `openshell.gateway.spiffe_broker`
overlay (`ci/values-supervisor-spiffe.yaml`). It exercises provider exchange and
abrupt supervisor takeover over TLS, using Sandbox CR UID registrations and
enforced Kubernetes authorization. Workload generation, Pod, process, and stable
SPIFFE identity survive; the new owner acquires a fresh provider token and the
broker rejects the old bearer. The fixture's 30-second workload X.509-SVIDs
exercise acquisition after credential rotation. The gateway Pod must pass
Restricted Pod Security admission. The task owns its SPIRE install and cluster.

The task creates three nodes (`HELM_K3S_AGENT_COUNT=2` adds two k3d agents) and
requires three SPIRE agents. Its cluster-scoped `server_spiffe_id_prefix` accepts
the concrete node identities, and a node alias distributes Sandbox registrations
to all eligible agents. The test selects each Broker backend in turn and forces
a new generation's acquisition, then replaces one agent and checks acquisition
without re-registering the sandbox. Exact-ID mode remains available for a fixed
endpoint. The installed SPIRE version and operator must expose the experimental
Broker API.

For remote clusters, set `OPENSHELL_E2E_HOST_GATEWAY_IP` to a host reachable
from Pods. Optional `OPENSHELL_E2E_SPIFFE_STS_PORT` and
`OPENSHELL_E2E_SPIFFE_TARGET_PORT` fix the test's host listener ports so an
operator can forward them through a reverse tunnel. Keep the tunnel available
through the test and remove it afterward. Use an explicit kubeconfig and context
to avoid repository environment defaults targeting a local cluster.
`OPENSHELL_E2E_SPIFFE_DRIVER_CONFIG_JSON` can pass a driver override to test
sandbox creation, such as a node selector for a Landlock-capable pool. Enable
`allow_driver_config` in the test gateway when using this override. GKE COS
nodes may lack Landlock; use compatible Ubuntu nodes for sandbox workloads.
For remote abrupt-failover coverage, set
`OPENSHELL_E2E_SUPERVISOR_PROCESS_CONTROL=1`. Before creating sandboxes, the test
adds an unprivileged process-control sidecar and enables the supervisor Pod's
shared process namespace. The sidecar sends SIGKILL to the supervisor as the
same UID; it needs no host PID namespace, capabilities, or runtime socket.
The default k3d/kind test continues to kill through the node container runtime.

### Shared supervisor high availability

`mise run e2e:kubernetes:shared-supervisor-spiffe-node-local` validates direct
node-local Broker TCP routing. The chart injects `OPENSHELL_NODE_IP` from
`status.hostIP` for `spiffe_broker.node_local_port`; it creates no Broker Service.
The task uses an external PostgreSQL fixture and a gateway Deployment, moves
the gateway across the SPIRE nodes, restricts Broker egress to its current node,
and forces fresh token acquisition on each node and after local agent replacement.
The shared-supervisor failover, credential rotation, and identity fencing checks
also run. Keep gateway/supervisor/sandbox images from the same branch.

The shared supervisor is always a pool of `supervisor.replicas` members
(default 1), or an HPA-managed pool when `supervisor.autoscaling.enabled=true`.
`ci/values-supervisor-autoscaling.yaml` enables the memory HPA (default `512Mi`
average per Pod, two to four members, one removal every two minutes after five
minutes of stabilization). It requires the resource metrics API, usually
Metrics Server. The HPA omits the Deployment's fixed replica count and ignores
`supervisor.replicas`; top-level `autoscaling` still controls only the gateway.
Do not combine the HPA overlay with the manual 1→2→1 scaling E2E: that test
expects Helm to own replicas. Changing HPA bounds or target must leave the
gateway config and supervisor Pod template unchanged, as checked by
`deploy/helm/test-supervisor-pool-scaling.sh`. With the gateway topology the
chart renders the gateway's
`[openshell.gateway.supervisor_pool]` tuning table (only
`recovery_timeout_secs` from `supervisor.recoveryTimeoutSeconds`, default 60;
the driver's `shared_supervisor` table stays unchanged), a
RollingUpdate Deployment (`maxSurge: 1`, `maxUnavailable: 0`) with a 60-second
grace period, no affinity unless `supervisor.affinity` is set (top-level
`affinity` applies only to the gateway), the pool identity env vars
(`OPENSHELL_SUPERVISOR_POOL_MEMBER_ID` and `_ADVERTISE_IP` from the Pod name
and IP), and a `maxUnavailable: 1` PodDisruptionBudget unless
`supervisor.podDisruptionBudget.enabled=false` (default `true`; toggling it
changes only the PodDisruptionBudget). Each supervisor resource has its own
template: `supervisor.yaml` (Deployment), `supervisor-service.yaml`,
`supervisor-networkpolicy.yaml`, `supervisor-token-secret.yaml`,
`supervisor-pdb.yaml`, and `supervisor-hpa.yaml`.
`ci/values-supervisor-ha.yaml` selects two replicas and a 60-second recovery
timeout. The topology needs gateway TLS: the e2e harness keeps it on with
`ci/values-tls-port-forward.yaml` and `OPENSHELL_E2E_KUBE_TLS=1`, which reaches
the gateway over mTLS through its port-forward. `mise run
e2e:kubernetes:shared-supervisor-ha` sets both, creates a task-scoped k3d
cluster, kills the supervisor Pod that owns a sandbox, and checks that the
sandbox keeps its Pod, generation, main process, and in-memory state.

The rendered gateway configuration and supervisor Pod template do not
depend on `supervisor.replicas`; `deploy/helm/test-supervisor-pool-scaling.sh`
(part of `mise run helm:test`) renders a pool at 1, 2, and 3 replicas and
fails if anything but the replica count differs. `mise run
e2e:kubernetes:shared-supervisor-scaling` installs a one-member pool
(`ci/values-supervisor-ha-scaling.yaml`), then the test runs `helm upgrade
--reuse-values --set supervisor.replicas=2` and back to 1 against the chart
the harness installed (`OPENSHELL_E2E_KUBE_CHART`), probing exec and
policy-mediated networking throughout. It checks that scaling up moves
nothing, that the member removed by scaling down hands its sandboxes to the
other (`cause=leaving`), that no workload, gateway Pod, or remaining member
restarts, and that a new sandbox runs after each step. It upgrades its
release, so it runs on a release of its own; the two pool tests also share a
nextest test group that never runs them at once. Against an existing cluster
whose kubeconfig the shell environment may override, set
`OPENSHELL_E2E_KUBECONFIG` (passed as `--kubeconfig` to kubectl and helm).

On k3d, deploy the pool with the Skaffold profile, which layers the overlay and
turns gateway TLS back on over `ci/values-skaffold.yaml`:

```bash
cd deploy/helm/openshell
KUBECONFIG=../../../kubeconfig skaffold run -p shared-supervisor-high-availability
cd ../../..
```

The plaintext registration from `helm:skaffold:run` does not apply. Keep
`mise run helm:k3s:forward` running and set up the client certificate as in
"With mTLS enabled" under
[Connecting through the forwarding task](#connecting-through-the-forwarding-task).
Both members share the single k3d node, so exercise Pod failures:

```bash
export KUBECONFIG=kubeconfig OPENSHELL_GATEWAY_ENDPOINT=https://localhost:8090
openshell sandbox create --detach --name ha-probe -- sh -c 'echo $$ > /sandbox/main.pid; exec sleep infinity'
openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid
openshell sandbox get ha-probe  # note the sandbox ID
# Supervisor logs name the instance by its route, not its sandbox ID.
kubectl get sandboxes.agents.x-k8s.io -A -l openshell.ai/sandbox-name=ha-probe \
  -o jsonpath='{.items[0].metadata.annotations.openshell\.ai/sandbox-runtime-supervisor-session}'
# The member whose latest admission, removal, or release line names the route owns it.
kubectl -n openshell logs -l app.kubernetes.io/name=openshell-supervisor --prefix --tail=-1 \
  | grep -E '(admitted assigned|removing withdrawn|relinquishing a) sandbox instance.* <route>$'
# Force-delete the owner, the closest local stand-in for a crash.
kubectl -n openshell delete pod <owner-pod> --grace-period=0 --force
kubectl -n openshell logs -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
  -c openshell-gateway --prefix --tail=200 | grep 'taking over a shared sandbox instance'
kubectl -n openshell logs -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload \
  --tail=200 | grep -E 'sandbox-supervisor-|Supervisor takeover confirmed'
openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid  # same PID
```

A plain `kubectl delete pod` exercises the graceful leave instead: the member
announces it is leaving and the gateway hands its instances over at once. A
member that stops without announcing it, as on node loss, is declared dead
only after the 15-second membership TTL.
`kubectl -n openshell rollout restart deployment/openshell-supervisor` replaces
every member one at a time. The PID must survive each case, while an open
`sandbox connect` or `exec` stream breaks and must reconnect. A
`sandbox-supervisor-recovery-expired` finding in the workload log means no
replacement confirmed within the recovery timeout, so the runtime terminated
the workload.

For RFC 0017 app-listener validation, enable the listener and supply the app TLS
Secret on the shared Deployment, plus a separate NetworkPolicy allowing the test
client/proxy to reach its app port. Runtime-port access does not grant app access.
Discovery returns the acknowledged owner's Pod IP and app port; no headless app
Service is needed. Exercise HTTP/WebSocket reconnection through graceful downscale
and abrupt loss, preserving workload generation/PID/in-memory state and the public
hostname. Verify stale-owner rejection independently of the replacement's success.
Shared serving leases expire within fifteen seconds of issuance and do not renew
from configuration alone. Do not claim seamless socket migration or infer external
ingress readiness from an advertisement.

### 2. Deploy OpenShell

**Iterative dev** (rebuilds on file changes, recommended during active development):
```bash
mise run helm:skaffold:dev
```

**One-shot deploy** (build once and leave running):
```bash
mise run helm:skaffold:run
```

Resource admission defaults to enabled and caller driver config to disabled.
Driver-config scenarios need an explicit `allowDriverConfig` opt-in; external
attachments also need administrator-controlled approval labels in the target
namespace. GPU attachments and operator-selected image-pull Secrets are exempt
from labels. Managed workspace image-pull Secrets are copied from the configured
source in the gateway namespace; do not grant approval to the gateway database
PVC or disable admission to make tests pass.

The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
and deploys the OpenShell Helm chart. The Kubernetes driver creates a
capability-free workload Pod and a directly managed capability-free supervisor
Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
workload Pod. New sandbox generations use `runtime-initiated` control: the sandbox
runtime dials its supervisor through `os-supervisor-<id>`. The legacy
`os-boundary-<id>` Service keeps selecting the workload for old gateways during
rolling upgrades. New gateways create both Services and bind the control-mode
annotation to its runtime generation. The driver adds a per-sandbox
`os-control-<id>` NetworkPolicy for that path only. Existing running generations
keep their connection until replaced; stop/start and gateway recovery that
recreates both Pods upgrade legacy generations automatically. Use compatible
gateway, supervisor, and sandbox runtime images together; there is no legacy
rollout flag. The
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`)
generates gateway and CLI TLS secrets on first install. Supervisor Pods project
only `ca.crt` and authenticate gateway RPCs with sandbox bearer tokens. User
client certificates and private keys remain outside supervisor and workload Pods.
The default Skaffold values export
gateway and Kubernetes-driver traces to the collector service installed by
`helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
the unified local forwarding task:

```bash
mise run helm:k3s:forward
```

The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to
`http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it
also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues
with the collector ports only. A successful plaintext `helm:skaffold:run`
registers the gateway under the worktree-specific k3d
cluster name and selects it as the active gateway. Keep the forwarding
task running while using those endpoints.

### Viewing local traces

The gateway exports OTLP/gRPC to
`http://openshell-collector.observability.svc.cluster.local:4317` through
`ci/values-local-tracing.yaml`, which only `skaffold.yaml` layers. The Kubernetes
E2E wrapper omits it, so E2E clusters export no traces. Forward OTLP/gRPC and the trace UI to the host:

```bash
mise run helm:k3s:forward
```

Open `http://127.0.0.1:18888` and exercise the gateway to inspect gateway and
Kubernetes compute-driver spans under their distinct service names, along with
Agent Sandbox controller reconciliation spans linked through the Sandbox
trace-context annotation. The same command exposes OTLP/gRPC on
`http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on
`http://127.0.0.1:8090`. The local `gateway`, `gateway:docker`,
`gateway:podman`, and `gateway:vm` tasks enable trace export, OCSF log export,
and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` is set, for
example `OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317`.
Spans an agent exports from inside a sandbox arrive through the supervisor's
relay with `openshell.telemetry.source=agent` and share `openshell.sandbox.id`
with that sandbox's supervisor spans.

The Skaffold profile for HA reverse-proxy development is available from
`deploy/helm/openshell/`:

```bash
# Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route.
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
```

The `high-availability` profile expects a Secret named `openshell-ha-pg` in the `openshell`
namespace with a `uri` key. For local manual testing, either create your own
PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in
`e2e/kubernetes/postgres-fixture.yaml`.

For the `high-availability` profile, return to the repository root and apply the
GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed
Envoy Gateway:

```bash
KUBECONFIG=kubeconfig mise run helm:gateway:apply
```

The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for
OpenShell's `GRPCRoute`. Keep that policy in `deploy/kube/manifests/envoy-gateway-openshell.yaml`,
not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay
streams during gateway rollouts and scale events.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS.

| Mode | `server.disableTls` | Gateway scheme |
|------|---------------------|----------------|
| Skaffold dev (default) | `true` | `http://` |
| TLS enabled | `false` (or omitted) | `https://` |

### Connecting through the forwarding task

Port `8080` is already bound by the k3d load balancer when Envoy Gateway is
active, so the forwarding task uses local port `8090` for the gateway. In a
second terminal, confirm that the gateway is registered and active:

```bash
openshell gateway list
```

**Plaintext (default Skaffold deploy):**

```bash
openshell sandbox list
```

**With mTLS enabled** — extract the client cert the PKI hook wrote to the cluster,
then place it where the CLI expects it. Run once after each fresh install:

```bash
mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key
```

The server cert SANs include `localhost` and `127.0.0.1`, so hostname verification
passes over a port-forward without any extra flags:

```bash
openshell sandbox list --gateway-endpoint https://localhost:8090
```

---

## Teardown

### Remove the Helm releases (keep cluster)

```bash
mise run helm:skaffold:delete
```

### Delete the cluster entirely

```bash
mise run helm:k3s:delete
```

This removes the k3d cluster and all resources. Kubeconfig context is left behind
but will point to a deleted cluster — safe to ignore or clean up manually.

---

## Optional Add-ons

Some add-ons can be enabled by uncommenting values in `skaffold.yaml`, but prefer
the dedicated Skaffold profiles when they exist. Profiles avoid leaving local
manual edits in the worktree.

### Envoy Gateway (Gateway API / GRPCRoute)

Use the `high-availability` Skaffold profile for HA reverse-proxy testing. The
profile intentionally includes Envoy Gateway so multi-replica behavior is
exercised through the same Gateway API path used by reverse-proxy deployments:

```bash
cd deploy/helm/openshell
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
cd ../../..
KUBECONFIG=kubeconfig mise run helm:gateway:apply
```

`values-gateway.yaml` creates the `openshell` `Gateway` (listener on port 80,
class `eg`) and an `openshell` `GRPCRoute` in the `openshell` namespace. The
`high-availability` profile
installs the Envoy Gateway Helm chart and layers both
`values-high-availability.yaml` and `values-gateway.yaml` onto the OpenShell
release.

`deploy/kube/manifests/envoy-gateway-openshell.yaml` creates:

- `GatewayClass/eg`
- `BackendTrafficPolicy/openshell-grpc-timeouts`

The Envoy Gateway proxy Service is usually exposed through the k3d load balancer
at `http://127.0.0.1:8080`. If the cluster was created with a different
`HELM_K3S_LB_HOST_PORT`, use that host port instead.

For manual tests against an existing cluster, prefer forwarding the Envoy proxy
Service rather than `svc/openshell`. That keeps client traffic on the same path
as a real reverse proxy while gateway pods rotate behind it:

```bash
KUBECONFIG=kubeconfig kubectl get svc -A \
  -l gateway.envoyproxy.io/owning-gateway-name=openshell
KUBECONFIG=kubeconfig kubectl -n <envoy-service-namespace> port-forward \
  svc/<envoy-service-name> 8080:80
openshell gateway add http://127.0.0.1:8080 --name openshell --local
```

When running e2e tests manually through Envoy, register gateway metadata (as
above) instead of relying only on `OPENSHELL_GATEWAY_ENDPOINT`; some tests call
`openshell gateway info` and expect metadata for the active gateway.

### Kubernetes E2E Notes

Use `mise run e2e:kubernetes` for the standard Helm-backed Kubernetes suite.
Set `OPENSHELL_E2E_KUBE_TEST=kubernetes_image_update` to scope the Rust suite to
combined image, command, environment, and metadata updates with retained sandbox,
SSH, and PVC identities. The next start verifies the new argv and environment. This target is
included in both existing Agent Sandbox API-version lanes. Docker separately
supports image-only updates on stopped, opt-in persistent `/sandbox` workspaces;
this Kubernetes lane does not qualify that path. Docker requires locally cached
images, a matching numeric target `USER UID:GID`, and no image-declared volumes.
Do not treat a skipped Docker lane as live image-replacement coverage.

The separate `docker-updates` testkit suite qualifies the full Docker path
through an owned mTLS gateway, including SSH exec, image replacement, restart,
credential-publication repair and deletion. Follow
`tests/suites/features/docker-updates/README.md`; the wrapper's
`OPENSHELL_E2E_DOCKER_IMAGE_UPDATES=1` opts in only its fresh gateway and refuses
an existing endpoint. On Docker Desktop without host networking, select the
documented bridge override rather than changing engine settings.

Image updates default to disabled outside test fixtures. The e2e wrapper opts
in only for a fresh gateway fixture that it owns; set
`OPENSHELL_E2E_KUBE_IMAGE_UPDATES=0` to prevent that opt-in. Do not propagate
activation to an existing shared gateway or enable it during a mixed-version
rollout. For manual testing, follow the retirement and activation checklist in
`docs/kubernetes/high-availability.mdx` before setting
`gatewayConfig: {"openshell.gateway": {enable_sandbox_updates: true}}`.
The gateway process equivalents are `--enable-sandbox-updates true` and
`OPENSHELL_ENABLE_SANDBOX_UPDATES=true`. Retire every old process and Job,
settle outstanding driver work, prevent old binaries returning, and do not
treat PostgreSQL connection markers or membership TTL expiry as retirement
proof. SQLite testing supports one gateway process only. Image-update clients
must call the separate `SandboxUpdates` service on the existing gateway
listener, not add a method to the baseline `OpenShell` service or inspect new
public `SandboxStatus` image fields. Resume a pending operation with
`SandboxUpdates.UpdateSandbox`, the current resource version, and no selected
fields, following the documented request-ID semantics.


For the opt-in application listener, build/import matching branch images and run
`mise run e2e:kubernetes:app-listener`; see `e2e/kubernetes/app-listener/README.md`
for image, topology, delivery-mode and takeover inputs. The task owns an isolated
Helm release/namespace, temporary TLS material, in-cluster clients, teardown and
evidence under `results/app-listener/`. It checks policy denial, Pod-IP ingress without app Services, shared-owner takeover, gateway-outage expiry and disabled-listener
gateway compatibility, without discovery or a proxy. CI runs four cases under
`test:e2e` and includes them in `Core E2E result`. Its reusable kind setup installs
pinned Calico only for this lane; other jobs retain kindnet. Local k3s already
enforces policy. The lower-level `verify.py` can target a predeployed release.
The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
longer forwards the unauthenticated health listener or runs a `/readyz` e2e
target. `/readyz` remains covered by server unit/integration tests.

Use `mise run e2e:kubernetes:ha-rebalancing` for full-suite HA coverage. The
task creates an external PostgreSQL fixture, installs Envoy Gateway, applies
`deploy/kube/manifests/envoy-gateway-openshell.yaml`, enables the chart
`GRPCRoute`, and runs the full Kubernetes e2e suite, including
`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and
exec through the Envoy proxy while gateway replicas scale up, scale down, and
rotate. It also keeps a long-running sandbox alive and runs upload/download
operations while gateway pods roll, so file sync exercises the same relay retry
path as interactive sessions.

If you reuse an existing Skaffold cluster for the full kube suite, make sure the
chart has `server.hostGatewayIP` set so sandbox pods can resolve
`host.openshell.internal` back to the test host. The e2e wrapper detects this on
chart installs; manual reuse may require:

```bash
HOST_GATEWAY_IP="${OPENSHELL_E2E_HOST_GATEWAY_IP:?set host gateway IP}"
KUBECONFIG=kubeconfig helm upgrade openshell deploy/helm/openshell \
  --namespace openshell --reuse-values \
  --set "server.hostGatewayIP=${HOST_GATEWAY_IP}" \
  --wait --timeout 5m
```

Use the IP that pods in that cluster use to reach listeners on the test host.

### BackendTLSPolicy (end-to-end TLS)

To enable end-to-end TLS between the Gateway proxy and the gateway pod, add
BackendTLSPolicy values to the Helm install:

```bash
helm upgrade --install openshell deploy/helm/openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.backendTLSPolicy.enabled=true \
  --set server.tls.enableMtls=false \
  ...
```

This requires `server.tls.enableMtls=false` because ingress proxies cannot
present client certificates to the backend. The certgen hook creates a backend
CA ConfigMap from the server Secret's `ca.crt` key. With cert-manager, a
separate post-install Job polls for the cert-manager-issued certificate (up to
`pkiInitJob.timeoutSeconds`); with built-in PKI the ConfigMap is created in the
same pre-install hook. The ConfigMap is reconciled on every upgrade so CA
rotations propagate automatically.

Key Helm values:
- `grpcRoute.backendTLSPolicy.enabled`: create the BackendTLSPolicy resource
- `grpcRoute.backendTLSPolicy.caCertificateConfigMapName`: override ConfigMap name
- `grpcRoute.backendTLSPolicy.hostname`: override backend validation hostname
- `server.tls.enableMtls`: must be `false` for BackendTLSPolicy
- `pkiInitJob.timeoutSeconds`: polling duration for cert-manager mode
- `pkiInitJob.failOnTimeout`: fail install if cert-manager times out

### Keycloak OIDC

Initial setup — rerun it whenever you want to rotate the development CA:

```bash
mise run keycloak:k8s:setup
```

This deploys Keycloak (`quay.io/keycloak/keycloak:24.0`) into the `keycloak` namespace,
imports the openshell realm from `scripts/keycloak-realm.json`, generates a short-lived
development TLS certificate, and publishes its trust anchor as the
`openshell-keycloak-ca` ConfigMap in the OpenShell namespace. The command prints a
port-forward command for acquiring tokens from the CLI. Rerunning setup rotates the
development certificate and trust anchor; redeploy the gateway afterward so it reloads
the mounted CA bundle. The chart renders the mount path as
`[openshell.gateway.oidc] ca_bundle`; the gateway adds that issuer CA to native roots
for OIDC discovery and JWKS requests without changing trust for other HTTPS clients.
The mounted bundle must be a regular file no larger than 1 MiB; the gateway rejects
other file types and oversized bundles during startup.

Then activate OIDC in the OpenShell Helm chart:
1. Uncomment `#- ci/values-keycloak.yaml` in `skaffold.yaml`
2. Redeploy: `mise run helm:skaffold:run`

To remove Keycloak:
```bash
mise run keycloak:k8s:teardown
```

### SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate
SPIFFE JWT-SVIDs for dynamic provider token grants:

1. Uncomment the `spire-crds` and `spire` releases in `deploy/helm/openshell/skaffold.yaml`
2. Uncomment `#- ci/values-spire.yaml` in the OpenShell release values files
3. Redeploy: `mise run helm:skaffold:run`

`ci/values-spire-stack.yaml` configures the local SPIRE trust domain as
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
OpenShell mounts the SPIFFE CSI Workload API socket at
`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
Kubernetes compute driver validates the projected token before the gateway
returns the current generation-bound session JWT. The driver also returns the
runtime identity recorded during provisioning. Restart preserves its namespace
and Sandbox CR UID, rejects ambiguous label matches, and rotates only the
supervisor Pod UID; bootstrap fails when the live identity does not match the
durable sandbox record.

### Vault Credential Driver

The `credential-driver-vault` Skaffold profile applies
`ci/values-credential-driver-vault.yaml`. Its external OpenBao/Vault backend
must expose HTTPS at the configured service DNS name and publish the issuing CA
certificate as the `ca.crt` key in the `openbao-ca` ConfigMap. Local e2e uses
OpenBao dev TLS and an `openbao-0` DNS alias matching its generated certificate.
The Helm value
`server.credentialDrivers.vault.caConfigMapName` mounts that key into the
gateway and renders the driver's `ca_bundle` setting. Non-loopback HTTP
addresses fail gateway startup, and hostname verification requires the service
DNS name in the server certificate SANs.

```bash
cd deploy/helm/openshell
skaffold run -p credential-driver-vault
kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200
```

---

## Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):
```bash
mise run helm:k3s:stop
mise run helm:k3s:start
```

Check cluster status:
```bash
mise run helm:k3s:status
```

---

## Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or
Skaffold inputs:

```bash
mise run helm:lint
```

If Helm reports missing chart dependencies, remove the specific stale subchart
archive or directory named by the error from `deploy/helm/openshell/charts/`,
then rerun the lint task.

For example, when lint reports `chart metadata is missing these dependencies:
postgresql`, remove stale PostgreSQL chart artifacts:

```bash
rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint
```

The `charts/` directory is ignored and regenerated by `helm dependency build`
for dependencies still declared in `Chart.yaml`.

---

## Key Files

| Path | Purpose |
|------|---------|
| `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays |
| `deploy/helm/openshell/values.yaml` | Default Helm values |
| `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
| `deploy/helm/openshell/ci/values-local-tracing.yaml` | OTLP export to the `helm:k3s:create` collector (Skaffold only) |
| `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-supervisor.yaml`, `ci/values-supervisor-ha.yaml`, `ci/values-supervisor-ha-scaling.yaml` | Shared supervisor pool overlays: one member, two members (Skaffold profile `shared-supervisor-high-availability`), or one member to scale to two and back |
| `deploy/helm/openshell/ci/values-tls-port-forward.yaml` | Keeps gateway TLS on over the Skaffold values for e2e topologies that need it (`OPENSHELL_E2E_KUBE_TLS=1`) |
| `deploy/helm/openshell/ci/values-autoscaling.yaml` | Render-only overlay for the optional gateway HorizontalPodAutoscaler (helm lint and helm-unittest) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
| `deploy/helm/openshell/ci/values-credential-driver-vault.yaml` | Vault credential-driver validation overlay with HTTPS and private-CA trust |
| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass and BackendTrafficPolicy for Envoy Gateway (`mise run helm:gateway:apply`) |
| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy, realm import, and development TLS trust anchor |

## Sandbox update regression lane

Run `mise run e2e:kubernetes:updates` for the testkit update suite against both
supported Agent Sandbox APIs with the combined Helm chart and its bundled
workspace resources. It verifies stopped-only updates, retained CR/PVC
identity and contents, command/environment changes, and request replay.
`OPENSHELL_E2E_UPDATE_ORIGINAL_IMAGE` and `OPENSHELL_E2E_UPDATE_TARGET_IMAGE` can
select mirrored Alpine 3.21 and 3.22 manifests. Defaults use the public registry
cache; keep immutable digests when overriding them.

## App listener fixture

The opt-in `deploy/helm/openshell/ci/values-app-listener.yaml` enables the
supervisor listener at port 9443 without discovery or a proxy. Provision the
operator TLS Secret and an ingress-client NetworkPolicy first. Follow
`e2e/kubernetes/app-listener/README.md` for direct in-cluster verification,
shared-member selection, and the HA follow-up checks. Keep the fixture disabled
in default profiles. Dedicated and shared supervisors expose app ports on their
Pod IPs without app Services. Select the member owning the sandbox; automatic
owner selection belongs to the later discovery API.
