---
name: substrate
description: >-
  How AX depends on Agent Substrate (modules, address contract, mocks, version bumps, E2E pin). Load when touching internal/substrate, bumping Substrate, or triaging make test-e2e.
title: substrate
canonical_url: https://skilld.dev/gh/google/ax/substrate
last_updated: 2026-11-18T22:09:13.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> If the user asked to install this Skill, run `npx skilld install google/ax/substrate`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/substrate`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/google/ax/substrate) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/substrate`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/substrate --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/substrate --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Agent Substrate in AX

AX schedules every `Task` as a sandboxed **actor** on
[Agent Substrate](https://github.com/agent-substrate/substrate). AX owns the
declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns
isolation, checkpoint/restore, worker placement and in-cluster routing.
Facts below were verified against `main` on 2026-11-06 — re-check `go.mod`
and `hack/setup-e2e-kind.sh` if they look stale.

## 1. Module map

| Module | Pinned | Provides | Imported by |
|---|---|---|---|
| `go.ate.dev/substrate` | `v1.0.0` | Control API protos `api/ateapi/v1` (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) and `pkg/postgressetup` | `internal/substrate`, `internal/substrate/substratetest`, `internal/install/substrate`, `internal/server`, `internal/controller`, `internal/apivalidation`, `test/e2e`, `pkg/apis/v1alpha1` (its `.proto` files import `ateapi.proto`) |
| `github.com/agent-substrate/env` | `v0.0.11-0.20260912052224-4468a200b170` (pseudo-version) | Guest services served inside the sandbox (process exec + file system gRPC) and `proto/ateenv/v1alpha` | `internal/guest`, `internal/metadata` |

There is no `replace` directive for either module. Local hacking against a
Substrate checkout goes through `SUBSTRATE_DIR` in E2E (section 5).

## 2. The Control API wrapper: `internal/substrate.Client`

All Control API traffic goes through this package. It encapsulates:

- **Endpoint defaults** — `api.ate-system.svc.cluster.local:443`; authority
  and TLS `ServerName` default to `api.ate-system.svc` when the target
  contains that string.
- **Auth** — bearer token from `SUBSTRATE_TOKEN_FILE` (in-cluster: projected SA
  token with audience `api.ate-system.svc`, see `deploy/ax-server.yaml`), CA
  from `SUBSTRATE_CA_FILE`.
- **Retry** — `IsTransientError` (`Unavailable`, `DeadlineExceeded`, `Aborted`) + `callWithRetry` (`retryOnCodes`).
- **Helpers** the reconcilers rely on — `EnsureAtespace`, `EnsureActorTemplate`,
  `EnsureActorTemplateWithImage`, `BuildActorTemplate`, `EnsureActor`,
  `EnsureActorFromTag`, `GetActorEgressPolicy`, `EnsureActorEgressPolicy`,
  `CreateTag`, `GetTag`, `ListTags`, `DeleteTag`, `SuspendActor`,
  `SuspendActorAndWaitForSnapshot`, `ResumeActor`, `DeleteActor`, `WorkerIP`,
  `ResourceLimits`.

### Where a new Control API call lives today

- `internal/substrate/client.go` is the only non-test file that calls
  `ateapipb.NewControlClient`; every existing RPC is a method on `*Client`,
  wrapped in `callWithRetry` (`codes.Unavailable` plus optional codes such as `codes.Aborted`).
- Each method has a table-driven case in `client_test.go` (`mockControlClient`).
- Callers in `internal/controller` exercise it through `mockControlServer`
  (`reconciler_test.go`) or `internal/substrate/substratetest` (`workspace_test.go`);
  `internal/server` and `cmd/ax` tests also use `internal/substrate/substratetest`
  (section 4).

## 3. Address contract

| Value | Occurrences |
|---|---|
| `api.ate-system.svc.cluster.local:443` + authority `api.ate-system.svc` | `cmd/ax-server/main.go` (`--substrate-endpoint` / `--substrate-authority` flag defaults), `internal/substrate/client.go` (3×), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `test/e2e/harness_test.go` |
| `atenet-router.ate-system.svc[.cluster.local]` | `internal/controller/reconciler.go` (`defaultRouterServerName`), `internal/tunnel/router_tls.go` (`routerServerName`), `cmd/ax/main.go` (ssh port-forward path), `internal/guest/client.go` (doc comment), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `docs/networking.md` |
| namespace `ate-system` | all of the above + `test/e2e/harness_test.go` (`ateSystemNamespace`), `test/e2e/manifests/credential-policy.yaml`, `internal/install/axdeploy/authz.go`, `internal/install/cluster/kind.go`, `internal/install/substrate/bootstrap.go`, `internal/install/substrate/pki.go`, `hack/sync-substrate-manifests.sh` (and generated `internal/install/substrate/manifests/*.yaml`) |
| pod labels `ate.dev/worker-pool` (worker pods) and `app: atenet-egress` (egress gateway) | the `ax-server` NetworkPolicy in `deploy/ax-server.yaml` and `test/e2e/manifests/ax-server.yaml`, which blocks both from `:8080`/`:9090` and allows only `app: atenet-egress` (in `ate-system`) to reach `:8081` (`ax-server-runner`) |
| `opentelemetry-collector.otel-system.svc:4317` (the collector Substrate's kind overlay deploys, `manifests/ate-install/kind/ate-otel-config.yaml` in Substrate) | `test/e2e/manifests/ax-server.yaml` (`ax-otel-config` ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |

Why it matters: tasks never get a Service/Ingress. Every request to a task is
proxied by the **atenet router**, which picks the target from the
`ate-target-actor` header and resumes a suspended actor on demand
(`docs/networking.md`). `ax ssh` port-forwards to `svc/atenet-router` on 443
(mTLS) and falls back to 80.

## 4. Testing against Substrate

### Unit tests: in-process mock, no cluster

```go
type mockControlServer struct {
    ateapipb.UnimplementedControlServer
    // record calls / inject failures here
}
grpcServer := grpc.NewServer()
ateapipb.RegisterControlServer(grpcServer, mockSrv)
// serve on a listener, point substrate.NewClient at it
```

Canonical examples: `internal/controller/reconciler_test.go` (reconciler state
transitions; the largest test file in the repo). `internal/server/server_test.go`
and `cmd/ax/main_test.go` run a real `TaskReconciler` against
`internal/substrate/substratetest`, an in-memory fake of the same API (atespaces,
actor templates, actors, and tags).
`internal/substrate/client_test.go` uses a client-side `mockControlClient`
instead. No mocking library is in `go.mod`.

### Substrate behaviour the code already depends on

- Actor template images must be digest-pinned (`@sha256:…`); a tag-only
  `spec.profile.image` fails at ActorTemplate creation with reason
  `TemplateCreationFailed` (`docs/manifests.md`). `make images` writes
  `bin/ax-server.digest` / `bin/ax-task-runner.digest` for this reason.
- After `ResumeActor` the actor may be on a different worker, so AX never
  stores a worker IP: the reconciler uses the IP `ResumeActor` returns only for
  its readiness probe (`internal/controller/reconciler.go`), and `ax ssh`
  connects through `atenet-router`.
- Egress: an actor without an egress policy has no outbound access. For a
  task that binds a Gateway, `bindGateway` (`internal/controller/reconciler.go`)
  asks Substrate whether the actor has a policy. If it has none, `bindGateway`
  copies the Gateway's rules to the actor as its one policy (named `default`).
  An actor that has a policy keeps it, so the rules are copied once and never
  updated. `bindGateway` runs after the actor is suspended (`CreateTask`,
  `SuspendTask`) and before it is resumed (`ResumeTask`). Substrate validates
  the rules when they are copied. `bindGateway` fails with reason
  `GatewayBindingFailed` when Substrate refuses them, or when the actor has no
  policy and the Gateway cannot be loaded. `SuspendTask` and `ResumeTask` do
  not fail on the Gateway themselves (`fetchGateway` in
  `internal/server/server.go`), so a suspend always reaches `SuspendActor`. A
  task without a Gateway makes no egress policy call at all.
- `status.actorUid` is read live by `setActorStatus` in
  `internal/server/server.go` and cleared by `storedStatus` before writes.
  It changes when Substrate recreates the actor and is empty when the actor
  is missing or cannot be read. `ax describe task` shows it as `Actor UID`.
- Guest services (process + fs) are mounted by `internal/metadata/server.go`
  only when `task.Spec.Debug` is true; `ax ssh` (`cmd/ax/main.go`) refuses to
  connect otherwise.
- Trust bundle: `BuildActorTemplate` mounts `egress-mitm.ate.dev` and
  `system-roots.ate.dev` at `v1alpha1.EgressTrustBundlePath` for every task,
  and the runner points the CA variables at it (`runner/env.go`). A
  Substrate install that lacks either bundle blocks actor startup.
- Credential injection replaces a header only on requests that carry it. The
  runner supplies a placeholder only for git: `GIT_ASKPASS` is
  `/ax/git-askpass`, a link to the running program, and package `runner`
  answers from `init` when a program that imports it is started under that
  name (`runner/env.go`). It sets no API key variable, because tools choose a
  provider by which key is set: a task sets its own placeholders in
  `spec.env`. A Gateway's spec is never passed into the sandbox.
- Readiness: Substrate and the reconciler probe the runner on port 80
  (`/healthz`, `/readyz?check=workspace`, served by `internal/metadata`);
  `/readyz` is 503 until all workspaces are materialized (`docs/sandbox.md`).
- Workspace snapshot tags: `SuspendActor` defaults to asynchronous snapshot
  upload, whereas `CreateTag` requires the source actor's snapshot to be
  recorded, so `WorkspaceManager` suspends setup actors with
  `SuspendActorAndWaitForSnapshot` (`DURABILITY_MODE_SYNCHRONOUS`,
  `internal/substrate/client.go`). Because `CreateTag` spans two transactions
  around an object copy, an interrupted call can leave a tag with
  `status.snapshot` unset or `SNAPSHOT_STORAGE_STATE_PENDING`; `isTagCompleted`
  (`internal/controller/workspace.go`) requires `SNAPSHOT_STORAGE_STATE_COMPLETED`
  (populated by `internal/substrate/substratetest/server.go` in tests).
- Snapshot storage: Actor templates created by AX leave `SnapshotConfig.StorageLocation`
  empty by default so Substrate uses its cluster-wide `--snapshot-location`. Specifying a custom
  snapshot bucket (`snapshotsBucket` in `BuildActorTemplate`) can only be used when the
  Substrate cluster has the `PerTemplateSnapshotLocation` preview feature gate enabled;
  otherwise Substrate rejects template creation with
  `actor_template.snapshot_config.storage_location: Forbidden`. Since v1.0.0 each path
  segment may contain only `[A-Za-z0-9_-]` and no segment may be `atespaces`; AX passes
  the value through unvalidated, so Substrate is the one that rejects it.

## 5. E2E and the Substrate version pin

`test/e2e` is compiled by `go test ./...` but **self-skips** unless
`AX_E2E=true` or `-args --e2e`. The real runs go through `make`:

| Target | Substrate version | When |
|---|---|---|
| `make test-e2e` | exactly `go list -m -f '{{.Version}}' go.ate.dev/substrate` | PR CI (skipped when only `paths-ignore` files change, e.g. docs) and the nightly schedule (06:00 UTC) in `e2e.yml` |
| `make test-e2e SUBSTRATE_REF=<release tag>` | override | local, or `workflow_dispatch` input `substrate_ref` in `e2e.yml` |
| `make e2e-setup` / `e2e-run` / `e2e-cleanup` | same as above | iterate without re-provisioning |

`hack/setup-e2e-kind.sh` sets up two clusters in one kubeconfig, `ax-e2e` and
a one-worker `ax-e2e-remote` that `TestE2E_Promote` promotes a task to
(`SKIP_REMOTE_KIND=true` leaves it out, and the test is skipped).

E2E never builds Substrate. `hack/setup-e2e-kind.sh` installs the release's
published images from `ghcr.io/agent-substrate/substrate/<image>:<tag>`
(`SUBSTRATE_IMAGE_REPO` to use a mirror) via `ATE_IMAGE_REPO`/`ATE_IMAGE_TAG`,
and pins the WorkerPool's worker image by digest. It exits early unless:

- `SUBSTRATE_REF` is a release tag, `vX.Y.Z` or `vX.Y.Z-rc.N`. Branches,
  commits, and Go pseudo-versions are rejected, and there is no run against
  Substrate `main`.
- The Substrate checkout is clean and at that tag. It still supplies the
  manifests and `ate-setup`, so they must match the images.
- That release's `ate-setup` supports `--image-repo`, and every image the
  install uses exists under the tag.

The install enables the `k8s.io` credential provider. For
`TestE2E_GatewayEgress` the script then:

- Applies `test/e2e/manifests/credential-policy.yaml`, which lets tasks in
  atespace `default` use Secrets labeled `ax.io/credential: "true"` in the AX
  namespace, and restarts the provider, which reads its policy only at startup.
- Runs Substrate's `hack/setup-e2e-egress-tls-kind.sh`, which makes the egress
  gateway trust Substrate's service-DNS CA.
- Deploys the echo server (`test/e2e/manifests/echo.yaml`), whose certificate
  comes from that CA.

Useful env (full list in `test/e2e/README.md`): `SUBSTRATE_DIR` (local
checkout; a detached worktree is created under `.tmp/` if HEAD ≠ ref),
`SUBSTRATE_REPO`, `E2E_ARGS='-test.run TestE2E_ModelCRUD'`.

## 6. Bumping Substrate

What the repo mechanically requires when `go.ate.dev/substrate` or
`github.com/agent-substrate/env` changes in `go.mod`:

```bash
go get go.ate.dev/substrate@vX.Y.Z            # and/or github.com/agent-substrate/env@<sha>
go mod tidy                                    # CI diffs go.mod/go.sum
make generate                                  # buf reads ateapi.proto from the new version
./hack/sync-substrate-manifests.sh             # re-renders internal/install/substrate/manifests/ for `ax local`
go build ./... && go test ./...                # unit tests compile against the new ateapipb
make test-e2e                                  # hack/setup-e2e-kind.sh installs the go.mod version
```

`go.ate.dev/substrate` must stay on a release tag: E2E installs that
release's published images, and a pseudo-version fails E2E setup.

`Gateway.spec.egress` is Substrate's `EgressRule`, so a bump can change what
users may write in a Gateway. Diff the egress messages in
`api/ateapi/v1/ateapi.proto` between the two versions. If the new version adds
a rule kind besides `http`, `https`, and `tlsPassthrough`, `ValidateGateway`
(`internal/apivalidation/gateway.go`) rejects rules of that kind until its
checks cover it. `docs/manifests.md` links to Substrate's egress credential
injection guide at the pinned version.
