Agent Substrate in AX
AX schedules every Task as a sandboxed actor on
Agent Substrate. AX owns the
declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns
isolation, checkpoint/restore, worker placement and in-cluster routing.
Facts below were verified against main on 2026-11-06 — re-check go.mod
and hack/setup-e2e-kind.sh if they look stale.
1. Module map
| Module | Pinned | Provides | Imported by |
|---|---|---|---|
go.ate.dev/substrate |
v1.0.0 |
Control API protos api/ateapi/v1 (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) and pkg/postgressetup |
internal/substrate, internal/substrate/substratetest, internal/install/substrate, internal/server, internal/controller, internal/apivalidation, test/e2e, pkg/apis/v1alpha1 (its .proto files import ateapi.proto) |
github.com/agent-substrate/env |
v0.0.11-0.20260912052224-4468a200b170 (pseudo-version) |
Guest services served inside the sandbox (process exec + file system gRPC) and proto/ateenv/v1alpha |
internal/guest, internal/metadata |
There is no replace directive for either module. Local hacking against a
Substrate checkout goes through SUBSTRATE_DIR in E2E (section 5).
2. The Control API wrapper: internal/substrate.Client
All Control API traffic goes through this package. It encapsulates:
- Endpoint defaults —
api.ate-system.svc.cluster.local:443; authority and TLSServerNamedefault toapi.ate-system.svcwhen the target contains that string. - Auth — bearer token from
SUBSTRATE_TOKEN_FILE(in-cluster: projected SA token with audienceapi.ate-system.svc, seedeploy/ax-server.yaml), CA fromSUBSTRATE_CA_FILE. - Retry —
IsTransientError(Unavailable,DeadlineExceeded,Aborted) +callWithRetry(retryOnCodes). - Helpers the reconcilers rely on —
EnsureAtespace,EnsureActorTemplate,EnsureActorTemplateWithImage,BuildActorTemplate,EnsureActor,EnsureActorFromTag,GetActorEgressPolicy,EnsureActorEgressPolicy,CreateTag,GetTag,ListTags,DeleteTag,SuspendActor,SuspendActorAndWaitForSnapshot,ResumeActor,DeleteActor,WorkerIP,ResourceLimits.
Where a new Control API call lives today
internal/substrate/client.gois the only non-test file that callsateapipb.NewControlClient; every existing RPC is a method on*Client, wrapped incallWithRetry(codes.Unavailableplus optional codes such ascodes.Aborted).- Each method has a table-driven case in
client_test.go(mockControlClient). - Callers in
internal/controllerexercise it throughmockControlServer(reconciler_test.go) orinternal/substrate/substratetest(workspace_test.go);internal/serverandcmd/axtests also useinternal/substrate/substratetest(section 4).
3. Address contract
| Value | Occurrences |
|---|---|
api.ate-system.svc.cluster.local:443 + authority api.ate-system.svc |
cmd/ax-server/main.go (--substrate-endpoint / --substrate-authority flag defaults), internal/substrate/client.go (3×), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, test/e2e/harness_test.go |
atenet-router.ate-system.svc[.cluster.local] |
internal/controller/reconciler.go (defaultRouterServerName), internal/tunnel/router_tls.go (routerServerName), cmd/ax/main.go (ssh port-forward path), internal/guest/client.go (doc comment), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, docs/networking.md |
namespace ate-system |
all of the above + test/e2e/harness_test.go (ateSystemNamespace), test/e2e/manifests/credential-policy.yaml, internal/install/axdeploy/authz.go, internal/install/cluster/kind.go, internal/install/substrate/bootstrap.go, internal/install/substrate/pki.go, hack/sync-substrate-manifests.sh (and generated internal/install/substrate/manifests/*.yaml) |
pod labels ate.dev/worker-pool (worker pods) and app: atenet-egress (egress gateway) |
the ax-server NetworkPolicy in deploy/ax-server.yaml and test/e2e/manifests/ax-server.yaml, which blocks both from :8080/:9090 and allows only app: atenet-egress (in ate-system) to reach :8081 (ax-server-runner) |
opentelemetry-collector.otel-system.svc:4317 (the collector Substrate's kind overlay deploys, manifests/ate-install/kind/ate-otel-config.yaml in Substrate) |
test/e2e/manifests/ax-server.yaml (ax-otel-config ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |
Why it matters: tasks never get a Service/Ingress. Every request to a task is
proxied by the atenet router, which picks the target from the
ate-target-actor header and resumes a suspended actor on demand
(docs/networking.md). ax ssh port-forwards to svc/atenet-router on 443
(mTLS) and falls back to 80.
4. Testing against Substrate
Unit tests: in-process mock, no cluster
type mockControlServer struct {
ateapipb.UnimplementedControlServer
// record calls / inject failures here
}
grpcServer := grpc.NewServer()
ateapipb.RegisterControlServer(grpcServer, mockSrv)
// serve on a listener, point substrate.NewClient at itCanonical examples: internal/controller/reconciler_test.go (reconciler state
transitions; the largest test file in the repo). internal/server/server_test.go
and cmd/ax/main_test.go run a real TaskReconciler against
internal/substrate/substratetest, an in-memory fake of the same API (atespaces,
actor templates, actors, and tags).
internal/substrate/client_test.go uses a client-side mockControlClient
instead. No mocking library is in go.mod.
Substrate behaviour the code already depends on
- Actor template images must be digest-pinned (
@sha256:…); a tag-onlyspec.profile.imagefails at ActorTemplate creation with reasonTemplateCreationFailed(docs/manifests.md).make imageswritesbin/ax-server.digest/bin/ax-task-runner.digestfor this reason. - After
ResumeActorthe actor may be on a different worker, so AX never stores a worker IP: the reconciler uses the IPResumeActorreturns only for its readiness probe (internal/controller/reconciler.go), andax sshconnects throughatenet-router. - Egress: an actor without an egress policy has no outbound access. For a
task that binds a Gateway,
bindGateway(internal/controller/reconciler.go) asks Substrate whether the actor has a policy. If it has none,bindGatewaycopies the Gateway's rules to the actor as its one policy (nameddefault). An actor that has a policy keeps it, so the rules are copied once and never updated.bindGatewayruns after the actor is suspended (CreateTask,SuspendTask) and before it is resumed (ResumeTask). Substrate validates the rules when they are copied.bindGatewayfails with reasonGatewayBindingFailedwhen Substrate refuses them, or when the actor has no policy and the Gateway cannot be loaded.SuspendTaskandResumeTaskdo not fail on the Gateway themselves (fetchGatewayininternal/server/server.go), so a suspend always reachesSuspendActor. A task without a Gateway makes no egress policy call at all. status.actorUidis read live bysetActorStatusininternal/server/server.goand cleared bystoredStatusbefore writes. It changes when Substrate recreates the actor and is empty when the actor is missing or cannot be read.ax describe taskshows it asActor UID.- Guest services (process + fs) are mounted by
internal/metadata/server.goonly whentask.Spec.Debugis true;ax ssh(cmd/ax/main.go) refuses to connect otherwise. - Trust bundle:
BuildActorTemplatemountsegress-mitm.ate.devandsystem-roots.ate.devatv1alpha1.EgressTrustBundlePathfor every task, and the runner points the CA variables at it (runner/env.go). A Substrate install that lacks either bundle blocks actor startup. - Credential injection replaces a header only on requests that carry it. The
runner supplies a placeholder only for git:
GIT_ASKPASSis/ax/git-askpass, a link to the running program, and packagerunneranswers frominitwhen a program that imports it is started under that name (runner/env.go). It sets no API key variable, because tools choose a provider by which key is set: a task sets its own placeholders inspec.env. A Gateway's spec is never passed into the sandbox. - Readiness: Substrate and the reconciler probe the runner on port 80
(
/healthz,/readyz?check=workspace, served byinternal/metadata);/readyzis 503 until all workspaces are materialized (docs/sandbox.md). - Workspace snapshot tags:
SuspendActordefaults to asynchronous snapshot upload, whereasCreateTagrequires the source actor's snapshot to be recorded, soWorkspaceManagersuspends setup actors withSuspendActorAndWaitForSnapshot(DURABILITY_MODE_SYNCHRONOUS,internal/substrate/client.go). BecauseCreateTagspans two transactions around an object copy, an interrupted call can leave a tag withstatus.snapshotunset orSNAPSHOT_STORAGE_STATE_PENDING;isTagCompleted(internal/controller/workspace.go) requiresSNAPSHOT_STORAGE_STATE_COMPLETED(populated byinternal/substrate/substratetest/server.goin tests). - Snapshot storage: Actor templates created by AX leave
SnapshotConfig.StorageLocationempty by default so Substrate uses its cluster-wide--snapshot-location. Specifying a custom snapshot bucket (snapshotsBucketinBuildActorTemplate) can only be used when the Substrate cluster has thePerTemplateSnapshotLocationpreview feature gate enabled; otherwise Substrate rejects template creation withactor_template.snapshot_config.storage_location: Forbidden. Since v1.0.0 each path segment may contain only[A-Za-z0-9_-]and no segment may beatespaces; AX passes the value through unvalidated, so Substrate is the one that rejects it.
5. E2E and the Substrate version pin
test/e2e is compiled by go test ./... but self-skips unless
AX_E2E=true or -args --e2e. The real runs go through make:
| Target | Substrate version | When |
|---|---|---|
make test-e2e |
exactly go list -m -f '{{.Version}}' go.ate.dev/substrate |
PR CI (skipped when only paths-ignore files change, e.g. docs) and the nightly schedule (06:00 UTC) in e2e.yml |
make test-e2e SUBSTRATE_REF=<release tag> |
override | local, or workflow_dispatch input substrate_ref in e2e.yml |
make e2e-setup / e2e-run / e2e-cleanup |
same as above | iterate without re-provisioning |
hack/setup-e2e-kind.sh sets up two clusters in one kubeconfig, ax-e2e and
a one-worker ax-e2e-remote that TestE2E_Promote promotes a task to
(SKIP_REMOTE_KIND=true leaves it out, and the test is skipped).
E2E never builds Substrate. hack/setup-e2e-kind.sh installs the release's
published images from ghcr.io/agent-substrate/substrate/<image>:<tag>
(SUBSTRATE_IMAGE_REPO to use a mirror) via ATE_IMAGE_REPO/ATE_IMAGE_TAG,
and pins the WorkerPool's worker image by digest. It exits early unless:
SUBSTRATE_REFis a release tag,vX.Y.ZorvX.Y.Z-rc.N. Branches, commits, and Go pseudo-versions are rejected, and there is no run against Substratemain.- The Substrate checkout is clean and at that tag. It still supplies the
manifests and
ate-setup, so they must match the images. - That release's
ate-setupsupports--image-repo, and every image the install uses exists under the tag.
The install enables the k8s.io credential provider. For
TestE2E_GatewayEgress the script then:
- Applies
test/e2e/manifests/credential-policy.yaml, which lets tasks in atespacedefaultuse Secrets labeledax.io/credential: "true"in the AX namespace, and restarts the provider, which reads its policy only at startup. - Runs Substrate's
hack/setup-e2e-egress-tls-kind.sh, which makes the egress gateway trust Substrate's service-DNS CA. - Deploys the echo server (
test/e2e/manifests/echo.yaml), whose certificate comes from that CA.
Useful env (full list in test/e2e/README.md): SUBSTRATE_DIR (local
checkout; a detached worktree is created under .tmp/ if HEAD ≠ ref),
SUBSTRATE_REPO, E2E_ARGS='-test.run TestE2E_ModelCRUD'.
6. Bumping Substrate
What the repo mechanically requires when go.ate.dev/substrate or
github.com/agent-substrate/env changes in go.mod:
go get go.ate.dev/substrate@vX.Y.Z # and/or github.com/agent-substrate/env@<sha>
go mod tidy # CI diffs go.mod/go.sum
make generate # buf reads ateapi.proto from the new version
./hack/sync-substrate-manifests.sh # re-renders internal/install/substrate/manifests/ for `ax local`
go build ./... && go test ./... # unit tests compile against the new ateapipb
make test-e2e # hack/setup-e2e-kind.sh installs the go.mod versiongo.ate.dev/substrate must stay on a release tag: E2E installs that
release's published images, and a pseudo-version fails E2E setup.
Gateway.spec.egress is Substrate's EgressRule, so a bump can change what
users may write in a Gateway. Diff the egress messages in
api/ateapi/v1/ateapi.proto between the two versions. If the new version adds
a rule kind besides http, https, and tlsPassthrough, ValidateGateway
(internal/apivalidation/gateway.go) rejects rules of that kind until its
checks cover it. docs/manifests.md links to Substrate's egress credential
injection guide at the pinned version.