Index of every page on this site: llms.txt
All skills
google avatar

/substrate

@de7de90
by googlegoogle/ax15k stars
798
Run it yourself

How AX depends on Agent Substrate (modules, address contract, mocks, version bumps, E2E pin). Load when touching internal/substrate, bumping Substrate, or triaging make test-e2e.

Use this Skill: https://skilld.dev/gh/google/ax/substrate

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈47 tokens for metadata: the name and description. ≈3.5k when used: this file.

Description uses 2.4% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈190 characters, or 2.4%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Agent Substrate in AX

AX schedules every Task as a sandboxed actor on Agent Substrate. AX owns the declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns isolation, checkpoint/restore, worker placement and in-cluster routing. Facts below were verified against main on 2026-11-06 — re-check go.mod and hack/setup-e2e-kind.sh if they look stale.

1. Module map

Module Pinned Provides Imported by
go.ate.dev/substrate v1.0.0 Control API protos api/ateapi/v1 (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) and pkg/postgressetup internal/substrate, internal/substrate/substratetest, internal/install/substrate, internal/server, internal/controller, internal/apivalidation, test/e2e, pkg/apis/v1alpha1 (its .proto files import ateapi.proto)
github.com/agent-substrate/env v0.0.11-0.20260912052224-4468a200b170 (pseudo-version) Guest services served inside the sandbox (process exec + file system gRPC) and proto/ateenv/v1alpha internal/guest, internal/metadata

There is no replace directive for either module. Local hacking against a Substrate checkout goes through SUBSTRATE_DIR in E2E (section 5).

2. The Control API wrapper: internal/substrate.Client

All Control API traffic goes through this package. It encapsulates:

  • Endpoint defaults — api.ate-system.svc.cluster.local:443; authority and TLS ServerName default to api.ate-system.svc when the target contains that string.
  • Auth — bearer token from SUBSTRATE_TOKEN_FILE (in-cluster: projected SA token with audience api.ate-system.svc, see deploy/ax-server.yaml), CA from SUBSTRATE_CA_FILE.
  • Retry — IsTransientError (Unavailable, DeadlineExceeded, Aborted) + callWithRetry (retryOnCodes).
  • Helpers the reconcilers rely on — EnsureAtespace, EnsureActorTemplate, EnsureActorTemplateWithImage, BuildActorTemplate, EnsureActor, EnsureActorFromTag, GetActorEgressPolicy, EnsureActorEgressPolicy, CreateTag, GetTag, ListTags, DeleteTag, SuspendActor, SuspendActorAndWaitForSnapshot, ResumeActor, DeleteActor, WorkerIP, ResourceLimits.

Where a new Control API call lives today

  • internal/substrate/client.go is the only non-test file that calls ateapipb.NewControlClient; every existing RPC is a method on *Client, wrapped in callWithRetry (codes.Unavailable plus optional codes such as codes.Aborted).
  • Each method has a table-driven case in client_test.go (mockControlClient).
  • Callers in internal/controller exercise it through mockControlServer (reconciler_test.go) or internal/substrate/substratetest (workspace_test.go); internal/server and cmd/ax tests also use internal/substrate/substratetest (section 4).

3. Address contract

Value Occurrences
api.ate-system.svc.cluster.local:443 + authority api.ate-system.svc cmd/ax-server/main.go (--substrate-endpoint / --substrate-authority flag defaults), internal/substrate/client.go (3×), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, test/e2e/harness_test.go
atenet-router.ate-system.svc[.cluster.local] internal/controller/reconciler.go (defaultRouterServerName), internal/tunnel/router_tls.go (routerServerName), cmd/ax/main.go (ssh port-forward path), internal/guest/client.go (doc comment), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, docs/networking.md
namespace ate-system all of the above + test/e2e/harness_test.go (ateSystemNamespace), test/e2e/manifests/credential-policy.yaml, internal/install/axdeploy/authz.go, internal/install/cluster/kind.go, internal/install/substrate/bootstrap.go, internal/install/substrate/pki.go, hack/sync-substrate-manifests.sh (and generated internal/install/substrate/manifests/*.yaml)
pod labels ate.dev/worker-pool (worker pods) and app: atenet-egress (egress gateway) the ax-server NetworkPolicy in deploy/ax-server.yaml and test/e2e/manifests/ax-server.yaml, which blocks both from :8080/:9090 and allows only app: atenet-egress (in ate-system) to reach :8081 (ax-server-runner)
opentelemetry-collector.otel-system.svc:4317 (the collector Substrate's kind overlay deploys, manifests/ate-install/kind/ate-otel-config.yaml in Substrate) test/e2e/manifests/ax-server.yaml (ax-otel-config ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors

Why it matters: tasks never get a Service/Ingress. Every request to a task is proxied by the atenet router, which picks the target from the ate-target-actor header and resumes a suspended actor on demand (docs/networking.md). ax ssh port-forwards to svc/atenet-router on 443 (mTLS) and falls back to 80.

4. Testing against Substrate

Unit tests: in-process mock, no cluster

type mockControlServer struct {
    ateapipb.UnimplementedControlServer
    // record calls / inject failures here
}
grpcServer := grpc.NewServer()
ateapipb.RegisterControlServer(grpcServer, mockSrv)
// serve on a listener, point substrate.NewClient at it

Canonical examples: internal/controller/reconciler_test.go (reconciler state transitions; the largest test file in the repo). internal/server/server_test.go and cmd/ax/main_test.go run a real TaskReconciler against internal/substrate/substratetest, an in-memory fake of the same API (atespaces, actor templates, actors, and tags). internal/substrate/client_test.go uses a client-side mockControlClient instead. No mocking library is in go.mod.

Substrate behaviour the code already depends on

  • Actor template images must be digest-pinned (@sha256:…); a tag-only spec.profile.image fails at ActorTemplate creation with reason TemplateCreationFailed (docs/manifests.md). make images writes bin/ax-server.digest / bin/ax-task-runner.digest for this reason.
  • After ResumeActor the actor may be on a different worker, so AX never stores a worker IP: the reconciler uses the IP ResumeActor returns only for its readiness probe (internal/controller/reconciler.go), and ax ssh connects through atenet-router.
  • Egress: an actor without an egress policy has no outbound access. For a task that binds a Gateway, bindGateway (internal/controller/reconciler.go) asks Substrate whether the actor has a policy. If it has none, bindGateway copies the Gateway's rules to the actor as its one policy (named default). An actor that has a policy keeps it, so the rules are copied once and never updated. bindGateway runs after the actor is suspended (CreateTask, SuspendTask) and before it is resumed (ResumeTask). Substrate validates the rules when they are copied. bindGateway fails with reason GatewayBindingFailed when Substrate refuses them, or when the actor has no policy and the Gateway cannot be loaded. SuspendTask and ResumeTask do not fail on the Gateway themselves (fetchGateway in internal/server/server.go), so a suspend always reaches SuspendActor. A task without a Gateway makes no egress policy call at all.
  • status.actorUid is read live by setActorStatus in internal/server/server.go and cleared by storedStatus before writes. It changes when Substrate recreates the actor and is empty when the actor is missing or cannot be read. ax describe task shows it as Actor UID.
  • Guest services (process + fs) are mounted by internal/metadata/server.go only when task.Spec.Debug is true; ax ssh (cmd/ax/main.go) refuses to connect otherwise.
  • Trust bundle: BuildActorTemplate mounts egress-mitm.ate.dev and system-roots.ate.dev at v1alpha1.EgressTrustBundlePath for every task, and the runner points the CA variables at it (runner/env.go). A Substrate install that lacks either bundle blocks actor startup.
  • Credential injection replaces a header only on requests that carry it. The runner supplies a placeholder only for git: GIT_ASKPASS is /ax/git-askpass, a link to the running program, and package runner answers from init when a program that imports it is started under that name (runner/env.go). It sets no API key variable, because tools choose a provider by which key is set: a task sets its own placeholders in spec.env. A Gateway's spec is never passed into the sandbox.
  • Readiness: Substrate and the reconciler probe the runner on port 80 (/healthz, /readyz?check=workspace, served by internal/metadata); /readyz is 503 until all workspaces are materialized (docs/sandbox.md).
  • Workspace snapshot tags: SuspendActor defaults to asynchronous snapshot upload, whereas CreateTag requires the source actor's snapshot to be recorded, so WorkspaceManager suspends setup actors with SuspendActorAndWaitForSnapshot (DURABILITY_MODE_SYNCHRONOUS, internal/substrate/client.go). Because CreateTag spans two transactions around an object copy, an interrupted call can leave a tag with status.snapshot unset or SNAPSHOT_STORAGE_STATE_PENDING; isTagCompleted (internal/controller/workspace.go) requires SNAPSHOT_STORAGE_STATE_COMPLETED (populated by internal/substrate/substratetest/server.go in tests).
  • Snapshot storage: Actor templates created by AX leave SnapshotConfig.StorageLocation empty by default so Substrate uses its cluster-wide --snapshot-location. Specifying a custom snapshot bucket (snapshotsBucket in BuildActorTemplate) can only be used when the Substrate cluster has the PerTemplateSnapshotLocation preview feature gate enabled; otherwise Substrate rejects template creation with actor_template.snapshot_config.storage_location: Forbidden. Since v1.0.0 each path segment may contain only [A-Za-z0-9_-] and no segment may be atespaces; AX passes the value through unvalidated, so Substrate is the one that rejects it.

5. E2E and the Substrate version pin

test/e2e is compiled by go test ./... but self-skips unless AX_E2E=true or -args --e2e. The real runs go through make:

Target Substrate version When
make test-e2e exactly go list -m -f '{{.Version}}' go.ate.dev/substrate PR CI (skipped when only paths-ignore files change, e.g. docs) and the nightly schedule (06:00 UTC) in e2e.yml
make test-e2e SUBSTRATE_REF=<release tag> override local, or workflow_dispatch input substrate_ref in e2e.yml
make e2e-setup / e2e-run / e2e-cleanup same as above iterate without re-provisioning

hack/setup-e2e-kind.sh sets up two clusters in one kubeconfig, ax-e2e and a one-worker ax-e2e-remote that TestE2E_Promote promotes a task to (SKIP_REMOTE_KIND=true leaves it out, and the test is skipped).

E2E never builds Substrate. hack/setup-e2e-kind.sh installs the release's published images from ghcr.io/agent-substrate/substrate/<image>:<tag> (SUBSTRATE_IMAGE_REPO to use a mirror) via ATE_IMAGE_REPO/ATE_IMAGE_TAG, and pins the WorkerPool's worker image by digest. It exits early unless:

  • SUBSTRATE_REF is a release tag, vX.Y.Z or vX.Y.Z-rc.N. Branches, commits, and Go pseudo-versions are rejected, and there is no run against Substrate main.
  • The Substrate checkout is clean and at that tag. It still supplies the manifests and ate-setup, so they must match the images.
  • That release's ate-setup supports --image-repo, and every image the install uses exists under the tag.

The install enables the k8s.io credential provider. For TestE2E_GatewayEgress the script then:

  • Applies test/e2e/manifests/credential-policy.yaml, which lets tasks in atespace default use Secrets labeled ax.io/credential: "true" in the AX namespace, and restarts the provider, which reads its policy only at startup.
  • Runs Substrate's hack/setup-e2e-egress-tls-kind.sh, which makes the egress gateway trust Substrate's service-DNS CA.
  • Deploys the echo server (test/e2e/manifests/echo.yaml), whose certificate comes from that CA.

Useful env (full list in test/e2e/README.md): SUBSTRATE_DIR (local checkout; a detached worktree is created under .tmp/ if HEAD ≠ ref), SUBSTRATE_REPO, E2E_ARGS='-test.run TestE2E_ModelCRUD'.

6. Bumping Substrate

What the repo mechanically requires when go.ate.dev/substrate or github.com/agent-substrate/env changes in go.mod:

go get go.ate.dev/substrate@vX.Y.Z            # and/or github.com/agent-substrate/env@<sha>
go mod tidy                                    # CI diffs go.mod/go.sum
make generate                                  # buf reads ateapi.proto from the new version
./hack/sync-substrate-manifests.sh             # re-renders internal/install/substrate/manifests/ for `ax local`
go build ./... && go test ./...                # unit tests compile against the new ateapipb
make test-e2e                                  # hack/setup-e2e-kind.sh installs the go.mod version

go.ate.dev/substrate must stay on a release tag: E2E installs that release's published images, and a pseudo-version fails E2E setup.

Gateway.spec.egress is Substrate's EgressRule, so a bump can change what users may write in a Gateway. Diff the egress messages in api/ateapi/v1/ateapi.proto between the two versions. If the new version adds a rule kind besides http, https, and tlsPassthrough, ValidateGateway (internal/apivalidation/gateway.go) rejects rules of that kind until its checks cover it. docs/manifests.md links to Substrate's egress credential injection guide at the pinned version.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts5d3 checks · Risk SAFE
  • Gen Agent Trust Hub5d

    This skill provides technical documentation for the AX project's integration with Agent Substrate. It covers architecture, dependency management, and testing workflows, serving as a developer guide without introducing security vulnerabilities.

  • Socket5d

    No alerts

  • Snyk5d

    Risk: LOW · No issues

Signed by skilld at de7de90. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 3 days ago

README badge

README badge for google/ax/substrate