Index of every page on this site: llms.txt
All skills
pydantic avatar

/pydantic-ai-harness

@b2d64a3
by pydanticpydantic/pydantic-ai21k stars
3,221
Run it yourself

Extend Pydantic AI agents with capabilities from pydantic-ai-harness -- the Coder coding agent, file and shell tools in local or sandboxed workspaces (Modal, E2B, Render, Sprites, Daytona, boxd, smolvm, BoxLite), Code Mode, sub-agents and planning, model routing, memory and skills, context compaction, guardrails and spend limits, web research and browsers, hosted SaaS integrations, step persistence, and background agents on Render Workflows. Use when the user mentions pydantic-ai-harness or pydantic_ai_harness, imports a harness capability such as Coder, CodeMode, FileSystem, Shell, SubAgents, ModelRouter, Memory, or ToolGuardrail, or wants a Pydantic AI agent that edits files, runs commands or agent-written Python, delegates, routes work between models, remembers, manages long context, stays within limits, or runs model and tool operations as chained Render task runs.

  • 11 files
  • 334 KB
  • MIT
  • Updated 2 days ago
  • GitHub
Use this Skill: https://skilld.dev/gh/pydantic/pydantic-ai/pydantic-ai-harness

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesCODING-AND-WORKSPACES.md

≈12k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Coding Agents and Workspaces

Harness capabilities that touch files or run commands (Coder, FileSystem, Shell, RepoContext, Macroscope) act in the run's workspace, never on a path they pick themselves. Attach the workspace as its own capability: LocalWorkspace for this machine, or ModalSandbox / E2BSandbox / RenderSandbox / SpritesSandbox / DaytonaSandbox / BoxdSandbox for an isolated cloud machine, KubernetesSandbox for a pod on your own cluster, or SmolSandbox / BoxLiteSandbox for a microVM on this machine, or DockerContainer for a container on this machine (it shares your kernel, so it is not a security boundary), or OpenSandbox for a sandbox on your own OpenSandbox server. Core workspace semantics (refs, workspace= precedence, ReadOnlyWorkspace, continuing from message history) are in the building-pydantic-ai-agents skill's WORKSPACES.md reference; this file covers the harness side.

Which one do I want?

Need Use
General coding agent: investigate, edit, test, delegate Coder() + a workspace
Narrower agent: file tools and/or commands with policy (allowlists, patterns, read-only) FileSystem() and/or Shell() + a workspace
Files and commands start in a specific directory LocalWorkspace('./repo'), or working_dir= on a sandbox
Untrusted model or repo, or no access to this machine Swap LocalWorkspace for ModalSandbox() / E2BSandbox() / RenderSandbox() / SpritesSandbox() / DaytonaSandbox() / BoxdSandbox(); tool capabilities stay the same. For a repo the operator didn't write, add authority='none' (every sandbox and SSHWorkspace takes authority and path_authority, like LocalWorkspace)
Isolation without a cloud account SmolSandbox(): a smolvm microVM on this machine (macOS Apple silicon, Linux x86_64/aarch64 with KVM)
Local microVM with an egress allow-list, or host mounts/ports/secrets BoxLiteSandbox(allow_internet_access=True, allowed_hosts=[...]): a BoxLite micro-VM; box_options passes other boxlite.BoxOptions
A disposable environment on this machine with Docker or Podman, not a security boundary DockerContainer(image='python:3.13-slim'): a local container on your kernel
Sandboxes on your own server, on Docker or Kubernetes OpenSandbox(image='python:3.13-slim'): an OpenSandbox sandbox
Work on another machine you can ssh to SSHWorkspace('user@host', working_dir=...); wrap it in BubblewrappedWorkspace(...) to confine its commands to the working directory on that host
Only your own tools use ctx.workspace A workspace capability alone

Workspace capabilities (LocalWorkspace and the sandboxes) register no tools. allowed_commands, root_dir, and patterns are guardrails against accidents, not isolation.

Attach a workspace, or the run fails

from pydantic_ai import Agent
from pydantic_ai.exceptions import UserError

from pydantic_ai_harness import Shell

agent = Agent('test', capabilities=[Shell()])  # no workspace attached
try:
    agent.run_sync('List the files.')
except UserError as e:
    print(str(e).split('.')[0])
    #> `Shell` needs a workspace, but none is attached to this run
  • Add LocalWorkspace('.') (from pydantic_ai.capabilities; - LocalWorkspace: . in an agent spec) or a sandbox capability, or pass workspace= to run().
  • Set the directory on the workspace: LocalWorkspace('./repo'). Coder('dir'), Shell(cwd=), FileSystem(cwd=), Macroscope(cwd=), RepoContext(workspace_dir=) warn and are ignored.
  • LocalWorkspace needs a POSIX host and is not a sandbox. Commands inherit only PATH, HOME, LANG, LC_ALL, LC_CTYPE; pass the rest with LocalWorkspace('.', env={'GH_TOKEN': ...}). Never pass os.environ (leaks LLM API keys to the model's commands). Under uv run, python/pytest resolve from the agent's venv via PATH.
  • For a token the sandbox's commands can use but not read (such as GH_TOKEN for git push/gh), use the provider's proxy option, which only shapes a new sandbox. Each provider's docs page has a section on it:
    • E2B: E2BSandbox(network={'rules': {host: [{'transform': {'headers': {...}}}]}}); HTTPS only.
    • boxd: BoxdSandbox(secrets=['GH_TOKEN']), naming a secret made beforehand with secrets.set(name, value, scope='private-unattached', domains=[...]); without domains the real value is set. boxd also substitutes request bodies, so a command can store the value on a named host (a GitHub issue or gist) and read it back: use a fine-grained token without Gists access.
    • Daytona: DaytonaSandbox(secrets={'GH_TOKEN': 'secret-name'}), mounting an organization Secret created with hosts (without them it goes to any host); HTTPS headers only, Basic auth included.
    • OpenSandbox: OpenSandboxBackend(credential_proxy=CredentialProxyConfig(enabled=True), network_policy=...) (backend only), then fill (await backend.get_sandbox()).credential_vault before passing the backend as workspace=. Needs the server's egress sidecar in dns+nft mode and a policy allowing every bound host.
  • On Windows, use WindowsWorkspace('.') (same options, also not a sandbox); Coder() without a workspace fails as on POSIX. Commands run in PowerShell and paths are Windows paths. Shell offers no run_command there and refuses max_file_bytes; grep and list_files read the files when rg is missing.
  • Harness files (Shell job logs, sticky-cd state, tool-output spills) go in .pydantic-ai-harness/ under the working directory, with a .gitignore of *. Spills are never pruned. FileSystem keeps that directory read-only.

Coder

Combined capability: default instructions + FileSystem + Shell + RepoContext + SubAgents + ClearToolResults(max_fraction=0.7), a ToolOutputLimits that truncates any tool result to 64,000 chars (no read_tool_result), RepairToolArguments, and prompt caching (unified cache=True).

uv add "pydantic-ai-harness[coder]"   # installs json-repair, and ripgrep (rg) on the agent host
from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace

from pydantic_ai_harness.coder import Coder

agent = Agent(
    'test',  # e.g. 'anthropic:claude-opus-5-5'
    capabilities=[
        LocalWorkspace('.'),
        Coder(instructions='Run `make test` before you finish.'),
    ],
)

Tools (no content hashes, 50,000 chars per read; shell is unrestricted, waits at most 270 s, returns the last 16,000 bytes of output after [... output truncated, N earlier bytes omitted] plus the full log's path (.pydantic-ai-harness/shell/<id>/output.log), and its commands outlive the run; delegate_task has one delegate, self):

  • read_file(path, offset=0, limit=None), write_file(path, content), edit_file(path, old_text=None, new_text=None, replacements=None)
  • list_files(path='.', glob=None), grep(pattern, path='.', glob=None, file_type=None, ignore_case=False, literal=False, context=0)
  • shell(command, mode='foreground', timeout=None) (mode='background' too), delegate_task(agent_name, task)

Keyword-only options: instructions=None (appended to the default guidance), autonomy='high' ('low' proposes a plan and waits for approval; 'medium' asks when a choice really matters; 'high' asks only for credentials, approvals, or real ambiguity and measures stated numeric targets; 'max' never asks, also deletes its scratch files, and treats an unmet stated criterion as a misreading. This is guidance, not a tool limit; Permissions asks before writes or commands. A delegate finishes on its own and measures numeric targets whatever the level; the type is Autonomy), unrestricted_filesystem=False (True sets FileSystem(root_dir='/', read_only_patterns=[])), repo_context=True (RepoContext walking up to the repository root and surfacing nested instruction files as the agent reads, inlining up to 32,000 characters of them per conversation before switching to pointers; pass a RepoContext to use it in place of the bundled one, whose settings it replaces entirely), sub_agents=True, agent_folders=None, task_control=None (a TaskControl is passed to SubAgents(task_control=...); it requires sub_agents=True), caching=True, strict=False.

  • Bind Coder on the Agent: passing it to run(capabilities=...) raises UserError unless sub_agents=False. Delegates re-run the bound agent (same model, workspace, and neighbouring capabilities such as approval gates), up to 3 levels deep.
  • Lazy sandbox creation: RepoContext reads the workspace at run start, which creates a sandbox before any tool call. Coder(repo_context=False) defers it (the instructions then stop naming the working directory). To use your own RepoContext, pass it as repo_context=RepoContext(...) rather than binding it next to Coder, which would load the files twice; that also keeps the project note.
  • File tools stay in the working directory with .git, .env*, keys, and secrets* read-only; shell bypasses all of that. To change a bundled setting (hashes, allowlist, list_directory), compose FileSystem/Shell/SubAgents yourself; RepoContext is configured through repo_context=RepoContext(...).
  • Prompt caching is on by default (cache=True): a coding run resends the same tools and instructions plus a growing conversation on every request, so each request reads back the previous one's prefix. What it does per provider and what it costs: https://pydantic.dev/docs/ai/capabilities/caching/#unified-caching-settings. A cache set on the model, in Agent(model_settings=...), or by a Caching listed before Coder wins, as do a later capability and run-level model_settings: model_settings={'cache': False} turns it off, {'cache': '1h'} keeps it longer. Coder(caching=False) leaves the default out.
  • Strict tool schemas are off by default (strict=False on Coder's own tools): unset, OpenAI constrains arguments to strict-compatible schemas and Gemini 2.5+ uses VALIDATED mode; Coder relies on RepairToolArguments and validation retries instead. Coder(strict=True) forces strict mode, strict=None leaves it to the provider. Tools bound next to Coder keep their own strict, but on Gemini a strict=False tool keeps the whole request on AUTO. See https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/#strict-mode.
  • Your own ClearToolResults / TieredCompaction / WarnNearLimits next to Coder run as well (no id clash), each at its own trigger. List your ToolOutputLimits after Coder so it sees raw returns; listed before, it sees Coder's 64,000-char result. For shell, bands must sit under ~16,000 chars.

pydantic_ai_harness.coder:coder_agent is a model-less Agent(name='coder') with LocalWorkspace('.') (cwd at import) and Coder():

uvx --with "pydantic-ai-harness[coder]" clai -a pydantic_ai_harness.coder:coder_agent -m anthropic:claude-opus-5-5

BenchmarkInstructions

For benchmark runs, bind BenchmarkInstructions() next to Coder(): it adds the benchmark-only guidance (self-contained deliverables checked from a clean directory, using every field of the provided data) that Coder's defaults leave out.

Sharing a workspace across runs and agents

# Same conversation, same files: the history carries the workspace ref.
result = coder.run_sync('Document it.', message_history=result.all_messages())
# Another agent (no workspace capability of its own), same files.
# ReadOnlyWorkspace (pydantic_ai.workspaces) leaves out shell and edit tools.
review = reviewer.run_sync('Review it.', workspace=ReadOnlyWorkspace(result.workspace))

History cannot move a LocalWorkspace to another directory. With a sandbox, agent.to_web() starts a new sandbox per message (no ref in the web protocol); the CLI keeps one per session.

FileSystem

Path-scoped file tools. tools= defaults to read_file, write_file, edit_file, list_directory, search_files, find_files, create_directory, file_info. Opt-in ripgrep tools list_files and grep must be named in tools (a sandbox image needs its own rg, else a POSIX git/grep/find fallback runs).

from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace

from pydantic_ai_harness import FileSystem

agent = Agent(
    'test',
    capabilities=[
        LocalWorkspace('.'),
        FileSystem(
            tools=['read_file', 'edit_file', 'list_files', 'grep'],
            allowed_patterns=['*.py', '*.toml'],
            denied_patterns=['**/node_modules/*'],
        ),
    ],
)

Tool arguments (scripted tests need them; expected_hash exists only with content_hashes=True, the default): read_file(path, offset=0, limit=None), write_file(path, content, expected_hash=None), edit_file(path, old_text=None, new_text=None, replacements=None, expected_hash=None), list_directory(path='.'), search_files(pattern, path='.', include_glob=None), find_files(pattern, path='.'), create_directory(path), file_info(path); list_files and grep as under Coder.

read_file returns [header, BinaryContent] for a PNG, JPEG, GIF, WebP or PDF up to 5 MB (detected from its leading bytes, not its extension), and a text description (size, media type, and for other binary files the first 64 bytes in hex) for any other binary file; text files come back as a string. A file over 64 MiB is described by its size instead of read: find the lines with search_files or grep.

read_file has the tool kind READ_FILE_KIND (file_system.read_file). ToolOutputLimits' read_tool_result and grep_tool_result, Memory's read_memory and RepoContext's inventory tool step aside for it when the files they read are in the run's workspace, also when FileSystem is renamed (FileSystem().prefix_tools('wiki')).

Parameters: root_dir (default the working directory; must be the working directory or an ancestor of it, so FileSystem(root_dir='data') under LocalWorkspace('.') raises UserError: attach LocalWorkspace('./data') instead; '/' disables containment), allowed_patterns / denied_patterns / read_only_patterns (fnmatch: * spans /; a directory pattern covers descendants; denied and read-only patterns ignore case), read_only=False (True registers only read tools), content_hashes=True (expected_hash concurrency check), max_read_lines=2000 (also caps a model's limit), max_read_chars=50_000, max_*_results=1000.

  • Passing read_only_patterns replaces the defaults (**/.git, **/.env, **/.env.*, *.pem, *.key, **/secrets*, **/.pydantic-ai-harness/**).
  • Walkers skip dotfiles and dot-directories unless a path or pattern names one explicitly, and filter entries, not the root, by allowed_patterns. A read-only workspace gets only read tools.
  • Veto writes with @agent.on_event(FileChangeRequestedEvent) (not FileChangeRequestEvent) + event.cancel(reason) (from pydantic_ai_harness.file_system); the event carries a unified diff.
  • Recognize the file tools by kind, never by name: each declares tool_kind file_system.<tool> (READ_FILE_KIND, WRITE_FILE_KIND, ... from pydantic_ai_harness.file_system), and its calls become typed parts (isinstance(part, ReadFileCallPart); part.typed_args is None until the arguments validate). Only read_file has a typed return part (ReadFileReturnPart, successful returns only).

SmartSearch

Plain-English code search (.smart_search, no extra). One tool, smart_search(query, directory='.', glob=None, limit=5, candidates=128), returns a SmartSearchResult (matches, omitted_matches, coverage, warnings). Files are listed with rg --files in the workspace (needs rg on the workspace PATH; hidden on a workspace that cannot run commands), cut into line windows (Python: along its ast), shortlisted with BM25, and each shortlisted snippet is judged by a model.

from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace

from pydantic_ai_harness import SmartSearch

agent = Agent('test', capabilities=[LocalWorkspace('.'), SmartSearch(model='test')])
  • model is required: any Pydantic AI model or name. A decision model is recommended: 'system-one:<name>' (or a SystemOneModel) for any model behind the System One API (needs SYSTEM_ONE_BASE_URL), or 'typesafe:jev-latest'. A named judge gets a 30 s request timeout; an instance keeps its own settings.
  • threshold=0.5 (tuned for Jev; retune for another judge), concurrency=8, guidance=None (default discovery instructions; '' for none).
  • cache_index=False: on, keeps up to four directories' indexes (per glob) in memory for the agent's life; later searches re-read files but re-chunk only changed ones (content hash), and parallel searches of one directory build it once. A search is capped at 200,000 files (after glob), 50M lines and 2 GiB (ModelRetry).
  • Each judged snippet is one request in the run's usage (tokens too), checked against its UsageLimits. A search judges at most half of what request_limit (default 50) has left after one kept for the model's next turn (24 on a first search), best-ranked first, and warns; with none left it fails. Raise request_limit for full searches. A failed judgment fails the search as a tool failure the model sees.
  • Sends the query, snippets and paths to the judge's provider; directory must be inside the working directory (symlinks followed), else ModelRetry.
  • Durable execution records each search's judgments under the default id (smart_search); keep it stable once deployed. Two sharing an id become one when they agree and raise UserError at build when they differ; to keep two, give each its own id and wrap them in PrefixTools.

Shell

Default tools run_command, start_command, check_command, stop_command; tools=['shell'] registers only the persistent shell tool instead.

from pydantic_ai_harness import LLM_API_KEY_ENV_PATTERNS, Shell

Shell(allowed_commands=['ls', 'cat', 'rg'])  # allowlist; default denylist is dropped
Shell(denied_commands=[])  # no command-name filtering
Shell(
    env={'PYTHONUNBUFFERED': '1', 'ANTHROPIC_API_KEY': 'sk-...'},
    denied_env_patterns=LLM_API_KEY_ENV_PATTERNS,  # filters Shell(env=) only
)
Shell(tools=['shell'], default_timeout=270)  # persistent commands

Tool arguments: run_command(command, timeout_seconds=None), start_command(command), check_command(command_id), stop_command(command_id); shell(command, mode='foreground', timeout=None).

Parameters: denied_commands (default rm, rmdir, mkfs, dd, format, shutdown, reboot, halt, poweroff, init), denied_operators, default_timeout=30.0, max_output_chars=50_000 (keeps the tail; shell already returns at most the last 16,000 bytes), max_file_bytes=None (per-file ulimit -f), max_log_bytes=100_000_000 (see below), persist_cwd=False (sticky cd), allow_interactive=False, env, denied_env_patterns, secret_env (added like env; exact occurrences masked as [REDACTED:<name>] in tool results, so not a value a command encodes or slices, nor logs the model reads with file tools; short values are masked everywhere).

  • max_log_bytes bounds start_command and shell logs, including with tools=['shell']: a log past it is emptied and gets a [Log exceeded max_log_bytes=...] notice while the command keeps running. None turns it off.
  • ValueError for: non-positive max_file_bytes or max_log_bytes; both allowed_commands and non-empty denied_commands; max_file_bytes with persist_cwd=True or tools=['shell']; tools=['shell'] with default_timeout outside (0, 270].
  • Only the first token is checked: python, git, make, uv can spawn anything.
  • denied_env_patterns does not filter LocalWorkspace(env=).
  • For output pushed to the model when a command ends, add BackgroundTools(tools=['run_command']). Commands get stdin at EOF; there is no tool to send input.
  • Background commands outlive the run; a later run on the same workspace can check or stop them. Clean up with stop_command or your own code. They need mv and base64 in the workspace, and setsid for process-group stops.
  • persist_cwd does not apply to shell. A read-only workspace gets no Shell tools.

Sandbox workspaces

ModalSandbox E2BSandbox RenderSandbox SpritesSandbox DaytonaSandbox BoxdSandbox
Extra [modal] [e2b] [render] [sprites] [daytona] [boxd]
Credentials modal token new, or MODAL_TOKEN_ID + MODAL_TOKEN_SECRET E2B_API_KEY RENDER_API_KEY + RENDER_WORKSPACE_ID SPRITE_TOKEN DAYTONA_API_KEY BOXD_API_KEY
Lifetime sandbox_timeout=86_400 (10 to 86,400), optional idle_timeout; then terminated sandbox_timeout=3_600 (Pro: up to 86,400); then paused, resumed on attach sandbox_timeout=3_600 (up to 86,400); then terminated Unlimited: sleeps when idle, persists until deleted Unlimited: stopped after auto_stop_interval idle minutes (default 15), started again on attach; archived after 7 days stopped, never deleted Unlimited: suspended, then hibernated, when idle (organization defaults; auto_suspend_timeout seconds), woken by the next command; a stopped machine is started on attach; no delete timer unless the organization sets one
Default cwd / user /root, root /home/user, user home directory, root /home/sprite, sprite /home/daytona, daytona /home/boxd, boxd
rg preinstalled Yes No Not documented No Yes Yes
New-sandbox options image, app_name, create_app_if_missing, client template, allow_internet_access, network, client allow_internet_access, client runtime, client snapshot, auto_stop_interval, allow_internet_access, secrets, client org, snapshot, vcpu, auto_suspend_timeout, secrets, client

All take working_dir (absolute; created on a new sandbox, must exist on an attached one) and env (nothing from your machine is passed through). All are asyncio-only and reject defer_loading=True.

One sandbox per conversation in a web service: store the ref and the history, reattach each turn, destroy at the end.

from dataclasses import dataclass
from typing import Any

from pydantic import TypeAdapter
from pydantic_ai import Agent, RunContext
from pydantic_ai.capabilities import Hooks
from pydantic_ai.messages import ModelMessagesTypeAdapter
from pydantic_ai.run import AgentRunResult
from pydantic_ai.workspaces import WorkspaceRef, WorkspaceUnavailableError

from pydantic_ai_harness.coder import Coder
from pydantic_ai_harness.modal_sandbox import ModalSandbox

REF = TypeAdapter(WorkspaceRef)  # {"provider": ..., "id": ...}; no credentials


@dataclass
class Conversation:  # your DB row
    sandbox_ref: bytes | None = None
    history: bytes | None = None  # result.all_messages_json()


hooks = Hooks()  # a failed run returns no result, so record its ref here


@hooks.on.run_error
async def keep_ref(
    ctx: RunContext[Conversation], *, error: BaseException
) -> AgentRunResult[Any]:
    if ctx.workspace.ref is not None:
        ctx.deps.sandbox_ref = REF.dump_json(ctx.workspace.ref)
    raise error


sandbox = ModalSandbox(working_dir='/root/project', idle_timeout=600)
agent = Agent(
    'anthropic:claude-opus-5-5',
    deps_type=Conversation,
    capabilities=[sandbox, Coder(), hooks],
)


async def turn(conv: Conversation, prompt: str) -> str:
    ref = REF.validate_json(conv.sandbox_ref) if conv.sandbox_ref else None
    history = ModelMessagesTypeAdapter.validate_json(conv.history) if conv.history else None
    try:
        result = await agent.run(prompt, deps=conv, message_history=history, workspace=ref)
    except WorkspaceUnavailableError:  # expired or destroyed: start a new sandbox
        result = await agent.run(prompt, deps=conv, message_history=history, workspace='new')
    if result.workspace.ref is not None:
        conv.sandbox_ref = REF.dump_json(result.workspace.ref)
    conv.history = result.all_messages_json()
    return result.output


async def end_conversation(conv: Conversation) -> None:  # or on a TTL sweep
    if conv.sandbox_ref is not None:
        await ModalSandbox().destroy(REF.validate_json(conv.sandbox_ref))
  • Pydantic AI never terminates the sandbox: it bills until await <Sandbox>().destroy(ref) or its lifetime ends. destroy acts by ID (any instance works; Modal, E2B, Render, Sprites, Daytona and boxd use the instance's client=), does not wake the sandbox, returns quietly if it is gone, and raises ValueError for another provider's ref. after_run skips failed runs; one-shot jobs destroy ctx.workspace.ref in run_error.
  • await BoxdSandbox().fork(ref) copies a boxd machine, files and running processes included, into a new isolated one and returns its ref: pass it as workspace= to try another approach from the same state, then destroy it. It wakes a suspended machine first, and refuses one that isn't isolated or is on a named boxd network.
  • workspace= wins over the ref in message_history, and UI adapters strip refs from client-sent history, so keep the ref server-side. Attaching to a gone sandbox raises WorkspaceUnavailableError (so do rejected credentials); no empty replacement is created unless you pass workspace='new'.
  • Default images lack project dependencies (even pytest). To prepare one up front, build ModalSandboxBackend(working_dir=...) (or E2B/Render/Sprites/Daytona/boxd), use Workspace(backend) (write_text, run([...], timeout=300)), pass workspace=backend, destroy backend.ref in finally; a Sprites, Daytona or boxd backend you build needs await backend.aclose().
  • Idempotent per-turn setup goes in a @hooks.on.before_run hook: await ctx.workspace.run('test -d repo/.git || git clone <url> repo', shell=True, timeout=300). Clone into a subdirectory: shell, background commands, and output spills create .pydantic-ai-harness/ in the working directory, so git clone <url> . fails on later turns (or use git init + git fetch).
  • For an existing native sandbox pass workspace=ModalSandboxBackend(sandbox=...) (same for E2B, Sprites and Daytona); the capability's settings don't apply to it.
  • On E2B and Sprites, a background child that inherits stdout/stderr keeps run() waiting: redirect its output (start_command already does). On Daytona and boxd, run() returns when the command exits, without what the child prints later.
  • ModalSandbox warns ModalSandboxNoToolsWarning when the run has no Shell/FileSystem tools; pass warn_if_no_tools=False if only your own tools use it.

KubernetesSandbox

KubernetesSandbox(*, warm_pool, namespace='default', connection='port-forward', sandbox_timeout=86_400, working_dir=None, env=None) (backend KubernetesSandboxBackend, extra [kubernetes]) claims a pod from a kubernetes-sigs/agent-sandbox SandboxWarmPool on first use. Credentials are your kubeconfig (or the pod's service account in the cluster); 'port-forward' also needs kubectl.

  • The pool's SandboxTemplate must run sandboxd; commands run in that container, so its image decides the tools. Commands start in sandboxd's root directory (/workspace by default).
  • The ref is WorkspaceRef(provider='kubernetes', id='<namespace>/<claim name>'). Runs never delete the claim; sandbox_timeout does, or await KubernetesSandbox(warm_pool=...).destroy(ref) (quiet if gone).
  • From inside the cluster use connection='in-cluster-pod-ip' or 'in-cluster-service'; the template's NetworkPolicy must admit the agent on ports 8080 and 9090. Port-forward does not work on gVisor or Kata.
  • File operations go through the sandbox's shell.

SmolSandbox

SmolSandbox(*, image=None, cpus=None, memory_mb=None, allow_internet_access=False, working_dir='/workspace', env=None) (backend SmolSandboxBackend, extra [smol]) boots a local smolvm microVM on first use. No credentials; needs macOS on Apple silicon, or Linux x86_64/aarch64 with glibc 2.34+ and access to /dev/kvm.

  • Without image it runs smolvm's Alpine (BusyBox, root, no Python/git/rg); pass an OCI image such as image='python:3.12-alpine'. No network unless allow_internet_access=True, which is unrestricted egress from this machine's network (LAN, metadata endpoints), not internet-only. working_dir is created.
  • The ref is WorkspaceRef(provider='smol', id='pydantic-ai-<32 hex>'); from history, workspace=ref, backend(ref) or destroy(ref) any other id raises ValueError, so a ref can't reach other tools' VMs. A VM on this machine goes in as workspace=SmolSandboxBackend(sandbox=machine) (or ref= with its name), which history doesn't reattach. The name isn't proof of origin (any process of this OS user can use it), so only pass refs the app stored. Runs never delete the VM, and VMs have no TTL: await SmolSandbox().destroy(ref) (boots a stopped VM to delete it; quiet if gone; waits for a boot in progress, so a finally that destroys backend.ref also covers a run cancelled mid-boot).
  • Reattaching in the same process works as for the cloud sandboxes. Across processes, smolmachines does not flush a VM's disk when the process exits, so stop it first: await (await SmolSandbox().backend(ref).get_sandbox()).stop().
  • File operations go through the VM's shell, 64 KiB per command; multi-MiB files take seconds.

BoxLiteSandbox

BoxLiteSandbox(*, image=None, cpus=None, memory_mb=None, allow_internet_access=False, allowed_hosts=None, working_dir='/workspace', env=None, box_options=None) (backend BoxLiteSandboxBackend, extra [boxlite]) creates a local BoxLite micro-VM on first use. No credentials or daemon; needs macOS 14+ on Apple silicon, or Linux x86_64/aarch64 with glibc 2.28+ and access to /dev/kvm; Python up to 3.14.

  • Without image it runs Alpine (BusyBox, root, no Python/git/rg); pass e.g. image='python:3.12-slim'. No network unless allow_internet_access=True; allowed_hosts=['pypi.org', ...] limits egress to those hosts, IPs or CIDRs. working_dir is created.
  • box_options passes other boxlite.BoxOptions arguments to a new box (volumes, ports, secrets, disk_size_gb, user); the ones the named settings own raise UserError. Runtime-wide settings (registry logins, home dir) go to boxlite.Boxlite.init_default(boxlite.Options(...)) before first use.
  • The ref is WorkspaceRef(provider='boxlite', id='pydantic-ai-<32 hex>'); from history, workspace=ref, backend(ref) or destroy(ref) any other id raises ValueError. An existing box goes in as workspace=BoxLiteSandboxBackend(sandbox=box) (or ref= with any name), which history doesn't reattach. Runs never delete the box: await BoxLiteSandbox().destroy(ref) (quiet if gone; waits for a creation in progress). A box stops, keeping its disk, when the process lets go of it; its ref boots it again. One process at a time can use a BoxLite home (~/.boxlite or BOXLITE_HOME); another gets WorkspaceUnavailableError.
  • Output is read while the command runs and for at most 2 s after it exits, so a background job never holds a command open; output past 10 MiB stops the command with WorkspaceOutputLimitError.

DockerContainer

DockerContainer(*, image=None, working_dir='/workspace', env=None, allow_internet_access=True, docker_args=(), executable='docker') (backend DockerContainerBackend) runs commands and file operations in a local container through the docker CLI (executable='podman' works). No extra, no SDK.

  • The first operation creates the container; the image needs a POSIX sh, and its own entrypoint and command don't run. Creating the container (and pulling) is not bounded by a command's timeout. Without image the capability only attaches and removes by ref, and leaves a run without a ref to other workspace capabilities. working_dir=None uses the image's WORKDIR; a missing working_dir is created on first use, in an attached container too.
  • The ref is WorkspaceRef(provider='docker', id='pydantic-ai-<hex>'); attaching starts a stopped container, so it comes back. The container outlives the run: await DockerContainer().destroy(ref). Leftovers carry the label ai.pydantic.workspace=true: docker ps --all --filter label=ai.pydantic.workspace=true.
  • Network on Docker's default bridge (internet plus LAN and host services); allow_internet_access=False adds --network none, which Docker refuses next to a --network in docker_args. docker_args: ['--memory', '2g'], ['--volume', f'{project}:/workspace'] to work on a host directory.
  • A missing working_dir is created as the container's user: with --user, use a mount or working_dir=None.
  • New containers get --pids-limit 2048; a --pids-limit in docker_args replaces it (-1 removes it).
  • With Docker, a background child holding stdout/stderr open keeps run() waiting: redirect it. With Podman, run() returns when the command exits.

OpenSandbox

OpenSandbox(*, image=None, connection=None, resource=None, allow_internet_access=True, network_policy=None, secure_access=False, sandbox_timeout=None, working_dir='/workspace', env=None) (backend OpenSandboxBackend, extra [opensandbox]) runs commands and file operations in a sandbox on your own OpenSandbox server, which runs it as a Docker container or a Kubernetes pod.

  • The server is found through OPEN_SANDBOX_DOMAIN (default localhost:8080) and OPEN_SANDBOX_API_KEY if it needs one, or connection=ConnectionConfig(...); on Kubernetes without the ingress gateway, ConnectionConfig(use_server_proxy=True). It needs execd 1.1.1 or later (the server's execd_image).
  • Requests use plain HTTP unless the domain starts with https://. Without connection, an https:// OPEN_SANDBOX_DOMAIN sends every request over HTTPS. A ConnectionConfig you pass is used as given; it needs both an https:// domain and protocol='https', because the SDK sends command and file requests with protocol. Command and file requests carry commands, file contents and env values.
  • The first operation creates the sandbox; the image needs a POSIX sh. Without image the capability only attaches and destroys by ref. The ref is WorkspaceRef(provider='opensandbox', id=<sandbox id>); the sandbox outlives the run: await OpenSandbox().destroy(ref), or sandbox_timeout= to have the server delete it after that many idle seconds (renewed by use; a single command must finish within it).
  • resource={'cpu': '500m', 'memory': '1Gi'} sizes a new sandbox (default one CPU, 2 GiB).
  • Each command takes about 0.2 s and file operations run as commands, so large files are slow (seconds per MiB).
  • allow_internet_access=False asks the server for a deny-all egress policy on a new sandbox; the default sends none, so the server's configuration applies. The server's egress sidecar enforces it: its default dns mode filters DNS lookups only, and dns+nft also blocks direct IP connections. asyncio only.
  • network_policy= takes the SDK's NetworkPolicy for a new sandbox, passed through unchanged; it cannot be combined with allow_internet_access=False (UserError; ValueError on the backend).
  • secure_access=True asks the server to turn on Secure Access for a new sandbox (Kubernetes with the ingress gateway; Docker rejects it). The gateway and server proxy then require the sandbox's token, which the SDK fetches with the server API key and sends automatically. Requests that reach the pod directly are not checked.
  • Without the server proxy, the SDK does not send the server API key to a sandbox's endpoint; the endpoint checks only what the server configures for it. Attaching never changes allow_internet_access, network_policy or secure_access.

Remote hosts and bubblewrap

SSHWorkspace

SSHWorkspace(destination, *, working_dir=None, read_only=False, env=None, ssh_args=()) (backend SSHWorkspaceBackend) runs commands and file operations on a remote host through your ssh client, so keys, ports and jump hosts come from ~/.ssh/config and ssh-agent. No extra, no SDK.

  • Key auth only: ssh runs with BatchMode=yes, so a password or passphrase prompt fails at once.
  • No isolation: commands have the remote user's full authority. The host needs a POSIX sh.
  • working_dir defaults to the login directory and must exist. env adds to the remote login environment; nothing from your machine is passed through.
  • The ref is WorkspaceRef(provider='ssh', id='user@host:/dir'); the capability accepts only its own, and SSHWorkspace(...).backend(ref) attaches without connecting. Nothing to destroy.
  • Every operation is a connection: turn on ControlMaster auto + ControlPersist for speed.
  • A background child holding stdout/stderr open keeps run() waiting (sshd waits for the output): redirect it, as in server > server.log 2>&1 &.

BubblewrappedWorkspace

BubblewrappedWorkspace(wrapped_capability, *, network=False, bwrap_args=()) runs the wrapped workspace capability's commands in a Linux bwrap wrapper on that workspace's host: BubblewrappedWorkspace(SSHWorkspace('dev@box', working_dir='/srv/app')) or BubblewrappedWorkspace(LocalWorkspace('.')). The host needs bwrap and user namespaces.

  • Best-effort convenience to keep a trusted agent from accidentally changing files outside its working directory; not a security boundary and not a sandbox for untrusted code or untrusted agents (commands share the host kernel and run as the host user; for untrusted agents use a VM or microVM-backed sandbox).
  • Read-only host, writable working dir, private /tmp, empty /run (no daemon sockets), required user namespace, no capabilities. bwrap_args come after the defaults (['--bind', p, p], ['--tmpfs', p]).
  • network=False (default): no network, plus a Codex-style seccomp filter: no socket connect/bind/listen (host Unix sockets and the wrapper's own loopback included), no Unix datagram sockets, no ptrace/io_uring. Tests that start a local server, and Python 3.14's default forkserver multiprocessing (use get_context('spawn')), need network=True: host network and ports, no filter.
  • File methods (and the FileSystem/Coder file tools) run in the wrapper as shell commands, so a swapped-in symlink can't lead a write outside the working dir.
  • Do not set working_dir to ~ (or a directory containing it): only the default startup locations (~/.ssh, ~/.pam_environment, ~/.bashrc, ~/.bashrc.d, ~/.cargo/env, ~/.local/bin/env, ~/.zshenv, ~/.cshrc, ~/.tcshrc, ~/.config/fish) are mounted read-only (created if missing; a symlink there is unavailable), while interactive login files (~/.profile, ~/.bash_profile) and configuration under a non-default $XDG_CONFIG_HOME remain writable. Always pass a project working_dir.
  • The host PID namespace is shared, so Shell background jobs survive the call (and the run), and wrapped commands can see and signal the host user's processes.
  • Commands read everything the host user can (hide ~/.ssh with --tmpfs), and files they write (Git hooks, Makefile) run outside the wrapper if you run them later.
  • The ref and backend are the wrapped workspace's. BubblewrapWorkspace is the wrapper for building a workspace= directly: BubblewrapWorkspace(Workspace(SSHWorkspaceBackend('dev@box', working_dir='/srv/app'))).

Durable execution

Put the workspace, harness capabilities, and TemporalDurability() / DBOSDurability() / PrefectDurability() on the agent at construction, with a name. The run records its workspace ref; every activity reattaches to that environment.

Feature Temporal DBOS Prefect
File tools, background jobs, persist_cwd, FileChangeRequestedEvent Yes Yes Yes
run_command, shell Yes, within the activity timeout Yes Yes
Live tool events File-change request and write events only Yes Yes
delegate_task To self only Yes Yes
  • Temporal activities get 60 s; run_command and shell get 300 s. Longer run_command timeouts need SetToolMetadata(tools=['run_command'], temporal=ActivityConfig(start_to_close_timeout=...)); for longer work use start_command or shell(mode='background'). Approval listeners re-run on replay.
  • A worker dying between sandbox creation and ref recording leaves an orphan sandbox. Adding or removing capabilities changes replay history: drain workflows or version workers first.

RepoContext

Loads one instruction file per directory (the first of filenames, AGENTS.md then CLAUDE.md) into instructions at run start (stable within a run; a change between runs of a conversation is appended rather than rewritten). With expose_inventory_tool=True it exposes inventory_agent_context() (locates .claude/.agents/.codex/.grok skills, agents, hooks), which is left out when FileSystem is on the agent and its asset roots are inside the working directory (the defaults are; unless_tool_kind=READ_FILE_KIND).

from pydantic_ai_harness import RepoContext

RepoContext(walk_up='repo_root')  # also load the ancestors up to the nearest `.git`

walk_up=None (the default) scans only the working directory; 'repo_root' walks up to the nearest directory holding one of root_markers (('.git',); Coder uses this); 'all' walks every ancestor. home_dir walks up to that ancestor (and stops 'all' there); it cannot be combined with walk_up='repo_root'. It is a workspace path (on E2B, /home/user, not Path.home()); ~ raises UserError. Use keyword-only autoload_symlink_roots=() to authorize shared instruction targets for autoload only; the default is empty. Authorize only a directory of trusted instructions, not / or all of home; repository links can select readable files beneath it. See Repo Context for resolution, labels, and backend requirements.

nested_traversal=True (the default) surfaces the instruction files between the working directory and whatever FileSystem reads, lists, writes or edits in, shallowest first, each directory once while its note is in the conversation (nested_inject='contents' by default, or 'pointer'), in the message tail so the cached prefix stays stable. Notes are workspace-sourced DataCapabilityParts, ContextFilePathPart (<context_file_path>) or ContextFilePart (<context_file path="...">), from pydantic_ai_harness.repo_context; find them by type, and their directory field. With it on, the instructions also gain one static sentence: each instruction file applies to its subtree, the deeper file wins, and the user's direct instructions win over all of them. nested_max_chars (32,000 by default) caps the 'contents' inlined per conversation, counting the inlined notes still in history, so a compaction frees room: a file that would go past it becomes a pointer; None removes the cap. It doesn't apply with nested_inject='pointer'. nested_exclude (('node_modules', '.venv', 'venv', 'site-packages') by default) lists fnmatch patterns for directory names below the working directory whose instruction files nested traversal never surfaces, nor anything beneath them.

To pin files a user accepted (instruction files, SKILL.md, sub-agent definitions), record file_digest(workspace, path) at acceptance and pass the digests to await pin_accepted_files(workspace, accepted, changed='none') (both from pydantic_ai_harness.repo_context) before each run: a changed file is capped at changed in authority_of until its new digest is recorded.

ReportStaleFiles

Tells the model when a file a FileSystem tool read or wrote has since changed or been deleted in the workspace, by someone else (a user, another agent, a build step). It records each file's content hash from FileReadEvent / FileWrittenEvent and, before each model request, reads the recorded files through ctx.workspace; a mismatch is named in a request-scoped StaleFilesPart (.report_stale_files, with the workspace's authority), on every request until the model reads the file again (a deleted file once). The model's own writes and edits refresh the record. Records are per conversation and workspace, in memory on the instance, so a continued run hears about changes made between runs.

from pydantic_ai_harness import FileSystem, ReportStaleFiles

capabilities = [FileSystem(), ReportStaleFiles(max_tracked_files=100, max_reported_files=20, max_checked_bytes=8 << 20)]

Before every request, each recorded file costs a sequential realpath, stat and read, most recent first, up to max_checked_bytes; lower max_tracked_files on a slow remote workspace. A file whose real path changed since it was read is skipped. Not supported under durable execution: the run fails at its start.

Macroscope

run_macroscope_review(base=None) runs macroscope codereview --raw in the workspace and returns a MacroscopeReview of MacroscopeIssue findings; the agent fixes them with its other tools. No extra, but the macroscope CLI must be installed and signed in inside the workspace. Options: base, command='macroscope', timeout=600.0, guidance (None default text, '' none), id='macroscope' (keyword-only; capability and toolset id, so two with the same id merge or raise, and it works under durable execution; a Temporal activity gets timeout + 30s). A review the CLI stopped early still returns its findings, with status='timed_out' or 'interrupted'. Pair with Coder or FileSystem/Shell: Agent(..., capabilities=[LocalWorkspace('.'), Coder(), Macroscope(base='main')]).

LocalStack

AWS CLI against emulated AWS. It does not use the run workspace: aws and docker run on the agent's host. No extra. Tools: aws_cli (command without aws or --endpoint-url; argv, no shell) and localstack_health.

LocalStack(allowed_services=['s3', 'dynamodb']) connects to an instance you started; LocalStack(manage_container=True) starts a fresh Docker container per run on the endpoint_url port, so concurrent runs must share one inside async with agent:. allowed_services/denied_services are mutually exclusive. The default image needs LOCALSTACK_AUTH_TOKEN (forwarded automatically). The AWS CLI can read and write host files (file://, s3 cp), but gets its own empty AWS config and credentials files instead of ~/.aws/config and ~/.aws/credentials.

See also

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at b2d64a3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
compatibility
Requires Python 3.11+
metadata
{
  "version": "0.2.0",
  "author": "pydantic"
}

README badge

README badge for pydantic/pydantic-ai/pydantic-ai-harness