Index of every page on this site: llms.txt
All skills
pydantic avatar

/pydantic-ai-harness

@b2d64a3
by pydanticpydantic/pydantic-ai21k stars
3,221
Run it yourself

Extend Pydantic AI agents with capabilities from pydantic-ai-harness -- the Coder coding agent, file and shell tools in local or sandboxed workspaces (Modal, E2B, Render, Sprites, Daytona, boxd, smolvm, BoxLite), Code Mode, sub-agents and planning, model routing, memory and skills, context compaction, guardrails and spend limits, web research and browsers, hosted SaaS integrations, step persistence, and background agents on Render Workflows. Use when the user mentions pydantic-ai-harness or pydantic_ai_harness, imports a harness capability such as Coder, CodeMode, FileSystem, Shell, SubAgents, ModelRouter, Memory, or ToolGuardrail, or wants a Pydantic AI agent that edits files, runs commands or agent-written Python, delegates, routes work between models, remembers, manages long context, stays within limits, or runs model and tool operations as chained Render task runs.

  • 11 files
  • 334 KB
  • MIT
  • Updated 2 days ago
  • GitHub
Use this Skill: https://skilld.dev/gh/pydantic/pydantic-ai/pydantic-ai-harness

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesCODE-MODE.md

≈5.8k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Code Mode

CodeMode wraps eligible tools into a single run_code tool so the model can write Python that loops, branches, aggregates, and parallelizes multiple tool calls inside a sandbox.

Use it when the agent needs to call several tools, transform intermediate results, run concurrent tool work with asyncio.gather, or anywhere a simple Python script is more reliable than the model alone, such as for mathematics.

Install

Code Mode needs the Monty sandbox, pulled in by the codemode extra:

uv add "pydantic-ai-harness[codemode,anthropic]"   # `code-mode` is an alias; `anthropic` for the examples

Without it, importing CodeMode raises ImportError: pydantic-monty is required for CodeMode.

Basic Pattern

from pydantic_ai import Agent

from pydantic_ai_harness import CodeMode

agent = Agent('anthropic:claude-opus-5-5', capabilities=[CodeMode()])


@agent.tool_plain
def get_weather(city: str) -> dict:
    return {'city': city, 'temp_f': 72, 'condition': 'sunny'}


@agent.tool_plain
def convert_temp(fahrenheit: float) -> float:
    return round((fahrenheit - 32) * 5 / 9, 1)

The model could generate code like:

import asyncio

paris, tokyo = await asyncio.gather(
    get_weather(city='Paris'),
    get_weather(city='Tokyo'),
)
paris_c = await convert_temp(fahrenheit=paris['temp_f'])
tokyo_c = await convert_temp(fahrenheit=tokyo['temp_f'])
{'paris': paris_c, 'tokyo': tokyo_c}

Choose Which Tools Are Sandboxed

The tools parameter (a Pydantic AI ToolSelector) controls which tools move behind run_code.

from pydantic_ai_harness import CodeMode

CodeMode(tools='all')
CodeMode(tools=['search', 'fetch'])
CodeMode(tools=lambda ctx, td: td.name in {'search', 'fetch'})
CodeMode(tools={'code_mode': True})

Metadata-based selection is useful when the project already groups tools into toolsets:

from pydantic_ai import Agent
from pydantic_ai.toolsets import FunctionToolset
from pydantic_ai_harness import CodeMode

search_tools = FunctionToolset(tools=[search, fetch]).with_metadata(code_mode=True)

agent = Agent(
    'anthropic:claude-opus-5-5',
    toolsets=[search_tools],
    capabilities=[CodeMode(tools={'code_mode': True})],
)

Non-matching tools remain regular tool calls. Some tools always stay native even with tools='all': framework control tools, undiscovered deferred (defer_loading=True) tools, requires_approval=True and external tools, native fallbacks, and other code-execution tools (any tool with code_arg_name metadata) -- including Shell's run_command, start_command, and shell, and CapabilityCreation's author_capability. Shell's check_command/stop_command and all FileSystem tools are folded in. Provider-native tools (native=True) execute server-side and never reach run_code.

With the default tools='all', every tool from MCP or hosted-integration capabilities (for example GitHub or Linear write tools) moves behind run_code too. Only tools registered with requires_approval=True and external tools stay native: tools gated with .approval_required() are still sandboxed, where a gated call fails as a retry unless a HandleDeferredToolCalls capability resolves it inline. When only some tools should be batched, tag those tools' toolset with .with_metadata(code_mode=True) and use CodeMode(tools={'code_mode': True}), so approval-gated and integration tools stay regular tool calls.

Return Values

run_code captures the last non-None expression automatically; an assignment on the last line returns nothing.

Scenario Return
Final expression, no print output Last expression value
Final assignment or None, no print output {}
Print output, no final expression {'output': '<printed text>'}
Print output and final expression {'output': '<printed text>', 'result': <last expression>}
Multimodal final expression Returned natively for model processing (with prints: a list, text first)

If the user expects a raw dict or list back, avoid unnecessary print() statements. Printed output is capped at 10 MiB. Tool results reach the sandbox as JSON: Decimal, UUID and datetime arrive as strings and mapping keys are stringified (bytes cross unchanged). A nested tool's ToolReturn gives the sandbox its return_value; its content (e.g. a screenshot) skips the sandbox and reaches the model with the run_code result, as for a direct call.

Limits and Retries

from pydantic_ai_harness import CodeMode

CodeMode(
    tools='all',
    max_retries=3,
    max_tool_calls=25,
    resource_limits={'max_duration_seconds': 10, 'max_memory': 128 * 1024 * 1024},
)
  • max_retries (default 3): execution errors (including syntax/type errors) are sent back to the model to redraft, up to this many times. A nested tool's ToolFailed the snippet does not catch is the exception: run_code returns a failed result, as when the model calls that tool directly, and spends no retry. Pool creation, connection, and checkout errors propagate to the code calling the agent, not the model. Inside Temporal they follow the workflow failure policy below. Capacity refusals get one checkout retry after a short backoff before propagating.
  • max_tool_calls (default 100): nested calls per run_code call. A call past the budget fails at its call site; if uncaught, the model gets a retry listing the calls that already started.
  • The run's UsageLimits(tool_calls_limit=...) counts nested calls and the run_code call that makes them. A nested call that would exceed it ends the run with UsageLimitExceeded, even when the snippet catches the error. A call launched early by speculate counts from the moment it starts, including one the snippet never claims. A launch the limit refused runs cold if the snippet asks for it.
  • resource_limits (default: 30 s execution and 256 MiB heap per snippet): keys max_duration_seconds, max_memory, max_suspensions (default 1,000 host interactions per REPL session, shared across snippets). Unknown keys raise UserError. Time awaiting tools does not count; sleeps get a separate allowance of max_duration_seconds. A snippet that hits the time limit is stopped and its REPL session is reset. 'unlimited' removes time and memory caps only when another boundary supplies them; max_suspensions can never be disabled.

REPL State

State persists between run_code calls during the same agent run. Imports, variables, and helper functions carry over until the run ends. The model passes restart: true in the run_code arguments to reset state. A timeout, worker crash, or interrupted eager statement also resets the session, and the retry message says so -- the next snippet must recreate its state.

A pool's request_timeout and a remote turn_timeout ask the model to do less work per call. Remote session and idle timeouts explain that state is lost. Sandbox memory-limit errors and MemoryErrors without a traceback reset the session and ask the model to use smaller batches. Other snippet or nested-tool MemoryErrors keep state unless their diagnostic is indistinguishable from a sandbox memory limit: Monty does not guarantee heap integrity after a limit fires. Exceeding the 10 MiB printed-output cap also resets the session because it has the same diagnostic. Remote host-side error details are omitted from retries because they may contain pool URLs or credentials; the original exception remains the cause for application-side diagnostics. Retries list nested calls that already started so the model can avoid repeating their side effects.

State does not carry over to the next agent run unless repl_store is set: an object with async load(ctx) -> bytes | None and save(ctx, state: bytes | None) methods (pydantic_ai_harness.code_mode.CodeModeReplStore), usually keyed by ctx.conversation_id. A run that calls run_code restores the saved state before its first snippet and saves the REPL when it ends; restart: true replaces it (save(ctx, None) when nothing replaces it), and a state saved under other resource_limits or that Monty cannot load is dropped so the run starts fresh. Return only states the application saved: Monty does not authenticate them. Restored state skips the first-snippet type check. Store errors are logged, not raised; a run whose load failed saves nothing unless it restarts, so the unread state survives. The store is not used under durable execution.

Sandbox Restrictions

Code runs inside Monty, an implementation of Python which intentionally supports only a subset of features.

Key restrictions:

  • No third-party imports
  • No import *
  • Only a small stdlib subset is allowed, and each must be imported before use: sys, typing, asyncio, math, json, re, unicodedata, datetime, time, random, os, pathlib, collections, itertools, functools, dataclasses, copy, base64, binascii, bisect, csv, decimal, fnmatch, hashlib, os.path, tomllib, annotationlib
  • asyncio.gather(...) accepts positional awaitables but no keyword arguments; other task creation and wait APIs are unavailable
  • No clock or randomness by default: datetime.datetime.now(), datetime.date.today(), time.time(), and unseeded random require an os_access handler (or, for the clock, os_policy); time.sleep and asyncio.sleep really wait, within the sleep allowance
  • Filesystem I/O requires an os_access handler or a mount; os.getenv and os.environ require an os_access handler
  • A sandboxed tool named after a Python builtin (list, open, print, ...) raises UserError; rename it or use toolset.prefixed(...)
  • Approval-required (requires_approval=True) and external (kind='external') tools stay native under tools='all', so their deferred calls pause the run as usual. An explicit selector can still sandbox them. A sandboxed call that needs approval or is deferred, including one from a toolset wrapped with .approval_required() or a tool that raises ApprovalRequired or CallDeferred from its body (both keep kind='function'), needs a HandleDeferredToolCalls (or equivalent) capability to resolve it inline; without one, it raises an error that surfaces to the model as a retry

The sandbox constrains the model-generated Python, not the implementation of the tools it calls. Wrapped tools retain their normal host and network access, so expose only tools with the authority and input validation appropriate for model-generated calls.

When a generated example keeps failing, check these restrictions before changing the rest of the agent.

Host Access: mount and os_access

Leave both unset unless the task requires host access, and grant only what the task needs. Both are fixed when CodeMode is built, so construct it per request to scope access to that request.

from pathlib import Path

from pydantic_monty import MountDir, OSAccess

from pydantic_ai_harness import CodeMode

work = Path('agent-work')
work.mkdir(exist_ok=True)  # the host directory must exist
CodeMode(mount=MountDir(virtual_path='/work', host_path=str(work), mode='read-write'))
CodeMode(os_access=OSAccess(environ={'API_BASE': 'https://api.example.com'}))
  • mount takes a MountDir or a list of them; host_path must already exist (MountDir raises TypeError otherwise). The default mode='overlay' is copy-on-write: writes are visible only within the current run_code call and never reach the host. Use mode='read-write' when writes must persist (including across run_code calls), mode='read-only' to forbid them.
  • os_access takes an AbstractOS (e.g. OSAccess, isolated in-memory filesystem and env, host clock) or a keyword-argument callback def handler(*, name, args, kwargs, **_) (may be async). Return any value (including None) to answer the call; return pydantic_monty.NOT_HANDLED to refuse it (raises in the sandbox and burns a retry). The positional (name, args, kwargs) form is deprecated.
  • Mounts are directories on the machine running the agent, not the run's workspace. With a remote sandbox such as ModalSandbox, FileSystem and Shell act in the sandbox while mounted pathlib code still touches the host -- for files the model shares with its commands, sandbox the workspace tools (FileSystem) instead of mounting.

Other Options

CodeMode(
    tools: ToolSelector = 'all',
    max_retries: int = 3,
    *,
    max_tool_calls: int = 100,
    os_access: CodeModeOS | None = None,
    mount: CodeModeMount | None = None,
    os_policy: CodeModeOSPolicy | None = None,  # Monty clock/timezone policies over the defaults
    resource_limits: CodeModeResourceLimits | Literal['unlimited'] | None = None,
    eager: bool = False,                   # run complete streamed statements before the call finishes
    speculate: Sequence[str] | Literal['declared'] | None = None,  # start read-only calls while streaming
    monty_sandbox: str | Callable[[], AsyncMontyRemote | Awaitable[AsyncMontyRemote]] | None = None,  # an http(s):// server root URL, or a function returning the pool
    repl_store: CodeModeReplStore | None = None,  # keep the REPL across agent runs
    dynamic_catalog: bool = True,          # keep the tool catalog in an appending instruction block
)
  • dynamic_catalog=True (default): moves sandboxed-tool signatures out of run_code's description into an instruction block (toolset:code_mode:catalog, on_change='append'), keeps deferred tools that run_code will take over off the wire until they're found, and appends the updated catalog when one is found (plus a short DiscoveryAnnouncementPart, an InstructionCapabilityPart whose tools field lists what it announces, when Tool Search found it), so discoveries don't rewrite the cached prompt. Discovery then goes through the local search_tools or load_capability, which also searches deferred tools a selector keeps native (the native tool search isn't sent then); with the 'bm25'/'regex' strategies, such tools are still declared for the native search. The catalog takes the toolset id code_mode, so another instruction-contributing toolset with that id raises a duplicate-id UserError; rename it. A callable selector must give the same answer for a tool before and after it's revealed, or the cache moves. dynamic_catalog=False keeps the catalog in run_code's description instead (the default in earlier releases), so a reveal rewrites the description and the cached prompt.
  • eager=True: side effects from early statements cannot be rolled back, and hooks/approval on run_code run only after the call finishes streaming. Only applies when run_code is the first tool call in the response.
  • speculate: pass tool names that are safe to run early, or 'declared' to trust Tool(metadata={'read_only': True}) and MCP readOnlyHint. Only calls with literal keyword arguments start early; unclaimed launches still cost what they cost. CodeMode.speculation_stats reports launched/adopted/evicted.
  • eager and speculate put runs in streaming mode, need asyncio, and are inactive under durable execution (Temporal, DBOS, Prefect).
  • os_policy: Monty's OSPolicy, merged key by key over the defaults (clock, sleeps, and random seeding routed to os_access). {'datetime': 'system', 'timezone': 'Europe/Paris'} gives the sandbox the worker's clock in that zone with no handler; {'timezone': ...} alone keeps the clock on os_access but makes astimezone() and %Z report the zone. Keep the defaults under Temporal: a 'system' clock or random_start is re-read on replay. Fixed values stay fixed.
  • With InputGuardrail(parallel=True), nested tool calls from eager and speculate wait for the guard's verdict, so a blocked prompt runs no tool.
  • monty_sandbox: an http:///https:// server root URL (no key; no path, query, or fragment; ws:///wss:// still work) or a sync or async function returning an AsyncMontyRemote (pass metadata= for an authorization header), to run code on a remote Monty worker over gRPC. A factory pool adds its request_timeout (10 s default) per-turn transport deadline; the configured max_duration_seconds also applies, except in Temporal workflow code. The factory runs once per run, or once per turn when turns run as activities. Only execution moves; tools, os_access, and prints stay host-side, and so do mounts unless turns run as activities, where the activity worker reads them. Run the worker on pydantic-monty>=1.5.0. monty_sandbox_url is the deprecated form.
  • Configure credentials for a remote monty_sandbox in the pool factory, not in the model's instructions or tool arguments.

Durable Execution

CodeMode works next to TemporalDurability() (install pydantic-ai-harness[codemode,temporal]), but run_code executes in workflow code and is re-run on replay. Keep snippets deterministic: put external reads, writes, and clock access in wrapped tools (which become activities), not in mount/os_access. Inside a Temporal workflow max_duration_seconds is disabled for code run in workflow code (memory and suspension caps still apply) and a workflow task that does not yield within 2 seconds fails, so move heavy computation into a tool.

With a remote monty_sandbox, each turn (the code between two waits on the host: nested calls, sleeps, deferred calls) runs as its own activity instead, so sandbox computation does not count against the 2 seconds and replay reads turn results from history. This needs a server that stores sessions (Logfire Agent Sandboxes does); on one that stores none, the run falls back to workflow code, and a storing server that cannot store a session's state raises a non-retried UserError. Runs stay workflow-side with id=None, under the deprecated TemporalAgent, with os_access set, with an 'overlay' mount (the default mode; use 'read-only' or 'read-write') or a write_bytes_limit, or when the workflow started before this change. Each turn takes and returns at most 1 MiB: longer output is cut, and longer tool results fail the snippet with a ValueError that gives the size and resets the session. Turns use TemporalDurability's activity_config; a turn that still fails after its retries fails the workflow with ActivityError. In a turn activity max_duration_seconds applies again, so keep start_to_close_timeout above it: Temporal's timeout does not stop a running turn. A retried turn repeats its 'read-write' mount writes. Recorded pool errors keep their type name, so non_retryable_error_types still match. A CodeMode a DynamicCapability returns keeps its turns in workflow code. Tool results and call arguments keep their types across turns, and the model sees the same result; nested-call arguments of a type Monty does not hand the host, or over 512 KiB, raise TypeError.

Setup errors reach workflow code with their original types. PydanticAIPlugin treats UserError as a workflow failure. Uncaught RuntimeError, MontyRefusedError, and other unclassified factory errors fail the workflow task instead: Temporal retries the task while the workflow remains running, so the client keeps waiting if setup keeps failing. Inspect the task history or worker logs for the original error. To end the workflow, catch expected setup exceptions and raise temporalio.exceptions.ApplicationError(..., non_retryable=True) from the original exception.

Agent Specs

When the user defines the agent in YAML or JSON, the loader needs to know how to build CodeMode. YAML files also need PyYAML: uv add "pydantic-ai-harness[codemode,anthropic]" "pydantic-ai-slim[spec]".

model: anthropic:claude-opus-5-5
capabilities:
  - CodeMode: {}   # or with arguments: CodeMode: {tools: ['search', 'fetch'], max_retries: 5}
from pydantic_ai import Agent

from pydantic_ai_harness import CodeMode

agent = Agent.from_file('agent.yaml', custom_capability_types=[CodeMode])

Observability

With Logfire or another OpenTelemetry backend, nested tool calls inside run_code produce child spans. The run_code tool return also carries history-only metadata keyed by nested call id (plus a speculation summary when speculate is set):

for msg in result.all_messages():
    for part in msg.parts:
        if isinstance(part, ToolReturnPart) and part.tool_kind == 'code_mode.run_code':
            tool_calls = part.metadata['tool_calls']    # dict[str, ToolCallPart]
            tool_returns = part.metadata['tool_returns'] # dict[str, ToolReturnPart]

For an offline test that drives run_code with a FunctionModel, see Testing and Debugging.

Recognizing run_code in Hooks

run_code declares the 'code_mode.run_code' tool kind, so its call parts are promoted to RunCodeCallPart (from pydantic_ai_harness.code_mode). Match it with isinstance, not by tool name; code holds the submitted snippet, and typed_args all the arguments validated as RunCodeArgs:

async def wrap_tool_execute(self, ctx, *, call, tool_def, args, handler):
    if isinstance(call, RunCodeCallPart) and call.code is not None:
        ...  # inspect or record call.code
    return await handler(args)

run_code parts recorded before the kind existed load as plain ToolCallParts; ToolCallPart.narrow_type(part, tool_kind='code_mode.run_code') promotes one.

IPython Code Mode (local agents only)

IPythonCodeMode gives the model the same run_code tool backed by a persistent IPython kernel: full CPython, any installed package, and state kept for the whole run. It is not a sandbox: model-written code runs with the agent process's permissions. Never use it in a webserver or on untrusted input unless the whole agent runs in a container or VM; use CodeMode there.

uv add "pydantic-ai-harness[ipython]"
from pydantic_ai import Agent

from pydantic_ai_harness import IPythonCodeMode

agent = Agent('anthropic:claude-opus-5-5', capabilities=[IPythonCodeMode()])
  • Tools become synchronous, keyword-only functions in the kernel; calls run one at a time and a failed call raises ToolCallError. tools= and max_tool_calls= work as in CodeMode.
  • timeout= (default 60 s, tool time excluded) interrupts long cells and keeps the kernel's state; a cell that ignores the interrupt or a kernel that exits gets a fresh kernel on the next call.
  • Each run starts a kernel on first use and stops it at the end. To keep state across runs, pass kernel=IPythonKernel() (from pydantic_ai_harness.ipython_code_mode) and shut it down yourself, e.g. with async with IPythonKernel() as kernel:.
  • Not supported: eager, speculate, dynamic_catalog, durable execution, mount/os_access.

See also

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at b2d64a3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
compatibility
Requires Python 3.11+
metadata
{
  "version": "0.2.0",
  "author": "pydantic"
}

README badge

README badge for pydantic/pydantic-ai/pydantic-ai-harness