Index of every page on this site: llms.txt
All skills
pydantic avatar

/pydantic-ai-harness

@b2d64a3
by pydanticpydantic/pydantic-ai21k stars
3,221
Run it yourself

Extend Pydantic AI agents with capabilities from pydantic-ai-harness -- the Coder coding agent, file and shell tools in local or sandboxed workspaces (Modal, E2B, Render, Sprites, Daytona, boxd, smolvm, BoxLite), Code Mode, sub-agents and planning, model routing, memory and skills, context compaction, guardrails and spend limits, web research and browsers, hosted SaaS integrations, step persistence, and background agents on Render Workflows. Use when the user mentions pydantic-ai-harness or pydantic_ai_harness, imports a harness capability such as Coder, CodeMode, FileSystem, Shell, SubAgents, ModelRouter, Memory, or ToolGuardrail, or wants a Pydantic AI agent that edits files, runs commands or agent-written Python, delegates, routes work between models, remembers, manages long context, stays within limits, or runs model and tool operations as chained Render task runs.

  • 11 files
  • 334 KB
  • MIT
  • Updated 2 days ago
  • GitHub
Use this Skill: https://skilld.dev/gh/pydantic/pydantic-ai/pydantic-ai-harness

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesTESTING-AND-DEBUGGING.md

≈4.5k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Testing and Debugging

Read this file when the user wants offline, deterministic tests for an agent that uses harness capabilities, or when a harness agent fails at run start or keeps retrying. For core TestModel, FunctionModel, Agent.override, and capture_run_messages() basics, load the building-pydantic-ai-agents skill's references/TESTING-AND-DEBUGGING.md instead of repeating them here.

The Pattern: Script the Model, Use Real Capabilities

Harness capabilities are real code (they write files, run commands, call answerers), so script the model with a FunctionModel and let the capabilities execute against temp directories and in-memory stores. Agent.override(model=..., workspace=...) swaps both without touching the production agent; workspace= takes a backend such as LocalWorkspaceBackend(tmp_dir). This also works on an agent with ModalSandbox/E2BSandbox/RenderSandbox/SpritesSandbox/DaytonaSandbox/BoxdSandbox/KubernetesSandbox/OpenSandbox attached: no sandbox is created, no factory needed.

  • FunctionModel: return a ToolCallPart on the first request and a TextPart once results are in.
  • TestModel(call_tools=[]): smoke test that the agent builds and runs. Plain TestModel() calls every tool with generated arguments -- real commands and writes for Shell/FileSystem -- so avoid it here.
  • Build the production agent, and every sub-agent, with defer_model_check=True (for example Agent('anthropic:claude-opus-5-5', defer_model_check=True, ...)): a provider model string otherwise needs the provider's API key when the Agent is constructed, so importing the module in a test without keys raises UserError. TrajectoryJudge(model='provider:...') builds its judge agent at construction and needs the key too; pass agent=Agent(..., defer_model_check=True, instructions=...) instead (you then write the judge instructions). Advisor, SubAgents(models=), SummarizingCompaction(model=), Summarize(model=), and LLMReminder resolve their model lazily.
  • Override a sub-agent's model next to the parent's: with parent.override(model=FunctionModel(script)), child.override(model=TestModel()):.
  • Add a throwaway tool to a production agent for one run: agent.run_sync(..., toolsets=[FunctionToolset([my_tool])]) (pydantic_ai.toolsets).
  • Hosted integrations (GitHub, Linear, LogfireMCP, ...) need a credential when the Agent is built. Pass auth as a function that reads the token from deps; a test that passes deps without a token gets a run with no tools from that integration. See Hosted Integrations.
  • Advisor under an overridden FunctionModel or TestModel runs as a local advisor function tool, so the scripted model sees and can call a tool named advisor.
  • Assert through result.all_messages(): ToolCallPart is in ModelResponse.parts, ToolReturnPart in ModelRequest.parts. A call that asked for a retry is a ToolReturnPart with outcome='retried'.

FileSystem, Shell, and Coder on a Temp Workspace

FileSystem, Shell, and Coder do all I/O through the run's workspace, so point it at a temp directory. LocalWorkspaceBackend is POSIX-only.

import tempfile
from pathlib import Path

from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import FunctionModel
from pydantic_ai.workspaces import LocalWorkspaceBackend

from pydantic_ai_harness import FileSystem, Shell

agent = Agent(capabilities=[FileSystem(), Shell()])  # production model set elsewhere


def script(messages, info):
    if len(messages) == 1:
        args = {'path': 'notes.txt', 'content': 'hi\n'}
        return ModelResponse(parts=[ToolCallPart('write_file', args)])
    if len(messages) == 3:  # the write result is in; read the file back
        return ModelResponse(parts=[ToolCallPart('run_command', {'command': 'cat notes.txt'})])
    return ModelResponse(parts=[TextPart('done')])


with tempfile.TemporaryDirectory() as tmp:
    with agent.override(model=FunctionModel(script), workspace=LocalWorkspaceBackend(tmp)):
        result = agent.run_sync('write a note')
    print(repr(Path(tmp, 'notes.txt').read_text()))
    #> 'hi\n'

returns = {
    part.tool_name: part.content
    for message in result.all_messages()
    for part in message.parts
    if isinstance(part, ToolReturnPart)
}
print(repr(returns['run_command']))
#> '[stdout]\nhi\n'
  • Tools in one response run concurrently, so the example reads the file back in the next model turn; do the same whenever a test needs ordering.
  • Coder() bundles FileSystem (FILE_TOOL_NAMES: read_file, write_file, edit_file, list_files, grep), Shell (the persistent shell tool only), RepoContext, SubAgents, and more. Script calls to those tool names. RepoContext (in Coder() by default) observes run events, which makes model requests stream, so a FunctionModel without stream_function= fails with AssertionError; script it with stream_function= (below) or use Coder(repo_context=False). The prebuilt coder_agent has no model and attaches LocalWorkspace on the current directory, so pass workspace= explicitly in tests.
  • Shell() denies rm, dd, shutdown, and similar by default; a denied command comes back to the model as a retry, not an exception.
  • pytest's tmp_path fixture works the same way: LocalWorkspaceBackend(tmp_path).
import tempfile

from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace
from pydantic_ai.models.function import DeltaToolCall, FunctionModel

from pydantic_ai_harness.coder import Coder


async def stream(messages, info):  # yield str for text, {index: DeltaToolCall} for calls
    if len(messages) == 1:
        args = '{"command": "echo hi"}'
        yield {0: DeltaToolCall(name='shell', json_args=args, tool_call_id='c1')}
    else:
        yield 'done'


with tempfile.TemporaryDirectory() as tmp:
    agent = Agent(
        FunctionModel(stream_function=stream),
        capabilities=[LocalWorkspace(tmp), Coder()],
    )
    print(agent.run_sync('say hi').output)
    #> done

AskUser with a Scripted Answerer

AskUser has no default answerer, so tests pass one that answers deterministically and records what it was asked. The model receives {header: [labels]}, or DECLINED text for AskUserResponse(cancelled=True).

from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart
from pydantic_ai.models.function import FunctionModel

from pydantic_ai_harness import AskUser
from pydantic_ai_harness.ask_user import AskUserAnswer, AskUserRequest, AskUserResponse

asked: list[AskUserRequest] = []


async def pick_last(request: AskUserRequest) -> AskUserResponse:
    asked.append(request)
    answers = [
        AskUserAnswer(header=q.header, selected=(q.options[-1].label,))
        for q in request.questions
    ]
    return AskUserResponse(answers=tuple(answers))


question = {
    'header': 'Database',
    'question': 'Which database?',
    'options': [{'label': 'SQLite'}, {'label': 'Postgres'}],
}


def script(messages, info):
    if len(messages) == 1:
        args = {'questions': [question]}
        return ModelResponse(parts=[ToolCallPart('ask_user_question', args)])
    return ModelResponse(parts=[TextPart(messages[-1].parts[-1].model_response_str())])


agent = Agent(FunctionModel(script), capabilities=[AskUser(answerer=pick_last)])
print(agent.run_sync('set up storage').output)
#> {"Database":["Postgres"]}
print(asked[0].questions[0].header)
#> Database
  • Schema violations (fewer than 2 options, duplicate headers, more than 10 questions, control characters) are retries to the model, and the answerer is never called.
  • An answerer response that does not fit its request (unknown header, label not offered, several picks on single-select) fails the whole run with ValueError, as check_response would; decline with cancelled=True instead.
  • AskUserRequestedEvent/AskUserAnsweredEvent let an observer capability (@on_event) assert on the exchange without being the answerer.
  • For AskUserAnswer(note=...), assert that the answer dictionary stays in ToolReturnPart.content and notes arrive in a separate UserPromptPart. Invalid notes raise ValueError through check_response. See Adding notes to an answer for supported content, framing and provenance.
  • Annotate ask_user_result values with AskUserResult from pydantic_ai_harness.ask_user. Pass the result directly to DeferredToolResults.calls for deferred answers, including notes.

Memory with an In-Memory Store

Memory() defaults to InMemoryMemoryStore. Seed it with InMemoryMemoryStore({'<scope>/<file>': text}), where scope is agent_name (default 'main'), prefixed by namespace/ when one is set, and read store.files afterwards. The snapshot is a MemorySnapshotPart kept in message history, which the model gets as data in a <memory> tag on the user channel. It's added on a run's first request only when memory changed since the last one.

from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart
from pydantic_ai.models.function import FunctionModel

from pydantic_ai_harness import Memory
from pydantic_ai_harness.memory import InMemoryMemoryStore

store = InMemoryMemoryStore({'main/MEMORY.md': '- prefers tabs'})
injected: list[str] = []


def script(messages, info):
    if len(messages) == 1:
        # The model gets the snapshot framed: its last user part's text is the `<memory>` element.
        injected.append(''.join(item.content for item in messages[0].parts[-1].content))
        return ModelResponse(parts=[ToolCallPart('write_memory', {'content': '- uses uv'})])
    return ModelResponse(parts=[TextPart('ok')])


Agent(FunctionModel(script), capabilities=[Memory(store=store)]).run_sync('hi')
print(injected[0].splitlines()[-2])
#> - prefers tabs
print(store.files['main/MEMORY.md'])
"""
- prefers tabs
- uses uv
"""

FileMemoryStore needs a workspace: without one the run fails at start with a UserError; attach a LocalWorkspace or pass FileMemoryStore(..., workspace=LocalWorkspaceBackend(tmp)).

CodeMode: Drive run_code Directly

Script the run_code call with the snippet you want to test. This needs the codemode extra; nested calls land in the return part's metadata.

from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import FunctionModel

from pydantic_ai_harness import CodeMode

CODE = """\
import asyncio
a, b = await asyncio.gather(double(x=2), double(x=5))
print('computed')
a + b"""


def script(messages, info):
    if len(messages) == 1:
        return ModelResponse(parts=[ToolCallPart('run_code', {'code': CODE})])
    return ModelResponse(parts=[TextPart('done')])


agent = Agent(FunctionModel(script), capabilities=[CodeMode()])


@agent.tool_plain
def double(x: int) -> int:
    return x * 2


result = agent.run_sync('go')
(ret,) = [
    p
    for m in result.all_messages()
    for p in m.parts
    if isinstance(p, ToolReturnPart) and p.tool_name == 'run_code'
]
print(ret.content)
#> {'output': 'computed\n', 'result': 14}
print([call.tool_name for call in ret.metadata['tool_calls'].values()])
#> ['double', 'double']
print(sorted(r.content for r in ret.metadata['tool_returns'].values()))
#> [4, 10]

info.function_tools inside the model function shows what the model was offered: here only run_code, since double moved into the sandbox.

Capture Spans

In an app, logfire.configure() plus logfire.instrument_pydantic_ai() traces every run; harness capabilities add their own spans (memory.write, nested tool calls under run_code). In tests, attach the core Instrumentation capability with an in-memory exporter:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from pydantic_ai import Agent
from pydantic_ai.capabilities import Instrumentation
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart
from pydantic_ai.models.function import FunctionModel
from pydantic_ai.models.instrumented import InstrumentationSettings

from pydantic_ai_harness import Memory

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracing = Instrumentation(settings=InstrumentationSettings(tracer_provider=provider))


def script(messages, info):
    if len(messages) == 1:
        return ModelResponse(parts=[ToolCallPart('write_memory', {'content': '- x'})])
    return ModelResponse(parts=[TextPart('ok')])


Agent(FunctionModel(script), capabilities=[Memory(), tracing]).run_sync('hi')
names = [span.name for span in exporter.get_finished_spans()]
print('memory.write' in names, 'execute_tool write_memory' in names)
#> True True

With Logfire's pytest plugin, the capfire fixture (logfire.testing.CaptureLogfire) captures the same spans: call Agent.instrument_all(True) in a fixture, then read capfire.exporter.exported_spans_as_dict().

Debug Common Failures

Missing workspace. FileSystem, Shell, Coder, and Memory(store=FileMemoryStore(...)) fail the run at start, before any model request:

from pydantic_ai import Agent
from pydantic_ai.exceptions import UserError

from pydantic_ai_harness import Shell

try:
    Agent('test', capabilities=[Shell()]).run_sync('hi')
except UserError as e:
    print(str(e).split('. ')[0])
    #> `Shell` needs a workspace, but none is attached to this run

Fix: add LocalWorkspace('.') (from pydantic_ai.capabilities import LocalWorkspace) or a sandbox capability such as ModalSandbox() to capabilities, or pass workspace= to the run. If the message says the history "continues in workspace <provider>:<id>", the conversation was started elsewhere; attach the capability for that provider or pass workspace='new'.

Missing extra. Importing a capability without its extra usually raises ImportError naming the package and the install command, e.g. pydantic-monty is required for CodeMode. Install it with: pip install "pydantic-ai-harness[code-mode]". Install the extra from that capability's reference or docs page.

Monty restrictions. Generated code that breaks a sandbox rule surfaces as a ToolReturnPart with outcome='retried' on run_code, and the run keeps going until max_retries is used up. Typical contents: ModuleNotFoundError: No module named 'numpy' (third-party import) and RuntimeError: 'datetime.now' is not supported in this environment (clock without os_access). See Code Mode for the rules and how to grant host access.

Asserting a SystemReminders reminder. A plain firing is a ModelRequest holding one ReminderPart(scope='request') (an InstructionCapabilityPart from pydantic_ai_harness.system_reminders; find it with isinstance) in result.all_messages(), whose content is the reminder text (wrapped in <tag> only when Reminder(tag=...) is set); inside a FunctionModel script the model sees only the current one, as <system>-tagged user text at the end of the last request. A Reminder(scope='conversation') firing is a ReminderPart with scope='conversation' instead, which every later request sends again. An LLMReminder firing is instead a lasting GeneratedReminderPart (a DataCapabilityPart): the generated text, rendered in <generated_reminder model="..."> on the user channel with a note in the application's voice, re-sent on every later request. interval counts model requests from 1; LLMReminder fires only every interval requests (10 by default), so a short scripted run needs LLMReminder(..., interval=1).

Looks like it never fires. Context-window-driven capabilities (the compaction family) cannot resolve a window for TestModel, so thresholds default high; pass context_window= or fallback_context_window= in the test.

Token thresholds under a scripted model. FunctionModel and TestModel estimate usage by counting words (splitting on whitespace and ",.:), and the compaction family anchors its token count on the latest response's input_tokens. A payload with no separators, such as 'x' * 4000, counts as about one token once a response follows it, so max_tokens/max_fraction may never fire. Use real words, or set the count from the script with ModelResponse(..., usage=RequestUsage(input_tokens=...)) (from pydantic_ai.usage import RequestUsage); tokenizer= measures only the messages after that anchor. ToolOutputLimits measures characters (or the compaction text estimate, or tokenizer=, with over_tokens=True) and does not read model usage.

Summary requests reach the script. SummarizingCompaction without model= sends its summary request to the run's model, so the scripted function receives it too: that call has no function tools (info.function_tools == []) and info.instructions starts with 'You are a context summarization assistant' (the default instructions=). Return the summary text there, or pass model= a separate FunctionModel.

See also

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at b2d64a3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
compatibility
Requires Python 3.11+
metadata
{
  "version": "0.2.0",
  "author": "pydantic"
}

README badge

README badge for pydantic/pydantic-ai/pydantic-ai-harness