Research and Browsing
Web research and browser capabilities: Researcher (a ready-made research stack), ExaSearch /
ExaAgent (Exa API), YouSearch / YouResearch (You.com API), KeenableSearch (Keenable API, keyless),
BrowserUse (hand a goal to an autonomous browser agent), and PlaywrightBrowser (the model drives Chromium itself). All are
capabilities passed in Agent(capabilities=[...]).
Which one to use
| Need | Use |
|---|---|
| Search and read public pages; follow whatever the model's provider offers | Core WebSearch(local=True) + WebFetch(local=True) from pydantic_ai.capabilities |
| A research agent with cited answers, sub-question delegation, and bounded tool output, no API keys beyond the model | Researcher |
| Same search behaviour on every model, excerpts per hit, domain filters, one vendor | ExaSearch or YouSearch |
| Same search behaviour on every model with no API key and no extra to install | KeenableSearch |
| Synthesized, cited answer or multi-step research in one tool call | ExaSearch(include_deep_search=True), ExaAgent, or YouResearch |
| Fuzzy goal on unknown pages ("find the Pro plan price") | BrowserUse |
| Known, repeatable flows, logged-in pages, JS-rendered SPAs, deterministic actions | PlaywrightBrowser |
Core is enough when the model's provider has native search/fetch or DuckDuckGo plus markdownify
fetching is acceptable. WebSearch() and WebFetch() with no arguments are native-only and raise on
models without native support; pass local=True for the fallback (needs pydantic-ai-slim[duckduckgo]
and pydantic-ai-slim[web-fetch]).
Tool-name collisions: core WebSearch (its native tool on Anthropic models), ExaSearch, YouSearch, and
KeenableSearch all expose a tool named web_search; ExaSearch, YouSearch, and KeenableSearch all expose
get_page. Use one search capability
per agent, wrap extras in core PrefixTools(wrapped=..., prefix='cb') (a second instance of the same
class also needs its own id=, since each carries a default one), or use WebSearch(native=False, local=True)
(needs the pydantic-ai-slim[duckduckgo] extra) whose DuckDuckGo tool is duckduckgo_search. Researcher includes core WebSearch, so do not add
ExaSearch/YouSearch/KeenableSearch next to it on Anthropic models without PrefixTools.
To use the provider's native search where the model has one and Exa or You.com elsewhere, set
ExaSearch(native=True) / YouSearch(native=True) (their web_search is then only sent to models without
native search, or whose native search can't apply the include_domains/exclude_domains filters, like Gemini's;
get_page stays), or pass WebSearch(local=ExaSearch().web_search_tool()) to keep core
WebSearch's native options. KeenableSearch().web_search_tool() works as that keyless fallback too, with web_search only (no
get_page); KeenableSearch has no native flag.
Researcher
A combined capability: default research instructions, WebSearch(local=True), WebFetch(local=True),
SubAgents with one web researcher delegate, and ToolOutputLimits (spills oversized tool results
to the workspace).
uv add "pydantic-ai-harness[researcher]"from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace
from pydantic_ai_harness import Researcher
from pydantic_ai_harness.researcher import DEFAULT_RESEARCHER_INSTRUCTIONS
agent = Agent('test', capabilities=[LocalWorkspace('.'), Researcher()])
custom = Researcher(instructions=DEFAULT_RESEARCHER_INSTRUCTIONS + 'Answer in French.\n')Researcher(*, instructions=DEFAULT_RESEARCHER_INSTRUCTIONS, subagents=None, store=None):
instructions: replace the defaults with a string, orNonefor no instructions.subagents:None= the built-inresearcherdelegate;[]disables delegation; or your ownSubAgententries.store: overflow store forToolOutputLimits.Nonespills into the run's workspace under.pydantic-ai-harness/tool-output/.
Gotchas:
- A run without a workspace fails at start with
UserError. AttachLocalWorkspace('.')or a sandbox capability, or passstore=LocalFileOverflowStore()(from pydantic_ai_harness.tool_output_limits import LocalFileOverflowStore) to spill on the agent's host. Custom delegates need their own store. Researcherhas no search-backend parameter. For Exa or You.com, compose the parts yourself:Capability(instructions=...), the search capability,SubAgents(agents=[...], agent_folders=None),ToolOutputLimits(). The search capability goes on the delegate that searches (see the Exa example below).researcher_agent(pydantic_ai_harness.researcher) is a model-lessAgentfor CLIs:uvx --with 'pydantic-ai-harness[researcher]' clai -a pydantic_ai_harness.researcher:researcher_agent -m <model>.- For typed findings, give the agent an
output_type; for parallel fan-out, addDynamicWorkflow.
ExaSearch and ExaAgent
ExaSearch adds web_search (excerpts per hit), get_page (full text of one URL), and opt-in
deep_search (Exa's multi-query search returning a cited answer). ExaAgent adds exa_agent, which
runs an Exa Agent API research task as a deferred tool call.
uv add "pydantic-ai-harness[exa,anthropic]" # and set EXA_API_KEYfrom pydantic_ai import Agent
from pydantic_ai_harness import ExaAgent, ExaSearch
agent = Agent(
'anthropic:claude-sonnet-5',
capabilities=[ExaSearch(max_results=5, include_deep_search=True), ExaAgent(effort='low')],
)ExaSearch fields: max_results=5 (1 to 100), max_text_chars=10_000 (1 to 10,000; get_page cap,
head kept), text_summary=False (True or a format string prepends a Summary: line),
include_deep_search=False, include_domains=[] / exclude_domains=[] (mutually exclusive; they filter
web_search and deep_search, not get_page),
guidance=None ('' = no instructions), client=None (ExaClient; default wraps exa_py.AsyncExa from
EXA_API_KEY, rebuilt on a new asyncio event loop once the previous one has stopped).
ExaAgent fields: effort=None (an exa_py AgentEffort, e.g. 'low' or 'auto'), execution='inline'
(polls to completion inside the run) or 'external', output_schema=None (Pydantic model class, validated;
or dict schema, not validated), system_prompt=None, poll_interval=1000 ms and timeout_ms=3_600_000
(both positive; the timeout applies to each run), guidance=None, runs=None.
Gotchas:
- A missing
EXA_API_KEYraisesUserErrorwhen theAgentis built (the capability itself constructs fine). Passclient=AsyncExa(api_key=...)/runs=to configure explicitly or inject a fake. execution='external'requiresoutput_type=[str, DeferredToolRequests]; resolve withagent_run_result(run, output_schema=...)and the run id atrequests.metadata[call.tool_call_id][RUN_ID_METADATA_KEY](both frompydantic_ai_harness.exa). Skip calls without that key (other tools' deferred calls), and store aModelRetrythatagent_run_resultraises on a schema mismatch as the call's result.- Empty pages and answers, HTTP error responses such as rate limits, and network failures become
ModelRetry(up to the tool's retry limit); 401/402/403 and otherexa_pyerrors propagate. Creating an Exa run is not retried after a 5xx or a network failure past sending, since the run may exist. - When a non-durable agent run is cancelled during inline polling, or polling fails with an error the model can't fix, the Exa runs nothing will read are cancelled (unfinished ones, and ones left for the model to follow up on), when
runshascancel. - Page text, titles, and excerpts are third-party content;
get_pagereads any URL the model passes. AddPromptInjectionDefenderor tool approval when that matters. - Citations: each tool returns a
ToolReturnwhosemetadata['sources']is a list ofExaSource(url,title), read fromToolReturnPart.metadata(not sent to the model). - Agent spec:
custom_capability_types=[ExaSearch, ExaAgent];client/runsare not serializable. - Durable execution (Temporal, DBOS, Prefect) records each Exa request, so recovery does not repeat it.
The records are named after the default
id(exa_search,exa_agent); keep it stable once deployed. - Tests: pass
client=any object satisfying the exportedExaClientprotocol (pydantic_ai_harness.exa), which needs two async methods,search(query, *, contents, num_results, type, output_schema, include_domains, exclude_domains)andget_contents(urls, *, text), both returning anexa_pySearchResponse.
Exa-backed research orchestrator: ExaSearch sits on the researcher delegate, and the orchestrator
only delegates. A delegate Agent with no model runs on the parent run's model.
from pydantic_ai import Agent
from pydantic_ai.capabilities import LocalWorkspace
from pydantic_ai_harness import ExaSearch, SubAgents, ToolOutputLimits
from pydantic_ai_harness.sub_agents import SubAgent
researcher = Agent(
name='researcher',
description='Research one sub-question on the web; report findings with source links',
capabilities=[ExaSearch(max_results=5), ToolOutputLimits()],
)
orchestrator = Agent(
'anthropic:claude-sonnet-5',
instructions='Split the question, delegate each part, then write a cited answer.',
capabilities=[
LocalWorkspace('.'), # ToolOutputLimits spills oversized results here
SubAgents(agents=[SubAgent(researcher)], agent_folders=None),
],
)YouSearch and YouResearch
YouSearch adds web_search and get_page. YouResearch adds answer (one-call cited answer),
research (multi-step, minutes), and finance_research. All fields are keyword-only.
uv add "pydantic-ai-harness[youdotcom,anthropic]" # and set YDC_API_KEYfrom pydantic_ai import Agent
from pydantic_ai_harness import YouResearch, YouSearch
agent = Agent(
'anthropic:claude-sonnet-5',
capabilities=[YouSearch(freshness='month'), YouResearch(research_effort='deep')],
)YouSearch: max_results=10 (1 to 20), extraction_mode='highlights' or 'full_page',
max_text_chars=10_000, include_domains / exclude_domains / boost_domains, freshness=None
('day'|'week'|'month'|'year'|'YYYY-MM-DDtoYYYY-MM-DD'), country=None, guidance=None,
timeout_ms=60_000, client=None.
YouResearch: research_effort='standard' ('lite'|'standard'|'deep'|'exhaustive'),
finance_effort='deep' ('deep'|'exhaustive'), the same domain/freshness/country filters,
output_schema=None (JSON-schema dict), guidance=None, timeout_ms=600_000, client=None.
Gotchas:
YDC_API_KEY(or legacyYOU_API_KEY_AUTH) is checked when theAgentis built; missing raisesUserError. Passingclient=skips that check: for tests, pass a fake satisfying the exportedYouClientprotocol (the asyncyoudotcom.Youmethods the toolsets call, such assearch_async).include_domainscannot be combined withexclude_domainsorboost_domains;output_schemawithresearch_effort='lite'raisesValueError.finance_researchignores the domain/freshness/country filters.- A no-match
web_searchreturnsNo results found for ...(not an error); 401/402/403 stop the run. - Citations under
metadata['sources']asYouSource, like Exa. - Durable execution records each You.com request under the default
id(you_search,you_research), like Exa.
KeenableSearch
KeenableSearch adds web_search (a short excerpt per hit) and get_page (one URL as markdown).
It needs no extra and no API key: Keenable's public endpoints are keyless, and the client uses httpx2,
which the harness already depends on.
uv add pydantic-ai-harness # KEENABLE_API_KEY is optional and only lifts rate limitsfrom pydantic_ai import Agent
from pydantic_ai_harness import KeenableSearch
agent = Agent('anthropic:claude-sonnet-5', capabilities=[KeenableSearch(num_results=5)])Fields: num_results=5 (1 to 50, sent as Keenable's max_results), max_snippet_chars=500
(per-hit excerpt cap), max_page_chars=10_000 (get_page cap; the truncation marker counts
against it, and a budget too small to hold the marker ends the page with … instead),
guidance=None, client=None.
Gotchas:
- Keenable's snippets run to thousands of characters, so
max_snippet_charsis what keepsweb_searchsmall;get_pageis how the model opts into a full page. - Budgets must be positive integers, and
num_resultsat most 50; anything else raisesValueErrorwhen the capability is built. KEENABLE_API_KEYswitches to the keyed endpoints;KEENABLE_API_URLoverrides the base URL and must behttps://(plainhttponly on loopback), orUserErroris raised.- Rate limits, 5xx, a 404, a non-JSON body, a search body without
results, and an empty page becomeModelRetry; 401/402/403 stop the run.get_pagereads Keenable's indexed copy, and Keenable documents that an unindexed URL fails to fetch. For tests, passclient=a fake satisfying the exportedKeenableClientprotocol. HttpKeenableClient(http_client=...)sends every request through yourhttpx2.AsyncClient(connection reuse, proxy, TLS, retrying transport); you close it. Without it, each request uses a fresh client.- A non-empty
guidancereplaces the default instructions, untrusted-content rule included, likeExaSearchandYouSearch. - Citations under
metadata['sources']asKeenableSource, like Exa.
BrowserUse
One browse_web tool that hands a self-contained goal to a browser-use agent driving real Chromium,
returning its final text (or validated JSON with output_schema).
uv add "pydantic-ai-harness[browser-use,anthropic]" # Python 3.11+ onlyfrom pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai_harness import BrowserUse
from pydantic_ai_harness.browser_use import BrowserAgentSettings
class Product(BaseModel):
name: str
price_usd: float
agent = Agent(
'anthropic:claude-sonnet-5',
capabilities=[
BrowserUse(
llm='anthropic:claude-sonnet-5',
allowed_domains=['example.com'],
output_schema=Product,
agent_settings=BrowserAgentSettings(use_judge=False),
)
],
)Key fields: llm=None, allowed_domains=None, block_ip_addresses=True, headless=None,
max_steps=50, use_vision=True ('auto'/False), output_schema=None, sensitive_data=None,
extend_system_message=None (steers the sub-agent), guidance=None (steers the host),
session_scope='call' or 'agent', cdp_url=None, browser_profile=None, browser_agent=None
(factory, for tests or custom setup), browser_use_telemetry=False.
Gotchas:
llm=Noneruns the sub-agent on the host run's model (RunContext.model) and inherits its instrumentation settings, including content redaction. browser-use's hostedChatBrowserUsemodel is opt-in viallm=ChatBrowserUse(): a separate account billed viaBROWSER_USE_API_KEY, outside your observability. Passllmfor a cheaper sub-agent model.- Costs:
use_vision=Truesends a screenshot every step; a judge call runs per task unlessBrowserAgentSettings(use_judge=False). - Usage: browser calls on Pydantic AI models (host model, model
llm,*_llmsettings) count toward the host run's usage andUsageLimits; exhaustion raisesUsageLimitExceededfrombrowse_web. Native browser-use chat models (ChatAnthropic,ChatBrowserUse) are not counted. - Flat
sensitive_datavalues require explicit-hostnameallowed_domains(globs rejected) and are substituted only on allowlisted pages; prefer the domain-scoped form{'https://site': {'key': 'value'}}. Secret values, and their JSON-escaped andquote_plus-encoded forms, are replaced with their placeholder in thebrowse_webresult. Screenshots are not redacted. session_scope='agent'reuses one browser across calls (serialized); close withaclose()orasync with BrowserUse(...) as browser:. UnderTemporalDurability,Agent(...)raisesUserErrorwith either scope (its toolset has no stable id).- Default
'call'scope launches one Chromium per concurrent call. file://navigation is always prohibited. Without an allowlist, abrowser_profile.prohibited_domainsthat reaches 100 entries once the harness'sfile://rules (and, withblock_ip_addresses, its localhost rules) are added raisesValueError. The default factory removesread_fileandupload_file. Withsensitive_dataset, every factory'ssettings.toolsdropsevaluate; a factory that builds its ownToolsmust drop it too.- browser-use's per-run telemetry event (task, visited URLs, result) is off for the capability's
agents;
browser_use_telemetry=Truesends it. SetANONYMIZED_TELEMETRY=falseto also stop browser-use's process-wide PostHog client (which captures uncaught exceptions).
PlaywrightBrowser
The host model drives a stateful Chromium through 18 tools: navigate, snapshot (accessibility tree
with aria-ref handles), click, type_text, press_key, select_option, hover, wait_for,
screenshot, get_text, scroll, go_back, go_forward, execute_js, console_messages, tabs,
handle_next_dialog, network_requests.
uv add "pydantic-ai-harness[playwright]"
uv run playwright install chromiumfrom pydantic_ai import Agent
from pydantic_ai_harness.playwright import PlaywrightBrowser
agent = Agent(
'test',
capabilities=[PlaywrightBrowser(allowed_domains=['example.com'], headless=True)],
)Fields: headless=True, allowed_domains=None (exact host plus subdomains), policy=None (full
EgressPolicy), block_private_addresses=True, screenshot_on_navigate=False,
max_content_tokens=4000, action_timeout_ms=5000, navigation_timeout_ms=60000 (0 disables
either), chromium_sandbox=True, auto_install_chromium=False, storage_state=None (Playwright
cookies + localStorage object, for logged-in runs), cdp_url=None (attach to a running Chromium).
cdp_url also takes a sync or async function of the run context that returns the endpoint: the
first browser tool call of a run calls it and the result is reused for that run; a timeout is a
tool error and is retried, an exception ends the run, and the capability only disconnects, so the
app deletes what the function created (record it on ctx.deps). cdp_context='default' reuses
sign-ins and tabs instead of creating a 'new' context. It requires a CDP endpoint and excludes
storage_state. The agent can act as every signed-in account, so attach it to a profile dedicated
to the agent, never an everyday browser, and set allowed_domains for untrusted pages. Only
http(s), about:blank and error-page tabs are adopted: any other tab (the browser's start page,
extension pages, file:, data:, blob:) is left out whatever the policy, and so is a tab holding a
non-web frame other than an inline srcdoc/data:/blob: one, such as an extension's injected
chrome-extension:// UI (use a profile whose extensions do not inject frames); for a tracked tab that
moves to one or gets such a frame, the call that finds it returns an error naming the page by its scheme and host only, and
the agent stops using that tab, which stays open in the browser. A frame that appears during a call can still be
captured by that call's snapshot or screenshot_on_navigate image; the next call drops the tab. Loaded tabs and frames must pass the egress policy
(each document once, at most five seconds per call; a call that runs out asks the model to retry), and new service-worker
registrations are refused; downloads follow the browser's settings.
Playwright routes service-worker requests through the guards unless
PLAYWRIGHT_DISABLE_SERVICE_WORKER_NETWORK is set. Connections opened before attachment bypass
them, as does a new WebSocket opened through the native WebSocket constructor that script running
before attachment kept, and a page loaded before attachment can register a worker until it navigates.
EgressPolicy (from pydantic_ai_harness.playwright) fields: allowed_domains=None,
blocked_domains=None, block_private_addresses=True, include_subdomains=True,
allowlist_reach=frozenset({'navigation', 'data'}), resolved_kinds (all kinds). Request kinds are
'navigation', 'subframe', 'data' (fetch, XHR, WebSocket, beacon) and 'subresource'
(images, scripts, fonts). refuse(request) checks in order: private IP literal, host that did not
resolve, private resolved address, blocked_domains, then allowed_domains only for kinds in
allowlist_reach. A host the domain lists refuse is not looked up. Add 'subframe' or
'subresource' to allowlist_reach to bound those too, or subclass and override refuse (call
super().refuse(request) for the default verdict).
from pydantic_ai_harness.playwright import EgressPolicy, EgressRequest
policy = EgressPolicy(
allowed_domains=['example.com'],
allowlist_reach=frozenset({'navigation', 'data', 'subframe'}),
)
public = ('93.184.215.14',)
print(policy.refuse(EgressRequest('https://evil.test/', 'subframe', resolved_addresses=public)))
#> domain not in allowed_domainsGotchas:
- Without network, every hostname is refused as unresolved when
block_private_addresses=True(the browser resolves names first), except one the domain lists refuse, which gets the domain-list reason without a lookup. Test the allowlist by callingpolicy.refuse(...)withresolved_addresses, as above. policytogether withallowed_domainsorblock_private_addresses=FalseraisesUserError; put those on theEgressPolicyinstead. A wildcard entry like'*.example.com'raises.- Chromium starts lazily on the first browser tool call. At run end a local browser closes and a CDP
connection disconnects; runs using
cdp_context='default'on one browser share its pages. A missing binary returns an install hint as the tool result and emitsBrowserUnavailableWarning. - Durable execution (Temporal, DBOS, Prefect capabilities) is rejected with
UserErrorat agent construction. storage_stateis credential material and is not accepted by agent-spec loading.- No uploads or downloads. Tool errors come back as strings the model can act on.
- Use one browser capability per agent;
BrowserUseandPlaywrightBrowserdo not share sessions.
See also
- https://pydantic.dev/docs/ai/harness/researcher/
- https://pydantic.dev/docs/ai/harness/exa-search/
- https://pydantic.dev/docs/ai/harness/youdotcom/
- https://pydantic.dev/docs/ai/harness/keenable-search/
- https://pydantic.dev/docs/ai/harness/browser-use/
- https://pydantic.dev/docs/ai/harness/playwright/
- https://pydantic.dev/docs/ai/capabilities/web-search/
- https://pydantic.dev/docs/ai/capabilities/web-fetch/