Benchmark Patterns

Use these patterns as evidence-backed starting points. Read the relevant domain section before designing a Task. Open the cited benchmark when its design can answer a concrete question about Task shape, Environment state, tool surfaces, simulation, or verification. Adapt the method to the current project. Do not copy a benchmark's private data, hidden answer, exact scenario, or limitations without review.

Benchmark design index
Automation Environments
Browser and Desktop Environments
Coding Environments and Tasks
CRM and Knowledge-Work Environments
Multi-Turn and Policy Environments
Terminal Environments and Tasks
Cross-domain quality checks

Benchmark design index

Use this table to decide which external design to inspect in more detail.

Benchmark	Domain	Design worth studying
AutomationBench	Business automation	Trigger data, pre-populated SaaS state, realistic tool surfaces, assertion-level scoring, and a separate strict completion result
AppWorld	Cross-application APIs	Stateful simulated applications, database-backed truth, cross-app workflows, and final-state verification
ToolSandbox	Stateful tool use	Controlled hidden state, milestones, tool-call consequences, and reproducible interaction state
TheAgentCompany	Workplace agents	Longer work across applications, documents, messages, and durable artifacts
WebArena	Browser agents	Reproducible websites, browser interaction, seeded application state, and state-based evaluation
WorkArena	Enterprise browser work	Role-aware enterprise tasks, realistic web workflows, and application state
WorkArena++	Multi-application enterprise work	Longer workflows across enterprise applications and role-sensitive state
OSWorld	Desktop agents	OS and application snapshots, computer input, cross-application work, and state restoration
SWE-bench	Coding agents	Real issues, pinned repository revisions, behavior tests, and patch verification
SWE-smith	Scalable coding Tasks	Reproducible mutation and execution pipelines; mutations still need usefulness review
CRMArena-Pro	CRM agents	CRM roles, linked business records, permissions, and realistic sales workflows
tau-bench	Policy and tool use	User simulation, policy-guided decisions, tool use, and database final state
tau-squared Bench	Longer multi-turn work	Both sides of a tool-mediated dialogue, longer interactions, and final-state checks
Terminal-Bench	Terminal agents	Isolated command-line Environments, natural instructions, tests, and reference solutions

A benchmark citation is not an authority claim. Check its current repository, paper, release, license, and task format before copying an implementation pattern. Record which design choice was adapted and why it fits this project.

Automation Environments

Use this pattern

Use for scheduled jobs, event handlers, background agents, retries, alerts, integrations, and work where time or repeated delivery changes correctness.

State and storage

Store current state and append-only events separately. Include:

Trigger ID and type.
Scheduled, occurred, observed, and completed times.
Actor or source.
Idempotency key, which prevents repeat delivery from causing a repeat effect.
Attempt count and error class.
Current workflow state.
Side effects and their external IDs.
Cause or parent event.

Use a fixed clock and deterministic event queue. Use an explicit state machine, which lists allowed states and changes. Store enough history to prove retry, deduplication, and recovery behavior.

For each case, define trigger data, pre-populated starting state, the allowed tool surface, and final-state assertions. This is the useful AutomationBench case shape. The trigger starts the workflow; it does not replace the World or the Task success contract.

Services, tools, and access

Simulate mutable queues, schedulers, webhooks, email, and third-party writes unless an approved sandbox is safe and resettable. Preserve payload schema, delivery order, duplicate delivery, error classes, retry limits, and side-effect IDs.

Catalog scheduler, queue, service, cloud, and model credentials by name and scope. Keep the fixed clock and hidden future events unavailable to the agent.

Setup and reset

Load base state and pending events.
Set the clock and visibility boundary.
Start the worker and dependent services.
Check queue and service readiness.
Save initial state, events, and external-effect ledgers.
Reset from the source snapshot, including retry counters and deduplication state.

Test reset after partial effects and timeouts.

Strong Task families

Process a scheduled action exactly once.
Recover from a transient error without duplicate effects.
Stop after a permanent policy or permission error.
Reconcile a delayed observation with newer state.
Handle concurrent triggers in a valid order.

Worked Task A: retry a transient delivery

Instruction:

Process the pending notification. The first delivery attempt has a transient error. Complete the delivery without creating a duplicate message.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
final_state	Workflow is complete	Final workflow row	Compare state with `completed`	Exact match
retry	One allowed retry occurred	Attempt and error rows	Match one transient failure then one success	Exact sequence
one_effect	One message exists	External-effect ledger	Count unique message IDs for the source event	One
repeat_safety	Both attempts share one repeat-protection key	Attempt records	Compare keys and source event ID	All equal
scope	Other work stays fixed	Before and after snapshots	Compare unrelated workflows and schedules	Equal

Known-good: one transient failure is followed by one successful attempt with the same repeat-protection key and one message. Wrong results: abandon, create two messages, change the key, exceed the retry limit, or change another action.

Worked Task B: handle duplicate triggers

Instruction:

Process the two visible trigger deliveries for the same source event. Apply the business change once and retain audit evidence for both deliveries.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
one_change	Business state changes once	State history	Count changes for the source event	One
delivery_log	Both deliveries are retained	Delivery records	Compare delivery ID set	Exact two IDs
one_effect	Both deliveries resolve to one effect	Delivery and effect records	Join by source event and effect ID	One shared effect
no_loss	Other visible triggers remain	Queue before and after	Compare unrelated trigger IDs and states	Equal

Known-good: both deliveries are logged and one business effect is committed. Wrong results: apply twice, drop the second delivery record, create two effect IDs, or consume an unrelated trigger.

Calibrate

Use a fixed clock. Run one delivery, duplicate delivery, transient retry, permanent failure, reordered events, partial effect, and timeout. Reset queue, attempt, repeat-protection, and effect state each time. Inspect the full event order and final state. Treat worker startup or clock-control failure as infrastructure.

Report assertion-level partial credit when useful for diagnosis or training. Keep a separate strict completion result that passes only when every required assertion passes.

Difficulty changes

Change one condition: longer delay, retryable then permanent error, concurrent worker, stale read, reordered event, or partial side effect. Use controlled time and known event order so failures remain reproducible.

Sources

AutomationBench: realistic business workflows over 47 simulated SaaS tools. Borrow its trigger data, pre-populated state, tool surface, assertion-based final-state checks, assertion-level partial credit, and separate strict completion result.
AppWorld: stateful APIs and database verification across applications.
ToolSandbox: stateful tool calls, milestones, and controlled interaction state.
TheAgentCompany: longer-running workplace workflows and artifacts.

Browser and Desktop Environments

Use this pattern

Use when the agent operates websites, browsers, desktop applications, files, and visual interfaces.

State and storage

Pin:

Website or application version.
Browser, extensions, profile, and viewport.
Operating system image and display settings.
Login state and assigned identity.
Application databases, files, downloads, and clipboard state.
Time, locale, timezone, and accessibility settings.

Use local sites or snapshots when live content changes results. Preserve realistic DOM, visual, pagination, latency, validation, and permission behavior. Use accessibility state only when the real Harness can use it.

Services, tools, and access

Choose live services only when accounts, data, reset, cost, and side effects are controlled. Prefer a local clone, frozen site, or sandbox for mutable workflows.

Match the real Harness input and observation surface:

Screenshots and cadence.
Mouse, keyboard, scroll, and window controls.
DOM or accessibility data when available in production.
Browser downloads and uploads.
Application launch and focus behavior.

Catalog model, website, SSO, OAuth, VPN, application, and cloud credentials by name, role, scope, provider, and network need. Never save cookies or secret values in specs or committed artifacts.

Setup and reset

Restore a pinned browser, VM, or application snapshot.
Establish approved login state through the setup boundary.
Place required files and clear unrelated downloads and clipboard data.
Set display, locale, clock, and window state.
Check readiness through the same surface the agent uses.
Save raw application state for the Verifier.
Reset the full profile or snapshot after every run.

Do not rely on clicking back or reversing UI actions as reset.

Strong Task families

Complete a multi-page business workflow.
Find and update the correct record among similar records.
Transfer data between a document and application.
Respect a permission or confirmation boundary.
Recover from validation or stale-page errors.

Worked Task A: update a user without changing access

Instruction:

Update the selected employee's department and phone number. Keep their role, group membership, and account status unchanged.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
identity	Correct employee was edited	Final application row	Match stable employee ID	Exact ID
fields	Department and phone match	Final application row	Compare normalized requested values	Both equal
access	Role, groups, and status stay fixed	Before and after access rows	Exact comparison	Equal
persistence	Saved values survive a new session	Reloaded row from clean session	Read again after restart	Both equal
scope	Other state stays fixed	Application state difference	Compare non-allowlisted rows and fields	No difference

Known-good: only the stable employee ID and requested fields change and remain after a new session. Wrong results: update a similar name, change access, leave an unsaved form, or edit another record while searching.

Worked Task B: file-to-application workflow

Instruction:

Read the approved local request file, create the matching application record, and save the confirmation PDF in /home/oai/share/confirmations.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
source	Created values match the approved request	Parsed source file and final row	Compare named fields	All equal
record	One record was created	Before and after application rows	Count new IDs and match type	One
artifact	One valid confirmation PDF exists	Required path and parsed PDF	Check path, file count, text, and record ID	One valid file
scope	No extra effect occurred	State, upload, message, and file differences	Compare with allowlist	No extra change

Use application state and file parsing for verification. Use visual comparison only for layout that cannot be checked structurally.

Known-good: one matching application row and one readable confirmation PDF. Wrong results: wrong source file, unsaved form, duplicate record, screenshot instead of PDF, wrong output path, or an extra upload or message.

Calibrate

Restore the snapshot before each reference, wrong-result, and model run. Test a similar identity, unsaved form, changed access, stale page, duplicate submit, and extra file. Verify raw application and file state after a fresh session. Treat login, display, or application startup failure as infrastructure.

Difficulty changes

Change one condition: similar records, more pages, modal or validation error, permission boundary, stale page, cross-application transfer, or required file artifact. Do not create difficulty through tiny targets, hidden UI, or unstable live content.

Sources

WebArena: reproducible websites, browser interaction, and state-based evaluation.
WorkArena: enterprise web tasks, roles, and workflow state.
OSWorld: operating-system and desktop application snapshots, computer input, and cross-application tasks.

Coding Environments and Tasks

Use this pattern

Use for bug repair, feature work, refactoring, tests, build systems, and codebase navigation.

State and storage

Pin repository URL, commit, submodules, large-file inputs, and toolchain.
Use a fresh worktree or copy per run.
Keep package caches outside the scored worktree or reset them explicitly.
Record repository instructions, generated-file sources, public exports, and build outputs.
Preserve enough repository history when the agent needs blame, prior changes, or release context.

Model relationships beyond files:

Public API to implementation.
Registration to runtime selection.
Schema to generated artifacts.
Source package to consumer import path.
Bug report to failing behavior and focused tests.

Services, tools, and access

Prefer a frozen repository and installed offline dependencies. Allow live package or web access only when the real job requires it and the run can remain safe and comparable.

Preserve the production Harness shell, editor, patch, search, test, and build interfaces. Record private registry credentials, source-host access, and model credentials by variable name only. Keep hidden tests and reference patches unavailable to the agent.

Setup and reset

Create a fresh worktree at the pinned commit.
Apply one controlled condition, such as an existing bug or generated mutation.
Confirm the public failure before the agent starts.
Capture Git status and relevant generated-file hashes.
Reset by discarding the whole task worktree, not by reversing guessed files.

Use a container or VM when system packages, services, or compiler state affect the result.

Strong Task families

Repair a real issue with a reproducible failing test.
Add a feature through the public API and consumer path.
Change lifecycle or state behavior across modules.
Update a schema and all required generated or migration surfaces.
Diagnose and fix a build, packaging, or release fault.

Do not score a preferred patch shape when tests and public behavior define correctness.

Worked Task A: repair duplicate pagination

Instruction:

Fix the duplicate-item pagination bug in /workspace. Keep the public API compatible. Add or update focused tests for duplicate-free traversal.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
behavior	Pagination has no duplicate items	Focused test output	Run the pinned reproduction and boundary tests	All pass
regression	Related behavior still works	Existing focused suite	Run the named test files	All pass
compatibility	Public call shape remains available	Consumer import and call test	Import and call the old public function	Test passes
scope	No unrelated change	Git diff and generated-file hashes	Compare with an allowlist	No extra path or hash change

Known-good: a patch that fixes cursor advancement and adds a focused boundary test. Wrong results: filtering duplicate output after retrieval, changing the public signature, disabling the failing test, or editing unrelated snapshots.

Worked Task B: add a public extension

Instruction:

Add the requested provider through the public package import path. Register it in runtime selection and add a consumer-facing test.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
implementation	Provider performs the requested behavior	Focused provider test	Run the named behavior case	Passes
registration	Runtime selects the provider	Factory test and registry state	Load valid config through the public factory	Correct class returned
public surface	Consumer import works	Clean consumer process	Import from the documented package path	Import succeeds
tests	Related code remains valid	Pinned internal and consumer suites	Run the named tests	All pass
scope	Other providers and defaults stay fixed	Git diff and config snapshot	Compare named files and defaults	No extra change

Known-good: implementation, registry entry, public export, and consumer test from the source files that own each surface. Wrong results: implementation without registration, internal export only, unit test only, or hand-edited generated output without its source update.

Calibrate

Run the reference patch, no-op patch, disabled-test shortcut, and unrelated edit. Run agent trials when they answer a named calibration question. Inspect the selected diffs and each named test. Treat toolchain, dependency, and build-start failures as infrastructure errors, not coding failures.

Difficulty changes

Change one condition: cross-package surface, longer call path, missing regression test, lifecycle interaction, migration, partial failure, or more plausible distractor code. Do not remove needed issue context or rely on a huge unrelated test suite to create difficulty.

Sources

SWE-bench: real repository issues, pinned commits, and test-based patch verification.
SWE-smith: programmatic mutation and scalable task generation. Borrow reproducible generation and execution; do not assume every mutation is useful work.
Terminal-Bench: isolated command-line Environments and reference solutions for tasks that cross system setup.

CRM and Knowledge-Work Environments

Use this pattern

Use for sales, support, operations, research, documents, messaging, meetings, and work that crosses business applications.

State and storage

Use a relational store for identities and mutable business records. Use files or document tables for email, notes, reports, transcripts, and attachments. Add search only when retrieval behavior is part of the Task.

Model:

People, organizations, accounts, and aliases across services.
Ownership, teams, roles, and permissions.
Opportunities, cases, tasks, and status history.
Messages, meetings, documents, and source records.
Product usage, contracts, and time-dependent facts.
Draft, sent, approved, and deleted states.
Conflicting, stale, and missing observations.

Create structural facts with code. Create selected prose through source-supported generation requests and criterion-level review.

Services, tools, and access

Choose per service:

Frozen for read-only corpora that must remain comparable.
Simulated for mutable CRM, messaging, calendar, or document state.
Live sandbox only when reset, privacy, cost, and side effects are controlled.
Disabled for irrelevant access that adds leakage or risk.

Match tool names, arguments, results, pagination, permission errors, and effects. Catalog CRM, email, calendar, document, model, and search credentials by name, account, role, scope, network need, and provider. Do not store values.

Setup and reset

Select a relationship-complete record set, not one isolated row.
Preserve identities, history, documents, and cross-service IDs.
Apply one named setup operation through a service or checked builder.
Give the agent only its assigned identity and production-shaped tools.
Reset from a clean database or snapshot after every run.
Save initial and final state plus message and document effects.

Strong Task families

Reconcile records across systems and update the source of truth.
Draft a message or report supported by several services.
Act on a request while respecting permissions and policy.
Detect stale or conflicting state and clarify before action.
Complete a multi-step workflow without duplicate or collateral effects.

Worked Task A: update an account after a meeting

Instruction:

Review the latest meeting and current account state. Update the account's next step and create an internal follow-up note. Do not send an external message.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
account	Correct account has the supported next step	Before and after CRM rows plus meeting	Match account ID and allowed values	Exact match
note	One supported internal note exists	New note and meeting facts	Check count and cited facts; judge only unsupported prose	One valid note
identity	Assigned user made permitted changes	Audit log, owner, and role	Match actor, owner, and permission rule	All match
no_message	No external draft or send exists	Message and draft tables	Compare before and after	Equal
scope	Other business records stay fixed	Stable state snapshot	Compare every non-allowlisted row and field	No difference

Known-good: supported next step plus one source-supported note. Wrong results: use an older meeting, update a similar account name, claim a decision not made, send an email, or change a related opportunity that was not requested.

Worked Task B: produce a portfolio brief

Instruction:

Create a brief for the assigned accounts. Rank urgent items, cite the source record for each claim, and state conflicts instead of resolving them without evidence.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
coverage	Each qualifying assigned account appears once	Assignment and issue rows plus brief	Recompute expected account IDs and counts	Exact set and count
ranking	Order follows the stated urgency rule	Source values and brief order	Recompute score and compare order	Exact order, allowing declared ties
sources	Each material claim names a valid record	Brief citations and source rows	Resolve each ID and compare its claimed fields	Every claim supported
uncertainty	Conflicts stay explicit	Conflicting rows and brief text	Check required conflict IDs; judge only whether wording preserves uncertainty	All conflicts stated; none invented

Known-good: every qualifying account appears once in recomputed order with valid citations and stated conflicts. Wrong results: omit an account, include an unassigned account, use stale state, cite a missing record, or hide a conflict. Use a limited judge only for claim support and useful wording.

Calibrate

Run the reference actions, wrong identity, stale-source, duplicate-effect, unsupported-claim, and collateral-change cases before model trials. Reset all services between runs. Inspect final state, audit rows, citations, and each meaning-based result. Do not treat service startup or login failure as Task failure.

Difficulty changes

Change one condition: similar names, longer history, cross-service identity, stale record, permission boundary, several valid actions, or collateral-effect risk. Do not make the Task hard by hiding a required source.

Sources

AppWorld: stateful application APIs, cross-application tasks, and database-based verification.
CRMArena-Pro: CRM roles, records, and business workflows. Use domain structure, not private data.
WorkArena++: enterprise application workflows and role-sensitive state.
tau-bench and tau-squared Bench: policy, user interaction, tool use, and final-state checks.
TheAgentCompany: longer workplace tasks across tools and artifacts.

Multi-Turn and Policy Environments

Use this pattern

Use when success depends on conversation, hidden user facts, clarification, consent, policy, escalation, or changes during a dialogue.

State and storage

Keep separate:

Agent-visible conversation.
User simulator facts and goals.
Policy rules.
Mutable service state.
Tool calls and effects.
Turn and time limits.

The simulator may know facts the agent must ask for. It must not reveal facts without the defined trigger or judge correctness itself. Store final service state independently from dialogue.

Services, tools, and access

Preserve production tool schemas, permission errors, and state changes. Use a simulated user for repeatability and a simulated or sandbox service for mutable effects.

Catalog model credentials for the agent, user simulator, and any judge separately. Pin each model. Keep simulator facts, policies, and expected results hidden from the agent.

User simulator contract

Define:

Goal and known facts.
Facts supplied initially.
Facts supplied only after a clear question.
Behavior on vague, repeated, or leading questions.
Consent and refusal behavior.
When the user corrects a mistake.
Stop condition and turn limit.

Do not let the simulator help the agent simply because progress stalled.

Setup and reset

Create fresh service state and conversation state.
Set the assigned identity, policy version, clock, and simulator facts.
Check tools before the first turn.
Save every turn, tool call, and state change.
Reset both conversation and service state after every run.

Strong Task families

Gather required facts before an action.
Apply policy while helping the user reach a valid outcome.
Clarify conflicting identity or intent.
Refuse or escalate a prohibited request.
Recover after the user changes a material fact.

Worked Task A: policy-bound refund

User start:

I need a refund for my last order.

Hidden user facts include order identity, reason, and whether the item was used. Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
identity	Customer and order are confirmed	Dialogue and service IDs	Find explicit confirmation before action	Correct IDs confirmed
facts	Required facts are obtained first	Ordered turns and policy fields	Check each required answer precedes the action	Complete and ordered
policy	Action follows the pinned rule	Policy and confirmed facts	Recompute allowed action	Exact match
state	Service records match that action	Final order, refund, and escalation rows	Compare expected fields and counts	Exact match
scope	No extra effect occurred	Before and after service state	Compare non-allowlisted rows	Equal

Known-good: confirm the order and required facts, then create only the refund or escalation allowed by policy. Wrong results: act on the latest order without confirmation, infer item state, invent an exception, create both refund and credit, or refuse an allowed request.

Worked Task B: clarify an identity conflict

User start:

Move Alex's meeting to Friday afternoon.

Two visible contacts match Alex. Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
clarification	Agent asks before changing state	Ordered dialogue	Find a question that names a distinguishing fact before the tool call	Present and ordered
selection	Confirmed contact owns the changed event	User reply and final event	Resolve confirmed contact ID and compare	Exact match
time	New time follows request and calendar rules	Final event, clock, and policy	Recompute allowed Friday window	Within window
scope	Other events and attendees stay fixed	Calendar difference	Compare with focal-event allowlist	No extra change

Dialogue checks should establish that clarification happened. Final calendar state proves the action.

Known-good: ask one distinguishing question, receive the answer, then move only the confirmed contact's event. Wrong results: guess from name, ask after the move, change both events, choose a blocked time, or change attendees.

Calibrate

Run scripted good, ambiguous, changed-request, refusal, repeated-question, and premature-action conversations before model trials. Inspect ordered turns and final service state. Reset both after every run. Treat simulator, model, or tool startup failure as infrastructure, not policy failure.

Difficulty changes

Change one condition: more hidden facts, policy exception, conflicting user statement, changed request, permission boundary, or tool state change. Do not make the simulator evasive beyond the defined user behavior.

Sources

tau-bench: user simulation, policy-guided tool use, and database final state.
tau-squared Bench: longer interactions and both sides of a tool-mediated dialogue.
ToolSandbox: stateful tool use, milestones, and controlled hidden state.

Terminal Environments and Tasks

Use this pattern

Use for command-line diagnosis, repair, build, data processing, services, systems work, and tasks that require a container or VM.

State and storage

Pin:

Base image or VM snapshot.
Operating system and architecture.
Installed tools and versions.
Files, permissions, owners, and working directory.
Processes, ports, services, and clocks.
Offline packages, datasets, and repositories.
CPU, memory, storage, GPU, and network limits.

Keep tests and reference solutions outside the agent-visible image. Store starting-state and final-state evidence for files, processes, services, and command results.

Services, tools, and access

Use a single container when possible. Use Compose only for real service boundaries; name the primary Harbor service main. Add health checks for dependencies.

Disable network unless the real Task requires it. Record registry, cloud, SSH, database, and model credential names and scopes. Never bake secrets into images or pass them as build arguments.

Setup and reset

Build from a pinned image.
Install pinned runtime dependencies or include approved offline assets.
Place agent-visible files only.
Start services and check observable readiness.
Capture baseline files, permissions, processes, and service state.
Reset by replacing the container or VM.

Separate Environment build errors, service startup failures, timeouts, and Verifier errors from Task failure.

Strong Task families

Diagnose and repair a broken service.
Restore or transform data under constraints.
Configure a tool or deployment correctly.
Repair a build or dependency fault.
Investigate logs and produce a tested fix.

Worked Task A: repair a local service

Instruction:

The service in /app fails its health check after restart. Diagnose and fix it. Keep its public endpoint and stored records compatible.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
readiness	Service becomes healthy	Process, port, and health output	Start with the pinned command and poll to the limit	Healthy in time
behavior	Public endpoint is correct	Request and response	Run named requests and compare status and body	All match
persistence	Existing records stay readable	Baseline and final store	Read and compare every pinned record	Equal
restart	Fix survives restart	Fresh process and requests	Stop, start, wait, and rerun checks	All pass
scope	Other system state stays fixed	Port, process, file, owner, and permission differences	Compare with allowlist	No extra change

Known-good: the normal start command reaches health, serves correct stored data, and does the same after restart. Wrong results: bypass health, hardcode a response, delete stored data, leave a foreground process only, or open an extra port.

Worked Task B: recover a damaged dataset

Instruction:

Recover all valid records from /data/input, write stable, sorted JSONL to /data/output/recovered.jsonl, and report rejected record IDs.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
recovery	Every valid record appears once	Independent parser and output	Recompute valid IDs and compare output IDs	Exact set, no duplicates
rejection	Every invalid ID has a valid reason	Independent errors and report	Compare ID set and allowed reason per ID	Exact match
format	Output is stable JSONL	Parsed output and second run	Check schema, sort key, bytes, and repeat build	All match
scope	Inputs and other files stay fixed	Before and after file hashes	Compare outside the output allowlist	Equal

Known-good: recover the exact valid ID set in stable order and report every invalid ID once. Wrong results: drop or duplicate rows, repair unsupported values, write only to stdout, change an input, use unstable order, or give an unsupported rejection reason.

Calibrate

Run the reference solution, no-op, hardcoded response, data deletion, foreground-only service, partial recovery, and unrelated-file edit. Rebuild the container or VM for each run and test restart. Treat image build, service startup, and resource failure as infrastructure.

Difficulty changes

Change one condition: more services, partial failure, permission fault, restart, larger data, limited resources, or cross-file consistency. Do not create difficulty through missing tools, impossible timeouts, or hidden network needs.

Sources

Terminal-Bench: isolated terminal Environments, natural instructions, tests, and reference solutions. Pin the exact 2.x or 3.x release because formats and Verifiers can change.
SWE-bench: repository repair and test-based correctness when the terminal Task includes code changes.

Cross-domain quality checks

For every adapted pattern, ask:

Does the Task require useful production work and the selected capability?
Is all needed information visible or normally discoverable?
Does the Environment preserve the relationships that change the decision?
Can independent evidence prove required and prohibited results?
Can an agent pass through wording, record order, leakage, or a shortcut?
Does the Task add a meaningful condition rather than rewrite an old prompt?
Which benchmark design was adapted, and which limitation was rejected?
If model contrast was useful, what did the selected complete traces reveal?

Store proven project-specific adaptations in the project World Skill. Keep the exact request, focal state, expected answer, and scoring criteria in the collocated Task.md.

/eval-engineering