Agent Instructions: Continuous Inference Format Optimization
You are an autonomous coding subagent tasked with systematically optimizing and improving an A2UI inference format (e.g., atom, express, elemental). Your goal is to maximize LLM generation pass rates (accuracy) while minimizing latency and token footprints.
🤖 Multi-Agent Subagent Worktree Protocol
When running iterative optimization cycles with subagents:
- Dedicated Worktree per Pass: Each subagent MUST operate inside its own isolated Git worktree branch (
/usr/local/google/home/gspencer/code/a2ui/worktrees/opt-<format>-pass<N>). - Subagent Delegation: The parent agent delegates each pass to a subagent (
invoke_subagentwithTypeName="self"andRole="Express Pass N Optimizer"). - Sequential Execution: Passes execute sequentially. The parent agent waits for the subagent's completion notification before launching the next pass.
- Automatic Guardrails & Archiving:
- The subagent tests code changes with
pytest. - Runs
optimize_format.py --format <format>. - Evaluates decision rules (KEEP if $S_{opt}$ improves and Pytest passes; REVERT if accuracy drops or output tokens expand $> +5%$).
- Archives the run into
eval/iterative_format_optimizer/history/<format>/using--archive. - Runs
sync_history.pyto keep the master history summary up to date.
- The subagent tests code changes with
Workspace Setup
Before making edits, spin up an isolated git worktree branch to run your experiments without conflicting with parallel agent runs or main code:
# Branch name convention: opt-<format>-<hypothesis_slug>[-<agent_id>]
# Directory convention: ../worktrees/<branch_name>
git worktree add ../worktrees/opt-atom-brackets -b opt-atom-bracketsExecute all subsequent steps inside that worktree directory.
⚡ Quick-Command Cheatsheet
Use these unified CLI commands to avoid writing custom Python scripts or manual archiving steps:
| Task | Unified Command |
|---|---|
| Test S-Expr Compilation | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --compile "(Card (Text \"Hi\"))" |
| Test S-Expr AST Parsing | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --parse "(Card (Text \"Hi\"))" |
| Test Decompilation | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --decompile '{"version":"v1.0",...}' |
| Run Fast Subset Eval | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom |
| Run Targeted Prompt Eval | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --prompt loginForm |
| Compare vs Baseline | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/compare_results.py --baseline eval/iterative_format_optimizer/baselines/atom/unbounded_run_meta.json eval/iterative_format_optimizer/logs/temp_optimization/ |
| Archive Run (Atomic) | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --archive --hypothesis "..." --status KEEP |
| Full Milestone Check | uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format atom --full |
🚫 Anti-Patterns & Operational Guardrails
- DO NOT write ad-hoc
python -cscripts to importAtomCompileror test S-expressions. Use--compileor--parseinstead. - DO NOT run manual 5-step shell archiving dances (
mkdir,cp,git diff > patch.diff). Useoptimize_format.py --archiveinstead. - DO NOT retry backtracked hypotheses. Always check
eval/iterative_format_optimizer/history_summary.mdfirst.
The Optimization Loop
For each iteration, perform the following steps:
Step 1: Analyze History & Formulate Hypothesis
- Check Historical Index & Past Runs: Read
eval/iterative_format_optimizer/history_summary.mdand scan recentrun_meta.jsonfiles ineval/iterative_format_optimizer/history/to inspect all previous experiments, their hypotheses, status (KEEPvsREVERT), and code diffs. - Anti-Repetition Constraint: DO NOT repeat or retry a hypothesis, prompt rule edit, or code modification that was already tested and reverted in a past run.
- Analyze Failure Patterns: Review baseline failure logs in
eval/baselines/{format}/results.jsonto identify unresolved failure patterns (e.g. nested layout errors, dangling string references, missing optional attributes). - Formulate a Minimal Hypothesis: State a specific, targeted hypothesis (e.g. "Replacing positional parameter placeholders with explicit keyword syntax will eliminate container nesting ambiguity").
Step 2: Implement and Verify Code
- Implement changes in:
- System prompts and format rules:
prompt_generator.py(or prompt template files) in the format's package. - Parser, compiler, or decompiler code: under the format's package.
- The format's package is
python/a2ui_agent/src/a2ui/inference_formats/express/for Express, andpython/a2ui_agent/src/a2ui/inference_formats/experimental/{format}/for an experimental format.
- System prompts and format rules:
- Catalog Agnosticism Constraint (Strict Non-Negotiable):
- All compilers, decompilers, prompt generators, and system prompt instruction templates MUST remain 100% catalog-agnostic.
- NO Hardcoded Component or Property Names in Code: Compilers and decompilers MUST NOT hardcode specific component names (e.g.
Card,Column,Row,List,Button) or catalog-specific property names (e.g.children,child,trigger,content,template,items) in parsing/compilation decision trees. - NO Catalog-Specific Prompt Rules or Examples: System prompt instruction blocks (e.g.
ATOM_RULES,EXPRESS_RULES,ELEMENTAL_RULES) and generated examples MUST NOT assume or hardcode rules/examples specific to individual basic catalog components or properties (e.g. referencingImage,Text,:url,:src,variant,"caption","body"). Instructions must remain format-grammar-centric using generic syntax placeholders ((ComponentName :key val child1 ...)). - Dynamic Schema Inspection: Compilers/decompilers and prompt generators MUST inspect catalog schema
$reftypes (e.g.common_types.json#/$defs/ChildList,common_types.json#/$defs/Child,common_types.json#/$defs/Action) and generate signatures dynamically viaCatalogSchemaHelper. - Synthetic Catalog Verification: All compiler/decompiler/prompt changes MUST pass the fuzzed synthetic catalog unit test (
test_fuzzed_synthetic_catalog_agnosticism) to prove they function on custom or fuzzed catalogs.
- Run pytest conformance unit tests locally first:
uv run pytest python/a2ui_agent/tests/ - If unit tests fail, classify the failure:
- Prompt-Only Regression: If only prompt/template files were modified and unit tests fail, it is a prompt regression. You MUST immediately revert the prompt modification.
- Parser/Compiler Regression: A change broke existing valid compiler behavior. You must fix the compiler code or revert the change.
- Format Capability Evolution: The compiler was deliberately modified to support new/updated syntax, causing old tests to fail. You must update the unit tests under
python/a2ui_agent/tests/{format}/to match and cover the new capability.
Step 3: Run the Evaluation
Run the orchestrator script to compile the metrics and generate the current diagnostic report. By default, this runs on a small-scale validation subset of 5 diverse prompts to minimize cost and execution time:
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format <format> --model <model>Note: For targeted debugging of a specific failure, you can run:
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format <format> --model <model> --prompt <prompt_name>Step 4: Evaluate Metrics & Compare Against Baseline
Run compare_results.py to compare your current run directory against the baseline directory (eval/iterative_format_optimizer/baselines/<format>). By default, it computes median metrics:
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/compare_results.py --baseline eval/iterative_format_optimizer/baselines/<format>/unbounded_run_meta.json <current_run_dir_or_eval_log>(To inspect mean averages instead of default medians, pass --average).
Analyze the generated Markdown table and Metric Definitions Key:
- Pytest Conformance: Must be
PASS. - Correctness Guardrails:
Schema Acc (Delta): Percentage of outputs passing strict compiler syntax and schema validation (a2ui_scorer).Quality Score (Delta): LLM-graded semantic intent accuracy score (measured_model_graded_qa).
- Efficiency Optimization Metrics:
Median Code Output Tok (Delta): Token count of generated code. Direct target of format design optimization.Non-reasoning Output Time (Median): Estimated time spent emitting code. Shrinks proportionally with code output tokens.Median Input Tok (Delta): System prompt and catalog schema token size. Lower input tokens prevent API throttling.Parallel Wall Latency (Delta): Batch wall-clock throughput under 10 concurrent tasks.
Step 5: Progression or Rollback Decision
Evaluate your iteration against these 4 decision rules:
Rule 1 (Correctness Guardrail - Non-negotiable):
Schema AccandQuality ScoreMUST NOT regress below Baseline.- If Accuracy Degrades $\rightarrow$ You MUST immediately roll back the changes using
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format <format> --revert.
- If Accuracy Degrades $\rightarrow$ You MUST immediately roll back the changes using
Rule 2 (Efficiency Regression Caps - Non-negotiable): Even if accuracy remains equal or 100%, you MUST REVERT if:
Median Code Output Tokincreases by > 5% vs baseline/previous run.Non-reasoning Output Time(streaming latency) increases by > 10% vs baseline/previous run.Median Reasoning Tokincreases by > 15% vs baseline/previous run (prevents prompt instruction search space ambiguity).
Rule 3 (Composite Optimization Score $S_{\text{opt}}$):
- Check
Score (S_opt)in the comparison table: $$S_{\text{opt}} = 0.50 \times \text{SchemaAcc} + 0.30 \times \text{QualityScore} - 0.15 \times \left(\frac{\text{CodeTok}}{\text{BaseCodeTok}}\right) - 0.05 \times \left(\frac{\text{ReasonTok}}{\text{BaseReasonTok}}\right) - 0.03 \times \left(\frac{\text{InputTok}}{\text{BaseInputTok}}\right)$$ - If $S{\text{opt}}(\text{Current}) > S*{\text{opt}}(\text{Baseline})$* $\rightarrow$ KEEP CHANGE.
- If $S{\text{opt}}(\text{Current}) \le S*{\text{opt}}(\text{Baseline})$* $\rightarrow$ REVERT CHANGE (
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format <format> --revert).
- Check
Step 6: Archive Iteration Run
To save the historical run context for future analysis and allow the orchestrator to automatically maintain the summary index:
- Determine the next run index (e.g.
run_003). - Create the run folder:
# Directory naming: run_<three_digit_index>_<commit_sha>_<slugified_summary> mkdir -p eval/iterative_format_optimizer/history/<format>/run_003_a8f9c1b_fix_brackets - Archive logs and files:
- Copy report:
cp eval/iterative_format_optimizer/current_report.md eval/iterative_format_optimizer/history/<format>/run_003_a8f9c1b_fix_brackets/report.md - Copy results log:
cp eval/iterative_format_optimizer/logs/temp_optimization/results.json eval/iterative_format_optimizer/history/<format>/run_003_a8f9c1b_fix_brackets/results.json - Generate diff patch:
git diff > eval/iterative_format_optimizer/history/<format>/run_003_a8f9c1b_fix_brackets/patch.diff
- Copy report:
- Write
run_meta.jsonfolder metadata detailing your hypothesis and annotations:{ "hypothesis": "Description of proposed fix", "notes": "Qualitative summary of results / failures", "status": "Kept" (or "Backtracked") } - Commit the code and the archived folder local to your branch.
Verification, History Synchronization & Two-Tiered Merging
- Tier 1: Subagent Fast Inner Loop (Validation Subset): Subagents execute rapid hypothesis iterations using the fast validation subset. When a run meets decision rules (
KEEP), the subagent archives the run and commits its branch as a Milestone Candidate. - Synchronizing Multi-Worktree History: Collect history across parallel agent worktrees into main history:
uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/sync_history.py - Tier 2: Outer-Loop Milestone Full Suite Check: Before merging a Milestone Candidate branch into
main, execute full suite verification across all evaluation samples:uv run python eval/iterative_format_optimizer/skills/inference-format-optimizer/scripts/optimize_format.py --format <format> --model <model> --full - PR Creation & Merging: Create a Pull Request to merge your worktree branch back to the main repository. When branches are merged, the history run folders merge conflict-free. Future agents starting from
mainwill automatically inherit the complete collective history of all previous runs.
Termination Triggers
Stop your execution loop and report results to the human operator when:
- Target Achieved: 100% pass rate is achieved on the full suite.
- Plateau: No accuracy improvements are observed after 3 consecutive iterations with different hypotheses.
- Max Iterations: You have completed 10 runs.
- Stuck / Blocked: If you encounter an infrastructure error or a requirement conflict that prompt/compiler tuning cannot fix, halt and ask for help.