/native

Agent Visual Loop — AI-Driven Native Screen Implementation, Debugging, and Capture

Reference for running an agent-in-the-loop workflow on native screens: the agent writes SwiftUI/Compose, builds, renders or captures the result, compares it against a target with a numeric oracle, and iterates under a bounded cap.

Scope: iOS (SwiftUI/UIKit) and Android (Compose/Views) screen implementation, visual debugging, and screenshot capture driven by a coding agent. Use when the task is "build/fix this screen and confirm it actually looks right", not when authoring a test suite (→ Voyager[ios] / Voyager / Radar) or designing the visual direction (→ Vision / Muse).

0. The Loop Contract

SPECIFY → IMPLEMENT → BUILD → OBSERVE → SCORE → ADJUST ─┐
   ▲                                                     │
   └──────────────── (bounded: ≤ 3 passes) ──────────────┘
                              │
                     escalate to human

The iteration cap and the "agent declares close enough" failure it defends against are both reported from practice, not theory — see §7 and The Swift Dev · twocentstudios.

1. Structure Before Pixels

Rule: read the accessibility tree first; fall back to screenshot + coordinates only when the tree is insufficient.

This is the operating principle of mobile-next/mobile-mcp, which drives automation from "the native accessibility tree (no vision model, no image tokens), falling back to screenshots + coordinates only when needed."

Why it matters beyond cost: a tree query is deterministic and fails loudly; a coordinate tap derived from a screenshot silently drifts the moment layout shifts. Reserve coordinates for surfaces that expose no usable tree.

Consequence for implementation work: accessibility identifiers are agent infrastructure, not just test infrastructure. A screen with a complete identifier taxonomy is drivable by an agent; one without it forces the agent into pixel guessing. Taxonomy and per-framework assignment rules → voyager/reference/ios-identifier-strategy.md.

2. iOS Tool Layer

Pick the lowest-ceremony option that closes the loop for the task at hand.

RenderPreview is the piece that changes the economics: it closes the visual feedback loop without a simulator boot or a developer switching to the canvas. Agents discover the target window via XcodeListWindows before operating. rudrank.com · Xcode 26.3 teardown

Full simctl / devicectl / xctrace command surface → reference/xcrun-cli.md.

3. Android Tool Layer

Sources: Agent Mode docs · Otter 3 Feature Drop release notes · Generate UI with image attachments

4. The Numeric Oracle

The agent must not be the judge of "matches the design". Give it a number.

# Absolute difference metric (lower is closer; 0 = identical)
magick compare -verbose -metric RMSE reference.png candidate.png null:

# Visual diff image for the human reviewer
magick compare reference.png candidate.png diff.png

# Dimension check — catches the whole class of "right pixels, wrong frame" errors
magick identify candidate.png

# Probe one pixel's RGB — resolves color-token disputes exactly
magick image.png -crop 1x1+200+300 txt:

Gate rules:

• Set the threshold before the loop starts, from the spec, not from whatever the first pass happened to score.
• Mask dynamic regions (clock, battery, signal, push banners, animated content) before comparing — otherwise the metric is noise and the threshold gets loosened to compensate.
• A pass that does not move the metric is not an improvement, regardless of how the diff "reads". Revert it.
• Report the metric per pass in the deliverable. A visual claim with no number attached is UNVERIFIED.

Technique attribution: twocentstudios.

5. Screenshot-Test Substrate

The loop is far cheaper when a repeatable render already exists. Reuse the project's snapshot layer instead of booting a simulator per pass.

iOS — pointfreeco/swift-snapshot-testing in an isolated test target:

xcodebuild test -only-testing:"ViewSnapshotTests/ViewVerificationTests" -quiet
# → ViewSnapshotTests/__Snapshots__/ViewVerificationTests/<name>.1.png

Android — three tools, different jobs:

Sources: Paparazzi vs Roborazzi vs Compose Preview comparison · ComposablePreviewScanner

6. Authoring for Agent-Readiness

Changes to the implementation that make the loop work. These are the highest-leverage edits and they belong in the feature code, not in a test helper.

• Named previews per state. @Preview("Loading"), @Preview("Empty"), @Preview("Error"), @Preview("Long content — ja"). The agent can only render states that have an entry point.
• Seeded debug state. A deterministic fixture behind a debug flag, so pass N and pass N+1 render the same data. Live/network data makes every diff meaningless.
• Complete accessibility identifiers. See §1 — this is what lets the agent verify by structure instead of guessing coordinates.
• Temporary tuning controls. A debug-only slider/stepper bound to the value under adjustment (padding, corner radius, opacity) lets a human converge in seconds and hand the agent a final number. Remove before merge.
• Dark mode + Dynamic Type previews for any screen with a contrast or truncation risk: xcrun simctl ui <UDID> appearance dark, … content_size extra-extra-extra-large.

7. Documented Failure Modes

Observed in practice; treat each as a standing guard, not a hypothetical.

8. Cross-References

• reference/xcrun-cli.md — full simctl / devicectl / xctrace / xcresulttool surface, status-bar override, recording recipes.
• reference/adb-cli.md — full adb surface, screencap / screenrecord / input / Perfetto.
• reference/patterns.md — where snapshot tooling sits in the overall per-platform test matrix.
• voyager/reference/ios-identifier-strategy.md — accessibility identifier taxonomy (the §1 prerequisite) and per-framework assignment rules.
• voyager/reference/ios-screenshot-strategies.md — XCUITest capture APIs, XCTAttachment lifetime, App Store screenshot pipelines.
• voyager/reference/ai-powered-e2e-testing.md — AI-assisted test authoring and self-healing, including the native-mobile tool matrix. This file is about implementation; that file is about the suite.
• pixel/reference/visual-verification.md — the equivalent numeric-verification discipline for web mockup reproduction.