/voyager

AI-Powered E2E Testing & Playwright MCP/Agents (2026-05)

Purpose: Use this file when Voyager must plan, generate, review, or heal tests with AI assistance.

Contents:

• MCP vs CLI vs Test Agents tradeoffs and token costs
• Planner / Generator / Healer roles
• Browser-agent landscape (Stagehand, Skyvern, Browser-Use, Computer Use, Chrome DevTools MCP)
• Cypress cy.prompt parity
• Auth blockers, review gate, adoption ladder, cost metrics

Stack snapshot (2026-05): Playwright 1.60 (Test Agents 1.56+, screencast / browser.bind 1.59+), @playwright/cli Skills mode preferred over MCP for coding agents (~4× cheaper tokens), Cypress 15.15 with cy.prompt beta GA in 15.13. Browser-agent space consolidated to five production-grade stacks: Playwright + Claude, Stagehand v3, Skyvern, Browser-Use, Anthropic Computer Use / OpenAI CUA. Chrome DevTools MCP is the dominant observation plane; Playwright MCP and playwright-cli remain the action plane.

AI Delivery Modes for Playwright

Rules of thumb:

• Default: @playwright/cli when the host is a coding agent. Microsoft's 2026 guidance (dev.microsoft.com — Complete Playwright End-to-End Story 2026) treats CLI as the primary path and MCP as the fallback for protocol-bound autonomous workflows.
• MCP when the agent runs outside the editor and needs the MCP standard's live streaming.
• Test Agents only when the suite already has stable auth, locators, fixtures, and a human review gate.
• Chrome DevTools MCP sits alongside either path for performance traces, console logs, source-mapped stack traces, and Lighthouse audits.

MCP Model

The MCP approach routes AI actions through browser tools backed by the accessibility tree instead of brittle DOM selectors.

// Brittle DOM-style interaction
await page.click('div.checkout-btn-v3');

// Stable semantic target
// Role: button, Name: "Checkout"

Why this matters:

• Better resilience to selector churn
• Lower need for vision-only tooling
• Closer alignment with accessibility-first locator strategy

MCP Server Options (2026-05)

Default routing in 2026: Playwright MCP for action, Chrome DevTools MCP for observation, Midscene for vision-only canvases. Reference: Steve Kinney — Driving vs Debugging the Browser.

Playwright Test Agents

Agent Roles

Typical Workflow

1. Plan: npx playwright test --plan "Test the login flow"
   -> generates specs/login.md

2. Generate: npx playwright test --generate
   -> generates tests/login.spec.ts

3. Run: npx playwright test
   -> execute and review the result

4. Heal: npx playwright test --heal
   -> repair and rerun when a failure is reviewable

5. Loop: --loop
   -> repeats plan -> generate -> run -> heal automatically

Primary Blocker: Authentication

Largest adoption blocker:

• AI agents cannot explore behind auth walls without prior setup.
• storageState or equivalent auth fixtures must already exist.
• Re-authenticating on every run can trigger rate limits or security alerts.

Required mitigations:

• Save auth state in auth.setup.ts.
• Centralize auth in globalSetup.
• Prefer dedicated test-only auth tokens or accounts.

Review Rules for AI-Generated Tests

Unresolved Risks

AI Test Review Checklist

## AI-Generated E2E Test Review

### Required checks
- [ ] Do assertions verify business behavior?
- [ ] Can the test run independently?
- [ ] Do selectors use `getByRole`, `getByLabel`, or `getByTestId`?
- [ ] Does the test avoid `waitForTimeout()`?
- [ ] Does the test name describe user behavior?

### Architecture checks
- [ ] Does the test follow the existing Page Object Model?
- [ ] Does it reuse existing helpers and fixtures?
- [ ] Is the locator strategy consistent with the suite?
- [ ] Is test data created through APIs or factories?

Team-Based AI Testing

Parallel AI roles work when responsibilities are explicit and routed to the right surface:

• Functional agent — happy-path verification through Playwright MCP / @playwright/cli (action plane).
• Accessibility agent — WCAG checks via axe-core + Intelligent Guided Tests. Acknowledge the ~57% automation ceiling (Deque); do not let an AI agent claim "a11y covered" from automation alone.
• Performance agent — Core Web Vitals + Lighthouse via Chrome DevTools MCP performance_start_trace / performance_stop_trace (observation plane).
• Visual regression agent — Applitools / App Percy / Chromatic. Pick by tier: pixel-diff (icons), perceptual (Storybook stories), visual AI (complex layouts).
• Browser-agent for cross-site or RPA-shaped journeys — Stagehand v3 / Skyvern / Browser-Use, with Browserbase or Anchor Browser as the cloud sandbox.
• Security probing is not a Voyager role — route to Sentinel / Probe / Vigil. Do not embed security payloads in functional tests.

Self-Healing Tool Comparison (2026-05)

Retired or de-emphasised in 2026 vs the 2025 list: Testim and Katalon Studio 11 are no longer load-bearing in this comparison — both still exist commercially but have been displaced by Stagehand / Skyvern / Browser-Use / cy.prompt for new projects. Voyager keeps them in scope only when the project already runs them.

LLM-Generated Test Accuracy (2026 industry benchmarks)

These figures vary significantly by app complexity, auth complexity, and selector strategy. Treat as directional, not absolute. WebVoyager / SeeAct benchmarks are now the de facto comparators.

Commercial platforms still relevant in 2026: Octomind, Momentic, Stagehand / Browserbase, Skyvern, Browser-Use, Currents, Bug0, Applitools, App Percy, testRigor, Mabl, MaestroGPT. Pick by workflow shape, not by feature list — see selection guide below.

Picking the right AI testing surface

• Existing Playwright suite + coding-agent host (Claude Code / Codex / Cursor) → @playwright/cli Skills mode, escalate to Test Agents (1.56+) for plan→generate→heal loops.
• Existing Cypress suite + non-engineer authors → cy.prompt (Cypress 15.13+).
• Pure browser-agent product (no test repo) reaching across many sites → Stagehand v3 (developer-first) or Browser-Use (open-source) on Browserbase for cloud sessions.
• RPA-shaped workflow (form-filling, document downloads, CAPTCHAs, credential vault) → Skyvern.
• Cross-app journeys that leave the browser → Anthropic Computer Use or OpenAI CUA.
• Canvas / non-DOM / cross-platform native targets → Midscene.js (or MaestroGPT for mobile only; see mobile-testing.md).
• Self-healing applied to an existing Playwright repo without rewriting → Octomind (source-level fix) or ZeroStep (drop-in helper).
• Observation / debugging / performance during AI runs → Chrome DevTools MCP alongside whichever action layer is in use.

Adoption Roadmap (2026)

Rule: do not jump to phase 5 before the suite already has stable auth, selectors, data strategy, screencast / video receipts, and a review gate. Track flake rate, token spend, and self-heal rewrite volume as gating metrics between phases.

Cost Metrics

Track:

• Token cost per generated or repaired test
• Monthly AI test spend
• Human review time
• Flaky rate and false-positive rate

Sources (2026 refresh):

• Playwright Release Notes (1.56–1.60)
• Currents: State of Playwright AI Ecosystem 2026
• Microsoft Developer Blog — Complete Playwright End-to-End Story 2026 (CLI > MCP token recommendation)
• Steve Kinney — Driving vs Debugging the Browser (Playwright MCP vs Chrome DevTools MCP)
• Chrome DevTools MCP — Chrome for Developers
• Browserbase — Stagehand v3 launch
• Skyvern — Browser Automation Platform
• Browser-Use — GitHub
• Anthropic — Computer Use Tool docs
• Cypress — cy.prompt general launch · Cypress AI Features
• Octomind — Playwright self-healing
• Midscene.js v1.0
• TestDino — Playwright 1.59 Release Notes
• BrowserStack: Playwright AI Test Generator

Native Mobile AI Testing (2025-2026)

The visual-AI / self-healing tooling that grew up around web E2E now ships native-mobile coverage. Use this section when the artifact is .ipa/.apk/.aab and AI assistance is in scope.

Tool selection

Heal at authoring time, not at run time. The durable version of self-healing is the agent picking a stable selector (accessibility identifier over text/coordinates) while writing the test — not a runtime engine silently substituting a locator on a suite that already shipped. Maestro's own framing of agentic testing is that "the agent fixes its own selector while it is writing the test, not while the test is running later" (https://maestro.dev/insights/self-healing-tests-fixing-flaky-ui-automation). Runtime healing remains useful as a signal — but a heal event is a finding to triage, not a pass; keep it surfaced in the report (see the review gate below).

When to choose what

• Visual regression on a real-device matrix → App Percy if you are already on BrowserStack; Applitools Eyes when you need richer match modes or design-side Figma comparison.
• Plain-English authoring on iOS + Android by non-engineers → testRigor; verify business assertions during review.
• Self-healing with low ceremony on existing native suites → Mabl.
• AI assistance inside an existing Maestro flow set → MaestroGPT / Maestro AI test analysis. Lowest authoring friction when Maestro is already adopted.

Review gate for native AI tests

Same as the web review checklist above, plus mobile-specific:

• Locator strategy is accessibilityIdentifier / testID / contentDescription — not text on a localized screen.
• Visual diffs mask dynamic regions (timestamps, signal bars, battery, push banners) before adjusting numeric thresholds.
• Self-heal events are surfaced in the report so silent regressions cannot ride along.
• AI-suggested tests respect the two-axis flake taxonomy (logic vs device) — see mobile-testing.md.