/playwright-regression-testing

Flaky Test Management

Part of the playwright-regression-testing skill. See SKILL.md for full context.

Detailed reference for managing flaky tests in Playwright regression suites. Covers retry policies, quarantine strategies, detection checklists, and suite health metrics.

Retry Policy

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0, // Retry only in CI
  reporter: [["html"], ["json", { outputFile: "test-results/results.json" }]],
  use: {
    trace: "on-first-retry", // Capture trace on retry
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
});

Quarantine Flaky Tests

Isolate known flaky tests to prevent blocking the pipeline while tracking them for fixes:

// Mark flaky tests with fixme — skipped but tracked
test.fixme("intermittent timeout on slow network", async ({ page }) => {
  // TODO: investigate — see issue #1234
});

// Quarantine: keep the execution tag (@regression) and track flakiness via annotation
test(
  "payment callback race condition",
  {
    tag: "@regression",
    annotation: {
      type: "issue",
      description: "https://github.com/org/repo/issues/1234",
    },
  },
  async ({ page }) => {
    test.skip(!!process.env.CI, "Quarantined — flaky in CI, see linked issue");
    // Test body here
  },
);

Flaky Test Detection Checklist

Suite Health Metrics

Track these metrics to maintain regression suite quality:

Troubleshooting