/growth

Full-Site SEO Audit

Purpose

Diagnose technical and structural issues that block traffic regardless of content quality. A page-level meta fix wastes effort if the site is uncrawlable, mis-indexed, or topologically incoherent. The audit produces a prioritized backlog, not a 200-row spreadsheet of findings.

2026 audit pressure points:

• Google March 2026 Core Update rolled out 2026-03-27 alongside a paired Spam Update (fastest spam update ever, completed in <20 hours). Both target AI-generated content lacking human editorial oversight via an upgraded SpamBrain (analysts attribute Gemini 4.0 Semantic Filter signals). 73% of top YMYL pages now display detailed author credentials, up from 58% pre-update. Audits must include AI-content authorship and E-E-A-T credential coverage. [Source: Search Engine Land / PPC.land coverage of March 2026 update, https://ppc.land/googles-march-2026-spam-update-is-live-what-changed-and-why-it-matters/]
• FAQ rich results retired (Jun 2026 — Rich Results Test support removed, Aug 2026 — Search Console API). FAQPage schema still parses for AI Overviews / ChatGPT / Perplexity citation. Audit existing FAQ markup for schema-content consistency, not rich-result eligibility. [Source: Search Engine Land, https://searchengineland.com/google-to-no-longer-support-faq-rich-results-476957]
• Schema.org v30.0 released 2026-03-19 (823 types), expanded credential / e-commerce / supply-chain schema for AI verification. [Source: https://schema.org/docs/releases.html]

Scope Boundary

• IN scope: crawlability, indexability, site architecture, internal linking, content gap, log-file analysis (Googlebot + AI bots), schema coverage, hreflang, canonical hygiene, redirect chains.
• OUT of scope: per-page meta authoring (seo), CRO experiments (cro), Core Web Vitals deep-dive (vitals), keyword universe construction (keyword), GEO content quality (geo).

Core Concepts

The 5-Layer Audit Stack

Audit in this order. Lower layers gate upper layers; do not optimize content on a site Google cannot crawl.

L1 Crawlability Checks

Important AI-bot distinction: training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and search/retrieval bots (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User) require different policy decisions. Blocking training bots does not affect AI-search citation. Blocking search/retrieval bots removes you from AI-search results.

Anthropic publishes a four-bot taxonomy (confirmed 2026-05): ClaudeBot (training), Claude-SearchBot (search index/retrieval), Claude-User (user-initiated fetch from Claude.ai), claude-code (Claude Code CLI WebFetch). Control each independently. [Source: ALM Corp, https://almcorp.com/blog/anthropic-claude-bots-robots-txt-strategy/]

Do not rely on llms.txt: SE Ranking found ~10.13% adoption across 300k domains but GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended overwhelmingly skip /llms.txt and crawl HTML directly. No major AI vendor has publicly committed to honoring llms.txt as of Q1 2026. [Source: AEO Press, https://www.aeo.press/ai/the-state-of-llms-txt-in-2026]

L2 Indexability Checks

L3 Architecture Checks

L4 Content Checks

L5 Schema and GEO Checks

Log-File Analysis

Server logs are the only ground truth for crawler behavior. Look for:

Tools: GoAccess, Splunk, Screaming Frog Log File Analyzer, Botify, Lumar.

Audit Output: The 30-Item Backlog

Resist the 200-row spreadsheet. Audits fail by overwhelming the implementer. Cap output at 30 prioritized items grouped by layer. Each item:

• Title (one line)
• Layer (L1–L5)
• Severity (Blocker / High / Medium / Low)
• Affected URL count
• Effort estimate (S / M / L)
• Recommended fix (concrete, not "improve X")
• Verification step

Workflow

1. Snapshot baseline — GSC export (last 90 days), site: count, top-100 page log-file extract.
2. Run L1 — robots, sitemap, redirect chains, AI bot policy.
3. Run L2 — canonical, noindex, hreflang, duplicate detection.
4. Run L3 — crawl with Screaming Frog or Sitebulb; map click depth and orphans.
5. Run L4 — content gap vs top-3 competitors per cluster; thin-content sweep.
6. Run L5 — schema coverage, GEO direct-answer check, AI bot log access.
7. Triage — group findings, drop noise, prioritize by traffic-at-risk × effort.
8. Cap to 30 items — anything below threshold goes into a parking-lot file, not the active backlog.
9. Hand off — to seo for per-page implementation, to vitals for performance items, to geo for AI-citation items, to bolt for performance code, to artisan for UI changes.

Output Template

audit_report:
  baseline:
    indexed_pages: 12480
    avg_click_depth: 3.7
    googlebot_fetches_30d: 184320
    ai_bots_in_logs: [GPTBot, ClaudeBot, PerplexityBot]
    ai_bots_missing: [OAI-SearchBot, Claude-SearchBot]  # CRITICAL
  layers:
    L1:
      passed: 6
      failed: 2
    L2:
      passed: 7
      failed: 1
    L3:
      passed: 5
      failed: 3
    L4:
      passed: 4
      failed: 3
    L5:
      passed: 4
      failed: 5
  backlog_top_30:
    - id: A-01
      layer: L5
      title: "OAI-SearchBot blocked by Cloudflare WAF rule #4421"
      severity: Blocker
      affected_urls: all
      effort: S
      fix: "Add Cloudflare rule to allow user-agent OAI-SearchBot/1.0 with rate limit 60/min"
      verify: "Confirm logs show 200 OK to OAI-SearchBot within 48h"
      handoff: scaffold | gear
    - id: A-02
      layer: L3
      title: "1,240 orphan blog posts (no internal links)"
      severity: High
      affected_urls: 1240
      effort: M
      fix: "Add 'Related posts' module powered by topical cluster ID; link 5 siblings per post"
      verify: "Re-crawl shows < 50 orphans"
      handoff: artisan
    # ... up to 30 items
  parking_lot_count: 87
  refresh_cadence: quarterly

Anti-Patterns

• 200-row audit spreadsheets — rejected by implementers, nothing ships.
• Auditing without log files — half of crawl issues are invisible from a desk crawler.
• Ignoring AI search-bot vs training-bot distinction — silently blocks AI citation.
• Trusting tool "site health score" as a single number — averages hide blockers.
• Recommending fixes without effort estimates — implementer cannot triage.
• Auditing on staging that disallows all bots — crawler-blocked staging gives false negatives.
• "Improve X" findings without concrete fix steps — non-actionable.
• Sitewide canonical-to-homepage — common bug; eliminates 99% of indexable pages.
• Treating noindex + canonical to another URL as equivalent — they signal different things; mixing them confuses Google.
• Auditing once per year — quarterly cadence is the minimum for SERP-relevant signals.

Deliverable Contract

A site audit is complete when:

• All 5 layers checked with explicit pass/fail per check.
• Log-file analysis covers ≥ 30 days for both Googlebot and AI bots.
• Backlog capped at 30 items, prioritized.
• Each item has severity, affected URL count, effort estimate, concrete fix, verification step, handoff agent.
• Parking-lot file documents excluded findings (so nothing is lost).
• Refresh cadence is set (quarterly default).

References

• Google Search Central, Search Quality Evaluator Guidelines (2024).
• Google Search Central, AI features and your website (2024) — AI bot taxonomy.
• Aleyda Solis, Crawling Mondays (technical SEO methodology).
• Bartosz Góralewicz, Onely — JS rendering and crawl auditing.
• Botify, The Crawl Budget Optimization Playbook (2024).
• Screaming Frog, SEO Vector User Guide.
• Tomek Rudzki, Indexing Insight — indexability deep-dive.
• John Mueller (Google), Search Off the Record podcast — canonical / hreflang clarifications.