/trawl

Crawl Monitoring Reference

Purpose: Design observability for a distributed crawler fleet — fetch rate, frontier depth, error taxonomy, cost-per-URL, politeness adherence, and graceful shutdown/resume. Trawl emits monitoring specs; Builder wires metrics; Beacon owns SLO/SLI enforcement.

Scope Boundary

• trawl monitoring: Crawler-specific observability (this document).
• trawl frontier / politeness / extraction (elsewhere): Components being monitored.
• Beacon (elsewhere): General SLO/SLI framework. Trawl hands off specific signals to Beacon.
• Stream (elsewhere): Downstream data quality (Great Expectations). Crawl monitoring ends before ETL.
• Triage (elsewhere): Incident response. Crawl monitoring feeds alerts.

Signal Categories

Core Metrics

Throughput

Saturation

Error taxonomy

Categorize every failure. Aggregate and dashboard.

Cost-per-URL

cost_per_url = (compute_cost + egress_cost + proxy_cost + storage_cost) / urls_fetched

Track monthly. Alert on 20%+ trend change.

Compute includes render layer (headless is 10-40x static). Egress includes response bytes + proxy middleman. Proxy cost varies by tier (datacenter / residential / mobile). Storage = compressed write + index.

Compliance

Freshness

For re-crawl pipelines:

Progress

If drain_rate < 1 sustained, the frontier grows unbounded → crawl will never finish.

RED Method per Worker

Apply Google-SRE's RED (Rate, Errors, Duration) at the worker layer:

USE Method for Infrastructure

Apply USE (Utilization, Saturation, Errors) at infrastructure layer:

Graceful Shutdown & Resume

A crawler must be restartable without losing data.

Shutdown protocol

1. Coordinator signals workers to drain.
2. Workers finish in-flight fetches; do not pick up new URLs.
3. Workers flush any buffered writes to durable storage.
4. Workers persist their checkpoint (which batch ID last completed).
5. Coordinator records shutdown timestamp + frontier snapshot.

Resume protocol

1. On start, coordinator loads last frontier snapshot.
2. Replay WAL of any URLs dispatched but not confirmed.
3. Re-dispatch unconfirmed URLs (idempotent via content hash).
4. Workers resume with their saved checkpoint.
5. Re-hydrate dedup filters from disk snapshots.

Checkpoint cadence

• Frontier snapshot: every N minutes (e.g., 5).
• Dedup filter snapshot: every M minutes (e.g., 15).
• Worker-level checkpoint: every K fetched URLs (e.g., 100).

Dashboard Layout (Canvas handoff)

Row 1: SLO overview
  - fetch success SLO (rolling 30d)
  - p99 fetch latency SLO
  - cost-per-URL trend

Row 2: Throughput
  - fleet fetch rate (per second)
  - per-region fetch rate
  - fetch rate per worker (heatmap)

Row 3: Errors (stacked area)
  - 4xx / 5xx / timeout / DNS / TLS / render
  - top N domains by error count

Row 4: Frontier
  - total pending (line)
  - drain rate (ratio)
  - per-domain queue heatmap

Row 5: Compliance
  - crawl-delay violations
  - robots refresh freshness
  - opt-out signal respect ratio

Row 6: Cost
  - cost per URL (daily)
  - by render mode (static vs headless)
  - by region / proxy tier

Alerting Tiers

Follow Google SRE multi-burn-rate alerting for SLO-based alerts.

Workflow

DEFINE     →  identify signals per layer (throughput / saturation / errors / cost / compliance)
           →  set SLO targets (success rate, latency, freshness)

INSTRUMENT →  design structured logs (OpenTelemetry Logs)
           →  design metrics (OpenTelemetry Metrics, Prometheus format)
           →  design traces (per-URL trace id, sample rate)

BUDGET     →  compute error budget from SLO
           →  design multi-burn-rate alerts (fast/slow)

DASHBOARD  →  lay out RED/USE + crawler-specific panels
           →  hand off to Canvas for visualization
           →  hand off to Beacon for SLO ownership

RUNBOOK    →  per-alert response runbook
           →  escalation tree
           →  hand off to Triage for incident response

SHUTDOWN   →  design graceful shutdown + resume protocol
           →  checkpoint cadences

Output Template

## Crawl Monitoring: [Fleet / Corpus]

### SLO Targets
| SLI | Target | Window |
|-----|--------|--------|
| fetch_success_rate | ≥ 95% | rolling 30d |
| fetch_latency_p95 | ≤ 3s | rolling 7d |
| robots_respect | = 100% | rolling 24h |
| cost_per_url | ≤ $[N] | rolling 30d |

### Metrics (OpenTelemetry)
- `crawl_fetch_rate_per_sec` (counter)
- `crawl_fetch_duration_seconds` (histogram)
- `crawl_error_count` (counter; labels: class, domain)
- `crawl_frontier_size` (gauge)
- `crawl_dedup_hit_rate` (gauge)
- `crawl_cost_per_url_cents` (gauge)
- `crawl_robots_age_seconds` (gauge)

### Error Taxonomy
[list all error classes with expected rates]

### Cost Model
- compute: $[N] per 1M URLs
- egress: $[N] per 1M URLs
- proxy: $[N] per 1M URLs
- storage: $[N] per 1M URLs
- **total**: $[N] per 1M URLs

### Alerts
| Severity | Condition | Runbook |
|----------|-----------|---------|
| P0 | ... | ... |
| P1 | ... | ... |
| ... | ... | ... |

### Graceful Shutdown
- Frontier snapshot: every [N] min
- Dedup snapshot: every [M] min
- Worker checkpoint: every [K] URLs
- Resume replay: WAL since last confirmed batch

### Handoffs
- Beacon: SLO ownership + error budget tracking
- Canvas: dashboard rendering
- Triage: per-alert runbooks
- Builder: implementation of instrumentation
- Ledger: cost dashboards and anomaly detection

Anti-Patterns

Deliverable Contract

When monitoring completes, emit:

• SLO table with SLI definitions and targets.
• Metric catalog (OpenTelemetry names, labels, type).
• Error taxonomy with expected rates.
• Cost model per URL.
• Alert matrix with runbook links.
• Graceful shutdown / resume protocol with checkpoint cadences.
• Dashboard layout spec for Canvas handoff.
• Handoffs: Beacon, Canvas, Triage, Builder, Ledger.

References

• Google SRE Book — chapters on SLIs, SLOs, Error Budgets, Multi-Burn-Rate Alerting
• Google SRE Workbook — practical SLO patterns
• OpenTelemetry — spec for metrics, logs, traces (Traces/Metrics/Logs all Stable across major SDKs as of early 2026; Profiling signal RC Q1 2026, GA targeted Q3 2026)
• Prometheus — metric naming conventions and histogram guidance
• Brendan Gregg — USE Method for resource analysis
• Tom Wilkie — RED Method for service-level metrics
• Common Crawl operational blog — web-scale crawl observability learnings (latest dumps CC-MAIN-2026-04)
• Google's Webmaster logs — crawler-side signals from the inverse perspective
• Cloudflare AI Crawl Control docs — 402 Pay-Per-Crawl protocol, crawler-price / crawler-exact-price / crawler-charged headers
• Cloudflare radar 2026-01: Googlebot reaches 1.76x more unique URLs than GPTBot, 2.99x more than Meta-ExternalAgent, 3.26x more than Bingbot, 167x more than PerplexityBot, 714x more than CCBot
• Site Reliability Engineering Anti-patterns (O'Reilly, 2020)