/trawl

Crawl Observability Design

Overview

Crawl observability ensures the system is operating correctly, efficiently, and within compliance bounds. It covers metrics collection, dashboards, alerting, cost modeling, and graceful shutdown monitoring.

Prometheus Metrics

Core Metrics

Derived Metrics (Recording Rules)

# Crawl throughput (URLs/sec, 5-minute average)
- record: crawler:throughput_5m
  expr: rate(crawler_urls_fetched_total[5m])

# Error rate by category (percentage)
- record: crawler:error_rate_4xx
  expr: rate(crawler_error_total{error_category="4xx"}[5m]) / rate(crawler_urls_fetched_total[5m])

- record: crawler:error_rate_5xx
  expr: rate(crawler_error_total{error_category="5xx"}[5m]) / rate(crawler_urls_fetched_total[5m])

# Cost per URL (derived from egress and compute)
- record: crawler:cost_per_url
  expr: (crawler_fetch_bytes_total * 0.00000001 + crawler_fetch_duration_seconds * 0.00005) / crawler_urls_fetched_total

# Frontier drain rate (URLs consumed per second)
- record: crawler:frontier_drain_rate
  expr: -deriv(crawler_frontier_depth[10m])

Error Category Labels

Grafana Dashboard Design

Dashboard Layout (4 rows)

Row 1: Overview
┌──────────────┬──────────────┬──────────────┬──────────────┐
│ URLs/sec     │ Total Fetched│ Frontier     │ Active       │
│ (gauge)      │ (counter)    │ Depth (gauge)│ Workers      │
└──────────────┴──────────────┴──────────────┴──────────────┘

Row 2: Performance
┌──────────────────────────┬──────────────────────────────┐
│ Fetch Latency (p50/p95)  │ Throughput by Domain (top 10)│
│ (heatmap)                │ (time series)                │
└──────────────────────────┴──────────────────────────────┘

Row 3: Errors & Compliance
┌──────────────────────────┬──────────────────────────────┐
│ Error Rate by Category   │ Compliance Blocks            │
│ (stacked area)           │ (by signal type, pie)        │
└──────────────────────────┴──────────────────────────────┘

Row 4: Cost & Health
┌──────────────────────────┬──────────────────────────────┐
│ Cost per URL (time series)│ Checkpoint Age (gauge)      │
│ (line, with budget line) │ (single stat, red > 5 min)  │
└──────────────────────────┴──────────────────────────────┘

Alert Thresholds

Critical Alerts (Page Immediately)

Warning Alerts (Notify)

Cost-per-URL Modeling

Component Breakdown

Estimation Formula

cost_per_url = compute_cost + egress_cost + proxy_cost + storage_cost + queue_cost

Where:
  compute_cost = avg_fetch_duration_sec × vcpu_cost_per_sec
  egress_cost  = avg_page_size_bytes × egress_cost_per_byte
  proxy_cost   = proxy_cost_per_request (if using proxy)
  storage_cost = avg_output_size_bytes × storage_cost_per_byte_month × retention_months
  queue_cost   = 3 × queue_operation_cost (enqueue + dequeue + dedup check)

Example (Medium tier, cloud-hosted, residential proxy):
  compute:  0.5s × $0.00004/vCPU-s  = $0.00002
  egress:   200KB × $0.09/GB         = $0.000018
  proxy:    $0.005/request            = $0.005
  storage:  50KB × $0.023/GB-mo × 3  = $0.0000035
  queue:    3 × $0.0000004            = $0.0000012
  ────────────────────────────────────────────
  Total:                              ≈ $0.0051/URL

Without proxy:                        ≈ $0.000042/URL

Budget Planning Table

Graceful Shutdown Observability

Shutdown Metrics

Shutdown sequence monitoring:
1. SIGTERM received → log timestamp
2. crawler_shutdown_initiated (gauge = 1)
3. crawler_inflight_requests (gauge, draining toward 0)
4. crawler_frontier_flush_status (0=pending, 1=complete)
5. crawler_kafka_offsets_committed (0=pending, 1=complete)
6. crawler_shutdown_complete (gauge = 1)
7. Process exit

Monitor: time between step 1 and step 6 should be < 30 seconds.
Alert if: crawler_inflight_requests > 0 after 30 seconds of shutdown.

Resume Verification

After restart, verify:
1. crawler_frontier_depth matches pre-shutdown value (±1%)
2. No duplicate fetches in first 5 minutes (check dedup hit rate)
3. Throughput reaches 80% of pre-shutdown level within 10 minutes
4. All workers reconnect within 2 minutes

SLO/SLI Alignment with Beacon

Trawl's observability design produces the inputs that Beacon uses for alerting. The mapping:

Instrument with OpenTelemetry SDK (Traces / Metrics / Logs all Stable in early 2026; Profiling signal at RC). Use OTLP exporter to Tempo/Jaeger (traces), Prometheus/Mimir (metrics), Loki/SigNoz (logs).