/beacon

LLM / AI Observability

GenAI semantic conventions, agentic tracing, token cost tracking, prompt quality metrics, Grafana dashboard design, GenAI observability checklist

1. GenAI Semantic Conventions (OTel)

OpenTelemetry defines standardized attribute names for AI/LLM telemetry under the gen_ai.* namespace (Semantic Conventions v1.40+).

Key Span Attributes

Key Metrics

Attribute Cardinality Warning

Do NOT include prompt content or response content as span attributes — they are high-cardinality and may contain PII. Use span events instead for optional, sampled capture.

# Correct: use span events for prompt content (opt-in, sampled)
span.add_event("gen_ai.content.prompt", {"gen_ai.prompt": prompt_text})
span.add_event("gen_ai.content.completion", {"gen_ai.completion": response_text})

2. Agentic Systems — Multi-Agent Tracing

Span Chain Design

For multi-agent or tool-calling workflows, each agent invocation and tool call should be a child span.

Trace: user-request-abc123
├── Span: orchestrator.plan          [gen_ai.operation.name=chat]
│   ├── gen_ai.usage.input_tokens=450
│   └── gen_ai.usage.output_tokens=120
├── Span: tool.web_search            [kind=CLIENT]
│   └── duration=320ms
├── Span: agent.researcher.execute   [gen_ai.operation.name=chat]
│   ├── gen_ai.usage.input_tokens=1200
│   └── gen_ai.usage.output_tokens=380
└── Span: agent.builder.execute      [gen_ai.operation.name=chat]
    ├── gen_ai.usage.input_tokens=890
    └── gen_ai.usage.output_tokens=640

Key Tracing Attributes for Agents

Context Propagation

Always propagate W3C TraceContext (traceparent, tracestate) headers when calling downstream agents or APIs. This ensures the full multi-agent workflow is visible as a single trace.

3. Token Cost Calculation Patterns

Basic Formula

cost = (input_tokens / 1000) × input_price_per_1k
      + (output_tokens / 1000) × output_price_per_1k

Provider Unit Price Reference (approximate, check current pricing)

Maintain a dated provider/model rate-card input instead of embedding mutable prices here. Verify official prices, billing units, service tier, context bands, cached input/write, batch and region; correlate usage with the actual model/run. _common/CLI_COMPATIBILITY.md identifies model availability, not billing entitlement. Unknown rates remain explicitly unverified and must not become a zero-cost dashboard series.

OTel Cost Metric

The multipliers below are illustrative, not a verified current tariff. Supply a dated rate card before production use.

Derive cost as a computed metric in the collector or Grafana:

# Prometheus recording rule example
# NOTE: per-token multipliers below (0.000003 / 0.000015) equal claude-sonnet-5's
# standard post-2026-09-01 rate ($3.00 / $15.00 per 1M tokens; see the pricing
# table above). The intro rate through 2026-08-31 is lower ($2.00 / $10.00 per
# 1M, i.e. 0.000002 / 0.000010) — swap multipliers on that date if this rule
# must track intro pricing until then.
- record: llm_request_cost_usd
  expr: |
    (
      gen_ai_client_token_usage_total{token_type="input", gen_ai_system="anthropic", gen_ai_request_model="claude-sonnet-5"}
      * 0.000003
    ) + (
      gen_ai_client_token_usage_total{token_type="output", gen_ai_system="anthropic", gen_ai_request_model="claude-sonnet-5"}
      * 0.000015
    )

Budget Alert Pattern

# Alert when daily LLM cost exceeds $50
- alert: LLMDailyCostBudgetExceeded
  expr: |
    increase(llm_request_cost_usd[24h]) > 50
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "LLM daily cost budget exceeded"
    runbook: "https://runbooks.internal/llm-cost-alert"

4. Prompt Quality Metrics

Key Metrics to Track

LLM-as-Judge Evaluation Pattern

# Async evaluation pipeline — do not block production path
async def evaluate_response_quality(
    input: str,
    response: str,
    trace_id: str,
) -> None:
    score = await judge_llm.evaluate(
        criteria=["relevance", "accuracy", "completeness"],
        input=input,
        response=response,
    )
    # Push to metrics backend
    metrics.histogram("llm.quality.relevance", score.relevance, tags={"trace_id": trace_id})
    metrics.histogram("llm.quality.accuracy", score.accuracy, tags={"trace_id": trace_id})

5. Grafana Dashboard Design for LLM

Panel Layout

Row 1: Overview
  ├── Total requests/min (stat)
  ├── P95 latency (stat)
  ├── Error rate % (stat)
  └── Estimated cost today (stat + trend)

Row 2: Token Usage
  ├── Input tokens/min by model (time series)
  ├── Output tokens/min by model (time series)
  └── Token ratio (output/input) — indicates verbosity (gauge)

Row 3: Quality & Reliability
  ├── Finish reasons distribution (pie chart)
  ├── Refusal rate trend (time series)
  └── Tool call success rate by tool (bar chart)

Row 4: Cost Analysis
  ├── Cost by model (time series stacked)
  ├── Cost by user/tenant (table, top 10)
  └── Projected monthly cost (stat + budget threshold line)

Row 5: Traces
  └── Weave trace explorer link filtered by gen_ai.system

Key Grafana Variables

variables:
  - name: gen_ai_system
    query: label_values(gen_ai_client_token_usage_total, gen_ai_system)
  - name: model
    query: label_values(gen_ai_client_token_usage_total{gen_ai_system="$gen_ai_system"}, gen_ai_request_model)
  - name: environment
    query: label_values(gen_ai_client_token_usage_total, environment)

6. GenAI Observability Checklist

Instrumentation

• OTel SDK initialized before any LLM client import
• gen_ai.* semantic conventions applied to all LLM calls
• W3C TraceContext propagated to downstream agents
• Token usage (input_tokens, output_tokens) recorded per request
• Finish reason captured on all responses
• Error spans recorded with error.type for 4xx/5xx/timeout

Metrics & Alerting

• gen_ai.client.operation.duration histogram with P50/P95/P99 SLOs defined
• gen_ai.client.token.usage tracked by model and token type
• Cost metric derived and dashboarded
• Daily/monthly cost budget alert configured
• High error rate alert (> 5% for 5 minutes)
• High latency alert (P95 > SLO threshold)

Privacy & Security

• Prompt/completion content NOT stored as span attributes in production
• PII/PHI redacted before any telemetry export
• Content logging is opt-in and subject to data retention policy
• API keys never appear in telemetry

Cost Optimization

• Model selection reviewed: use smaller model for simple tasks
• Prompt length monitored and optimized (track input token trend)
• Caching strategy evaluated for repeated queries
• Token budget set per user/tenant if multi-tenant

Source: OTel GenAI Semantic Conventions · OTel AI Agent Observability · Grafana LLM Observability