Index of every page on this site: llms.txt
All skills
datadog-labs avatar

/cost-spike-investigation

@618fae3 official
Run it yourself

Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).

  • 1 file
  • 5.3 KB
  • Updated 7 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/cost-spike-investigation

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈51 tokens for metadata: the name and description. ≈1.2k when used: this file.

Description uses 2.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈215 characters, or 2.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Audit Trail: Cost / Usage Spike Investigation

Identify what caused a Datadog usage spike by correlating billing data with configuration change history.

The causal chain is: someone changed something → that change increased data volume → usage spiked → cost went up. Usage Metering tells you when and what; Audit Trail tells you who made the change.

Prerequisites

pup auth login   # OAuth2 (recommended) — covers audit queries
# Usage Metering queries also need DD_API_KEY + DD_APP_KEY
export DD_API_KEY=<your-api-key>
export DD_APP_KEY=<your-app-key>
export DD_SITE=datadoghq.com

Scope Boundary

This skill identifies configuration changes that may have caused a spike. It does not identify which specific user or process submitted the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.

Investigation Workflow

Step 1 — Identify the spike window and product family

START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")
END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  --data-urlencode "filter[timestamp][start]=${START}" \
  --data-urlencode "filter[timestamp][end]=${END}" \
  --data-urlencode "filter[product_families]=all" \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      product: .attributes.product_family,
      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]
    }]'

Product families with LLM/AI coverage: llm_observability, bits_ai, logs, apm

Step 2 — Pinpoint the spike

From Step 1, identify the hour/day where volume jumped. Note the timestamp as SPIKE_TIME.

Step 3 — Search Audit Trail for config changes in the 24h preceding the spike

pup audit-logs search \
  --query "@action:(created OR modified OR deleted)" \
  --from "SPIKE_TIME_MINUS_24H" \
  --to "SPIKE_TIME" \
  --limit 200 \
  -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      actor_type: .attributes.attributes.evt.actor.type,
      action: .attributes.attributes.action,
      event_category: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

Note: --from and --to accept ISO timestamps (e.g., 2026-05-01T14:00:00Z) or relative values (1h, 24h, 7d).

Step 4 — Narrow to product-relevant config changes

Filter to the audit categories most likely to affect the spiking product:

If this product spiked Add to query
llm_observability @evt.name:(Integration OR APM OR "Log Management")
logs / indexed_logs @evt.name:"Log Management" @asset.type:(pipeline OR index OR exclusion_filter)
apm / indexed_spans @evt.name:APM @asset.type:(retention_filter OR sampling_rate)
rum @evt.name:RUM
metrics @evt.name:Metrics

Example for LLM Observability spike:

pup audit-logs search \
  --query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \
  --from "SPIKE_TIME_MINUS_24H" \
  --to "SPIKE_TIME" \
  --limit 100 \
  -o json \
  | jq '[.data[] | {
      timestamp: .attributes.timestamp,
      user: .attributes.attributes.usr.email,
      action: .attributes.attributes.action,
      category: .attributes.attributes.evt.name,
      resource_type: .attributes.attributes.asset.type,
      resource_id: .attributes.attributes.asset.id
    }]'

Output Format

Usage spike detected:
  Product: <product_family>
  Spike time: <SPIKE_TIME>
  Volume: <baseline> → <spike_value> (<magnitude>×)

Configuration changes in 24h preceding spike:
  <timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>

Likely causal change: <most-proximate change matching the product family>

Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)

Next steps:
  - Confirm with <user_email> whether the change was intentional
  - If unintentional: revert <resource_id> and monitor volume
  - If intentional: update cost forecasts and alert thresholds

When No Causal Change Is Found

  1. The change may predate the 24h window — expand to 72h
  2. The increase may be from application-side instrumentation changes — check deploys
  3. The increase may be organic traffic growth — correlate with product launch or traffic event

References

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 618fae3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 7 months ago
Other metadata
metadata
{
  "version": "0.1.0",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,audit,cost,usage,spike,finops,dd-audit",
  "alwaysApply": "false"
}
  • datadog
  • audit
  • cost
  • usage
  • finops
  • spike
  • investigation
  • llm-observability
  • apm

README badge

README badge for datadog-labs/agent-skills/cost-spike-investigation

Correlates Datadog Usage Metering data with Audit Trail logs to identify configuration changes that caused a cost or usage spike. Targets billing investigations where you need to link when volume increased to who changed what in the preceding 24 hours.

Generated from the current SKILL.md.

Does this skill work with all Datadog product families?
It covers llm_observability, bits_ai, logs, apm, rum, and metrics. The skill includes product-specific audit queries for each family, but the core workflow applies to any product with Usage Metering and Audit Trail data.
What authentication does this skill require?
OAuth2 via `pup auth login` for audit queries (recommended), plus DD_API_KEY and DD_APP_KEY environment variables for Usage Metering API calls.
Can this skill identify which service or user submitted the data causing the spike?
No. The skill identifies configuration changes that may have caused the spike, not which specific user or process submitted the data itself. For per-submission attribution, you need LLM Observability traces or APM instrumentation.
What should I do if no causal change appears in the 24-hour window?
The skill suggests expanding the search to 72 hours, checking application-side instrumentation changes or deploys, or correlating with organic traffic growth events.

Generated from the current SKILL.md. These answers refresh after source changes.