Index of every page on this site: llms.txt
All skills
datadog-labs avatar

/dd-monitors

@618fae3 official
Run it yourself

Monitor management - list, search, file-based create, and alerting best practices.

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/dd-monitors

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈24 tokens for metadata: the name and description. ≈1.1k when used: this file.

Description uses 1.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈96 characters, or 1.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Datadog Monitors

Create, manage, and maintain monitors for alerting.

Prerequisites

This requires pup in your path. See Setup Pup.

Command Execution Order (Token-Efficient)

For scoped commands, use this order:

  1. Check context first (prior outputs, conversation, saved values).
  2. If a required value is missing, run a discovery command first.
  3. If still ambiguous, ask the user to confirm.
  4. Then run the target command.
  5. Avoid speculative commands likely to fail.

Quick Start

pup auth login

Common Operations

List Monitors

pup monitors list
pup monitors list --tags "team:platform"

Get Monitor

pup monitors get <id>

Create Monitor

pup monitors create --file monitor.json

Silence Alerts (Downtime)

# No pup monitors mute/unmute commands.
# Use downtime payloads to silence monitor notifications.
pup downtime create --file downtime.json
pup downtime cancel <downtime_id>

Monitor Creation Best Practices

1. Avoid Alert Fatigue

Rule Why
No flapping alerts Use last_Xm not last_1m
Meaningful thresholds Based on SLOs, not guesses
Actionable alerts If no action needed, don't alert
Include runbook @runbook-url in message
# WRONG - will flap constantly
query = "avg(last_1m):avg:system.cpu.user{*} > 50"  # ❌ Too sensitive

# CORRECT - stable alerting
query = "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"  # ✅ Reasonable window

2. Use Proper Scoping

# WRONG - alerts on everything
query = "avg(last_5m):avg:system.cpu.user{*} > 80"  # ❌ No scope

# CORRECT - scoped to what matters
query = "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"  # ✅

3. Set Recovery Thresholds

monitor = {
    "query": "avg(last_5m):avg:system.cpu.user{env:prod} > 80",
    "options": {
        "thresholds": {
            "critical": 80,
            "critical_recovery": 70,  # ✅ Prevents flapping
            "warning": 60,
            "warning_recovery": 50
        }
    }
}

4. Include Context in Messages

message = """
## High CPU Alert

Host: {{host.name}}
Current Value: {{value}}
Threshold: {{threshold}}

### Runbook
1. Check top processes: `ssh {{host.name}} 'top -bn1 | head -20'`
2. Check recent deploys
3. Scale if needed

@slack-ops @pagerduty-oncall
"""

NEVER Delete Monitors Directly

Use safe deletion workflow (same as dashboards):

def safe_mark_monitor_for_deletion(monitor_id: str, client) -> bool:
    """Mark monitor instead of deleting."""
    monitor = client.get_monitor(monitor_id)
    name = monitor.get("name", "")
    
    if "[MARKED FOR DELETION]" in name:
        print(f"Already marked: {name}")
        return False
    
    new_name = f"[MARKED FOR DELETION] {name}"
    client.update_monitor(monitor_id, {"name": new_name})
    print(f"✓ Marked: {new_name}")
    return True

Monitor Types

Type Use Case
metric alert CPU, memory, custom metrics
query alert Complex metric queries
service check Agent check status
event alert Event stream patterns
log alert Log pattern matching
composite Combine multiple monitors
apm APM metrics

Audit Monitors

# Find monitors without owners
pup monitors list | jq '.data[] | select(.tags | contains(["team:"]) | not) | {id, name}'

# Find noisy monitors (high alert count)
pup monitors list | jq '.data | sort_by(.overall_state_modified) | .[:10] | .[] | {id, name, status: .overall_state}'

Downtime vs Muting

Use When
Downtime Any planned silence window
Monitor edit Query/threshold behavior changes
# Downtime (preferred)
pup downtime create --file downtime.json

Failure Handling

Problem Fix
Alert not firing Check query returns data, thresholds
Too many alerts Increase window, add recovery threshold
No data alerts Check agent connectivity, metric exists
Auth error pup auth refresh

References

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts16d5 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    The skill provides comprehensive instructions for managing Datadog monitors using the 'pup' CLI tool. It includes best practices for stable alerting, proper scoping, and safe deletion workflows. No security issues were identified.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: LOW · No issues

  • Runlayer8mo

    1 file scanned · No issues

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 618fae3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 2 weeks ago
Other metadata
metadata
{
  "version": "1.0.2",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,monitors,alerting,alerts,dd-monitors",
  "globs": "**/datadog*.yaml,**/*monitor*",
  "alwaysApply": "false"
}
  • datadog
  • monitors
  • alerting
  • alerts
  • pup
  • metric-alert
  • downtime
  • slo

README badge

README badge for datadog-labs/agent-skills/dd-monitors

Manages Datadog monitors via the pup CLI: list, search, create from JSON files, and apply alerting best practices like recovery thresholds and scoped queries. Includes guidance on avoiding alert fatigue, setting meaningful thresholds based on SLOs, and safely marking monitors for deletion rather than direct removal.

Generated from the current SKILL.md.

What does this skill require to run?
The skill requires pup (Datadog's CLI tool) to be installed and available in your path. You must authenticate with pup auth login before running monitor commands.
Can I create monitors from files?
Yes. Use pup monitors create --file monitor.json to create monitors from a JSON file. The skill includes best practices for structuring monitor payloads.
How do I silence alerts without deleting monitors?
Use downtime payloads with pup downtime create --file downtime.json to temporarily silence notifications. The skill does not support pup monitors mute/unmute commands.
Does this skill help prevent alert fatigue?
Yes. The skill includes guidelines on avoiding flapping alerts (use longer time windows like last_5m), setting recovery thresholds, meaningful scoping, and including runbooks in alert messages.
What monitor types does this support?
The skill covers metric alerts, query alerts, service checks, event alerts, log alerts, composite monitors, and APM monitors. All types are created and managed through pup monitors commands.

Generated from the current SKILL.md. These answers refresh after source changes.