Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/built-in-metrics

@f0ee110 official
Run it yourself

Instrument an existing codebase so LaunchDarkly records config metrics (duration, tokens, success/error, tool calls). Walks the ladder (convenience function → config() with a handler package → custom handler → manual track()) and picks the lowest-ceremony option that still captures what the Monitoring tab needs.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/built-in-metrics

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesmetrics-api.md

≈935 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Metrics API

Retrieve config metrics via the LaunchDarkly REST API. The MCP server has no tool for these totals.

Endpoint

GET /api/v2/projects/{projectKey}/ai-configs/{configKey}/metrics

Authentication

Requires an API token with ai-configs:read permission. Ask the user to paste one for the session; don't search environment variables or config files for it.

headers = {
    "Authorization": "your-api-token",
    "LD-API-Version": "beta"
}

Implementation

import requests
import time
import os

def get_ai_config_metrics(project_key: str, config_key: str, env: str = "production", hours: int = 24):
    """Get config metrics for the last N hours."""
    API_TOKEN = "{api_token}"  # token the user provided for this session

    # The endpoint takes Unix timestamps in milliseconds
    now = int(time.time() * 1000)
    start = now - (hours * 3600 * 1000)

    url = f"https://app.launchdarkly.com/api/v2/projects/{project_key}/ai-configs/{config_key}/metrics"

    params = {
        "from": start,
        "to": now,
        "env": env
    }

    headers = {
        "Authorization": API_TOKEN,
        "LD-API-Version": "beta"
    }

    response = requests.get(url, headers=headers, params=params)

    if response.status_code == 200:
        metrics = response.json()
        print(f"[OK] Metrics for {config_key} (last {hours} hours, {env}):")
        print(f"     Generations: {metrics.get('generationCount', 0):,}")
        print(f"     Success: {metrics.get('generationSuccessCount', 0):,}")
        print(f"     Errors: {metrics.get('generationErrorCount', 0):,}")
        print(f"     Input Tokens: {metrics.get('inputTokens', 0):,}")
        print(f"     Output Tokens: {metrics.get('outputTokens', 0):,}")
        print(f"     Total Tokens: {metrics.get('totalTokens', 0):,}")
        print(f"     Input Cost: ${metrics.get('inputCost', 0):.4f}")
        print(f"     Output Cost: ${metrics.get('outputCost', 0):.4f}")
        print(f"     Duration (ms): {metrics.get('durationMs', 0):,}")
        print(f"     TTFT (ms): {metrics.get('timeToFirstTokenMs', 0):,}")
        print(f"     Thumbs Up: {metrics.get('thumbsUp', 0)}")
        print(f"     Thumbs Down: {metrics.get('thumbsDown', 0)}")
        return metrics
    else:
        print(f"[ERROR] Failed to get metrics: {response.status_code}")
        return None

Response Fields

Field Description
generationCount Total number of generations
generationSuccessCount Successful generations
generationErrorCount Failed generations
inputTokens Total input tokens used
outputTokens Total output tokens generated
totalTokens Sum of input + output tokens
inputCost Cost for input tokens
outputCost Cost for output tokens
durationMs Total duration in milliseconds
timeToFirstTokenMs Time to first token (streaming)
thumbsUp Positive feedback count
thumbsDown Negative feedback count

Query Parameters

Parameter Type Description
from int Unix timestamp in milliseconds for start of range
to int Unix timestamp in milliseconds for end of range
env string Environment key (default: "production")

Notes

  • Time range is specified in Unix timestamps in milliseconds. Passing seconds returns all-zero counts rather than an error.
  • timeToFirstTokenMs stays at 0 for apps on the current AI SDKs, which don't record time to first token
  • Costs are calculated based on model pricing and token usage
  • Feedback counts require user feedback tracking implementation
  • Rate limits apply; see API documentation for details

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts3d3 checks · Risk SAFE
  • Gen Agent Trust Hub3d

    The skill is safe and provides legitimate code patterns for instrumenting AI agents with LaunchDarkly metrics. It identifies a standard surface for indirect prompt injection as it involves processing user inputs and template variables within LLM prompts.

  • Socket3d

    No alerts

  • Snyk3d

    Risk: LOW · No issues

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 days ago
metadata
{
  "author": "launchdarkly",
  "version": "2.0.0-experimental"
}
Other metadata
compatibility
Requires `launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk` (Python, `pip install "launchdarkly-ai-server[ld]"`) or `@launchdarkly/ai-node` 1.0.0+ (Node.js), a handler package for your provider, and an existing config.
  • AI/ML
  • launchdarkly
  • metrics
  • instrumentation
  • tracking
  • monitoring
  • openai
  • langchain
  • anthropic

README badge

README badge for launchdarkly/ai-tooling/built-in-metrics

Instruments an existing codebase with LaunchDarkly config tracking by walking a four-tier ladder from managed runner down to raw manual calls, selecting the lowest-ceremony option that still captures duration, tokens, and success/error. Targets Python and Node codebases using OpenAI, LangChain, Vercel AI SDK, Anthropic, Gemini, Bedrock, or custom HTTP providers.

Generated from the current SKILL.md.

What LaunchDarkly SDK version does this skill require?
The skill requires launchdarkly-server-sdk-ai version 0.20.0 or later (Python: `launchdarkly-server-sdk-ai>=0.20.0`, Node: `@launchdarkly/server-sdk-ai>=0.20.0`) and an existing LaunchDarkly config.
Does this skill work with streaming responses?
Yes, but streaming with time-to-first-token (TTFT) tracking requires Tier 4 (raw manual tracking). Node offers `trackStreamMetricsOf` for the streaming wrapper, but TTFT must be tracked explicitly via `trackTimeToFirstToken`.
Which AI providers are supported?
The skill supports OpenAI, LangChain, Vercel AI SDK, AWS Bedrock, Anthropic, Gemini, Google GenAI, Strands Agents, and custom HTTP providers. Provider package availability and tracking tier options differ by framework and language—see the included reference matrix.
Can I use this with chat loops or only one-shot completions?
The skill supports both. Chat loops use Tier 1 (managed runner, highest-priority tier with zero tracker calls), while one-shot completions, agent steps, and other non-chat patterns use Tiers 2–4 depending on available provider packages.
What metrics does this capture?
The skill captures duration, input/output token counts, success/error status, and time-to-first-token for streaming—the four core metrics the LaunchDarkly Monitoring tab displays. The exact tracking method depends on which tier you implement (Tier 1 captures all automatically, Tier 2–3 require minimal code, Tier 4 is fully manual).

Generated from the current SKILL.md. These answers refresh after source changes.