Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/built-in-metrics

@f0ee110 official
Run it yourself

Instrument an existing codebase so LaunchDarkly records config metrics (duration, tokens, success/error, tool calls). Walks the ladder (convenience function → config() with a handler package → custom handler → manual track()) and picks the lowest-ceremony option that still captures what the Monitoring tab needs.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/built-in-metrics

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈84 tokens for metadata: the name and description. ≈2.6k when used: this file. ≈8.9k more on demand in 8 files.

Description uses 4.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈332 characters, or 4.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Agent Metrics Instrumentation

You're using a skill that gets LaunchDarkly agent metrics flowing from an existing provider call. Your job is to audit what's already there, pick the right tier from the ladder below, and implement it with the least ceremony that still captures the metrics the Monitoring tab needs.

The single most important thing to get right: in the current SDKs, metrics are a side effect of routing the call through the SDK. There is no tracker to create and no wrapper to call. If the provider call runs inside config().invoke() (or a convenience function), the SDK records the metrics. If it runs outside, nothing is recorded.

What the SDK records

Every invoke() and stream() call records these events against the served variation:

Event Metric
$ld:ai:duration:total Duration, in milliseconds
$ld:ai:generation:success / $ld:ai:generation:error Generation counts and error rate
$ld:ai:tokens:input, :output, :total Token usage, from the handler's normalized usage
$ld:ai:tool_call One event per tool call the SDK dispatches

The current SDKs do not record time to first token or user feedback. For feedback, see Tier 4.

The four-tier ladder

Walk from the top and stop at the first tier that fits:

Tier Pattern Use when
1 — Convenience function openai_messages(key, input, context) / openaiMessages(key, input, context), one per handler package One provider in one mode, no tools, no streaming. The shortest path.
2 — config() + handler package config(key=..., handler=[...]).invoke(input, context) / .stream(...) A handler package covers your provider, and you need more than one provider or mode, tools, template variables, streaming, or background judges.
3 — Custom handler create_handler((provider, mode), fn) / createHandler([provider, mode], fn), passed to config() No handler package fits (Gemini SDK, Bedrock via boto3, Strands, custom HTTP), or you want to keep an existing provider call as is.
4 — Manual track() get_client().track(event_key, context, result.track_data, value) Events the SDK doesn't record: user feedback and business outcomes. Never for duration, tokens, or success, which the SDK already records.

Workflow

1. Explore the existing call site

Before picking a tier, find the provider call and answer these questions:

  • Provider? OpenAI, Anthropic, Bedrock, Gemini, custom HTTP? → cross-reference with the handler matrix below.
  • Framework? Raw provider SDK, LangChain / LangGraph, OpenAI Agents SDK, Claude Agent SDK, Strands?
  • Mode? A completion (messages) or an agent (agent)? The served variation's mode selects the handler.
  • Streaming? Use stream() instead of invoke(). Metrics are recorded the same way, but time to first token isn't.
  • Language? Python or Node.js? Handler coverage is the same in both.
  • Already using a config? If not, route to configs-create first.
  • On a legacy SDK? If the code imports ldai / @launchdarkly/server-sdk-ai, or calls create_tracker(), track_metrics_of(), trackMetricsOf(), completion_config(), or create_model(), it's on the legacy surface. Replace it as part of this work (see Replacing legacy tracker code), or use legacy-sdk-migrate for a full codebase upgrade. For call sites with hardcoded models or prompts, hand off to migrate instead.

2. Look up your handler

Install the core package ("launchdarkly-ai-server[ld]" or @launchdarkly/ai-node) plus one handler package per provider and mode you serve:

Provider / framework Python handler package Node.js handler package Reference
OpenAI launchdarkly-ai-openai-messages, -openai-agents @launchdarkly/ai-openai-messages, -openai-agents openai-tracking.md
Anthropic launchdarkly-ai-claude-messages, -claude-agents @launchdarkly/ai-claude-messages, -claude-agents anthropic-tracking.md
LangChain / LangGraph (any provider) launchdarkly-ai-langchain-messages, -langchain-agents @launchdarkly/ai-langchain-messages, -langchain-agents langchain-tracking.md
AWS Bedrock LangChain handler + langchain-aws, or a custom handler LangChain handler, or a custom handler bedrock-tracking.md
Gemini / Google GenAI LangChain handler with a model factory, or a custom handler Same gemini-tracking.md
Strands Agents Custom handler Custom handler strands-tracking.md
Cohere, Mistral, custom HTTP Custom handler Custom handler anthropic-tracking.md shows the shape
Any provider, streaming stream() stream() streaming-tracking.md

The LangChain handlers register for every provider ("*") and only apply when no exact-provider handler matches.

3. Implement from the matching reference

Open the reference file and follow the pattern. Stop at the first tier that matches the app's shape.

Guardrails that apply to every tier:

  1. Handle disabled configs. invoke() raises a RuntimeError when the served variation is disabled or the config can't be reached. If you fall back on RuntimeError, first check (await inspect_config(key, context))["enabled"] (or (await inspectConfig(key, context)).enabled in Node.js) and re-raise when it's True — the config was served, so the error came from the run.
  2. Don't add tracking on top. The SDK already records duration, tokens, success, and errors. A manual track("$ld:ai:tokens:total", ...) next to invoke() double counts.
  3. Let errors propagate. When a handler raises, the SDK records the duration and a generation error, then re-raises to the caller. Don't catch and swallow the exception inside a custom handler, or the call is recorded as a success.
  4. Match the provider name exactly. Handlers route on the config's provider name and mode, case-sensitively: ("OpenAI", "messages"), ("Anthropic", "messages"), ("Bedrock", "messages"), ("Gemini", "messages").
  5. Shut down cleanly. Call await shutdown() before the process exits. It flushes pending events, then closes the client. Trailing events are at risk of being lost otherwise, in short-lived scripts and long-running services alike.

4. Verify

Confirm the Monitoring tab fills in:

  • Run one real request through the instrumented path.
  • Open the config in LaunchDarkly → Monitoring tab. Duration, token counts, and generation counts should appear within 1–2 minutes.
  • Or query the metrics API directly for generation, token, and duration totals.
  • Force an error (bad API key, unreachable endpoint) and confirm the error count increments.
  • If you see generations but no tokens, the handler isn't returning usage. Custom handlers must return {"output": ..., "usage": {"input": n, "output": n}}.

Replacing legacy tracker code

Legacy Current
LDAIClient(ldclient.get()) / initAi(ldClient) Lazy init from LD_SDK_KEY, or init_client() / initClient()
completion_config() / agent_config() + provider call config(key=..., handler=...).invoke(input, context)
create_model() + model.run() config().invoke() or a convenience function
tracker.track_metrics_of(extractor, fn) / trackMetricsOf Delete. Route the call through a handler package or a custom handler.
get_ai_metrics_from_response / custom LDAIMetrics extractors Delete. Handlers return normalized usage.
track_duration, track_tokens, track_success, track_error Delete. Recorded on every call.
track_tool_call(s) Delete. Recorded for each tool the SDK dispatches.
track_time_to_first_token Not recorded by the current SDKs.
track_feedback({kind}) get_client().track("$ld:ai:feedback:user:positive" or ":negative", context, result.track_data, 1) (Tier 4)
track_judge_result Attached judges record their own scores. See online-evals.
ldclient.get().flush() + close() await shutdown()

Tier 4 — User feedback

The SDK doesn't record thumbs-up / thumbs-down, but the Monitoring tab still reads the feedback events. Keep the track_data from the invoke() result and send the event when the user responds:

from ldclient import Context
from launchdarkly_ai_server import get_client

result = await config(key="support-agent", handler=handler).invoke(user_input, context)

# Later, when the user clicks thumbs up or down:
event = "$ld:ai:feedback:user:positive" if thumbs_up else "$ld:ai:feedback:user:negative"
get_client().track(event, Context.from_dict(context), result.track_data, 1)
import { getClient } from '@launchdarkly/ai-node';

const result = await config({ key: 'support-agent', handler }).invoke(userInput, context);

const event = thumbsUp ? '$ld:ai:feedback:user:positive' : '$ld:ai:feedback:user:negative';
getClient().track(event, context, result.trackData, 1);

track_data identifies the config, variation, version, and model, so the feedback lands on the variation that produced the response. stream() doesn't return track_data in the current SDKs, so feedback on streamed responses can't be attributed to a variation yet. In Python, convert the context dict with Context.from_dict() before calling track().

For business outcomes such as conversions or resolutions, use the custom-metrics skill with the same track_data.

Related skills

  • configs-create — prerequisite if the app doesn't have a config yet
  • custom-metrics — business metrics (conversion, resolution, retention) layered on top of the agent metrics this skill captures
  • online-evals — automatic quality scoring (LLM-as-judge) on sampled live requests; complementary to the metrics here
  • tools — tool implementations and custom handler details
  • migrate — Stage 4 of the hardcoded-to-AgentControl migration delegates to this skill

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts3d3 checks · Risk SAFE
  • Gen Agent Trust Hub3d

    The skill is safe and provides legitimate code patterns for instrumenting AI agents with LaunchDarkly metrics. It identifies a standard surface for indirect prompt injection as it involves processing user inputs and template variables within LLM prompts.

  • Socket3d

    No alerts

  • Snyk3d

    Risk: LOW · No issues

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 days ago
metadata
{
  "author": "launchdarkly",
  "version": "2.0.0-experimental"
}
Other metadata
compatibility
Requires `launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk` (Python, `pip install "launchdarkly-ai-server[ld]"`) or `@launchdarkly/ai-node` 1.0.0+ (Node.js), a handler package for your provider, and an existing config.
  • AI/ML
  • launchdarkly
  • metrics
  • instrumentation
  • tracking
  • monitoring
  • openai
  • langchain
  • anthropic

README badge

README badge for launchdarkly/ai-tooling/built-in-metrics

Instruments an existing codebase with LaunchDarkly config tracking by walking a four-tier ladder from managed runner down to raw manual calls, selecting the lowest-ceremony option that still captures duration, tokens, and success/error. Targets Python and Node codebases using OpenAI, LangChain, Vercel AI SDK, Anthropic, Gemini, Bedrock, or custom HTTP providers.

Generated from the current SKILL.md.

What LaunchDarkly SDK version does this skill require?
The skill requires launchdarkly-server-sdk-ai version 0.20.0 or later (Python: `launchdarkly-server-sdk-ai>=0.20.0`, Node: `@launchdarkly/server-sdk-ai>=0.20.0`) and an existing LaunchDarkly config.
Does this skill work with streaming responses?
Yes, but streaming with time-to-first-token (TTFT) tracking requires Tier 4 (raw manual tracking). Node offers `trackStreamMetricsOf` for the streaming wrapper, but TTFT must be tracked explicitly via `trackTimeToFirstToken`.
Which AI providers are supported?
The skill supports OpenAI, LangChain, Vercel AI SDK, AWS Bedrock, Anthropic, Gemini, Google GenAI, Strands Agents, and custom HTTP providers. Provider package availability and tracking tier options differ by framework and language—see the included reference matrix.
Can I use this with chat loops or only one-shot completions?
The skill supports both. Chat loops use Tier 1 (managed runner, highest-priority tier with zero tracker calls), while one-shot completions, agent steps, and other non-chat patterns use Tiers 2–4 depending on available provider packages.
What metrics does this capture?
The skill captures duration, input/output token counts, success/error status, and time-to-first-token for streaming—the four core metrics the LaunchDarkly Monitoring tab displays. The exact tracking method depends on which tier you implement (Tier 1 captures all automatically, Tier 2–3 require minimal code, Tier 4 is fully manual).

Generated from the current SKILL.md. These answers refresh after source changes.