---
name: built-in-metrics
description: "Instrument an existing codebase so LaunchDarkly records config metrics (duration, tokens, success/error, tool calls). Walks the ladder (convenience function → config() with a handler package → custom handler → manual track()) and picks the lowest-ceremony option that still captures what the Monitoring tab needs."
license: Apache-2.0
compatibility: Requires `launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk` (Python, `pip install "launchdarkly-ai-server[ld]"`) or `@launchdarkly/ai-node` 1.0.0+ (Node.js), a handler package for your provider, and an existing config.
metadata:
  author: launchdarkly
  version: "2.0.0-experimental"
title: built-in-metrics
canonical_url: https://skilld.dev/gh/launchdarkly/ai-tooling/built-in-metrics
last_updated: 2026-11-17T01:05:02.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/anthropic-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/anthropic-tracking.md), [references/bedrock-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/bedrock-tracking.md), [references/gemini-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/gemini-tracking.md), [references/langchain-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/langchain-tracking.md), [references/metrics-api.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/metrics-api.md), [references/openai-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/openai-tracking.md), [references/strands-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/strands-tracking.md), [references/streaming-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/streaming-tracking.md).
>
> If the user asked to install this Skill, run `npx skilld install launchdarkly/ai-tooling/built-in-metrics`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/built-in-metrics`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/launchdarkly/ai-tooling/built-in-metrics) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/built-in-metrics`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/built-in-metrics --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/built-in-metrics --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Agent Metrics Instrumentation

You're using a skill that gets LaunchDarkly agent metrics flowing from an existing provider call. Your job is to audit what's already there, pick the right tier from the ladder below, and implement it with the **least ceremony that still captures the metrics the Monitoring tab needs**.

The single most important thing to get right: **in the current SDKs, metrics are a side effect of routing the call through the SDK.** There is no tracker to create and no wrapper to call. If the provider call runs inside `config().invoke()` (or a convenience function), the SDK records the metrics. If it runs outside, nothing is recorded.

## What the SDK records

Every `invoke()` and `stream()` call records these events against the served variation:

| Event | Metric |
|---|---|
| `$ld:ai:duration:total` | Duration, in milliseconds |
| `$ld:ai:generation:success` / `$ld:ai:generation:error` | Generation counts and error rate |
| `$ld:ai:tokens:input`, `:output`, `:total` | Token usage, from the handler's normalized usage |
| `$ld:ai:tool_call` | One event per tool call the SDK dispatches |

The current SDKs do **not** record time to first token or user feedback. For feedback, see Tier 4.

## The four-tier ladder

Walk from the top and stop at the first tier that fits:

| Tier | Pattern | Use when |
|------|---------|----------|
| **1 — Convenience function** | `openai_messages(key, input, context)` / `openaiMessages(key, input, context)`, one per handler package | One provider in one mode, no tools, no streaming. The shortest path. |
| **2 — `config()` + handler package** | `config(key=..., handler=[...]).invoke(input, context)` / `.stream(...)` | A handler package covers your provider, and you need more than one provider or mode, tools, template variables, streaming, or background judges. |
| **3 — Custom handler** | `create_handler((provider, mode), fn)` / `createHandler([provider, mode], fn)`, passed to `config()` | No handler package fits (Gemini SDK, Bedrock via boto3, Strands, custom HTTP), or you want to keep an existing provider call as is. |
| **4 — Manual `track()`** | `get_client().track(event_key, context, result.track_data, value)` | Events the SDK doesn't record: user feedback and business outcomes. Never for duration, tokens, or success, which the SDK already records. |

## Workflow

### 1. Explore the existing call site

Before picking a tier, find the provider call and answer these questions:

- [ ] **Provider?** OpenAI, Anthropic, Bedrock, Gemini, custom HTTP? → cross-reference with the handler matrix below.
- [ ] **Framework?** Raw provider SDK, LangChain / LangGraph, OpenAI Agents SDK, Claude Agent SDK, Strands?
- [ ] **Mode?** A completion (`messages`) or an agent (`agent`)? The served variation's mode selects the handler.
- [ ] **Streaming?** Use `stream()` instead of `invoke()`. Metrics are recorded the same way, but time to first token isn't.
- [ ] **Language?** Python or Node.js? Handler coverage is the same in both.
- [ ] **Already using a config?** If not, route to `configs-create` first.
- [ ] **On a legacy SDK?** If the code imports `ldai` / `@launchdarkly/server-sdk-ai`, or calls `create_tracker()`, `track_metrics_of()`, `trackMetricsOf()`, `completion_config()`, or `create_model()`, it's on the legacy surface. Replace it as part of this work (see [Replacing legacy tracker code](#replacing-legacy-tracker-code)), or use `legacy-sdk-migrate` for a full codebase upgrade. For call sites with hardcoded models or prompts, hand off to `migrate` instead.

### 2. Look up your handler

Install the core package (`"launchdarkly-ai-server[ld]"` or `@launchdarkly/ai-node`) plus one handler package per provider and mode you serve:

| Provider / framework | Python handler package | Node.js handler package | Reference |
|---|---|---|---|
| OpenAI | `launchdarkly-ai-openai-messages`, `-openai-agents` | `@launchdarkly/ai-openai-messages`, `-openai-agents` | [openai-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/openai-tracking.md) |
| Anthropic | `launchdarkly-ai-claude-messages`, `-claude-agents` | `@launchdarkly/ai-claude-messages`, `-claude-agents` | [anthropic-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/anthropic-tracking.md) |
| LangChain / LangGraph (any provider) | `launchdarkly-ai-langchain-messages`, `-langchain-agents` | `@launchdarkly/ai-langchain-messages`, `-langchain-agents` | [langchain-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/langchain-tracking.md) |
| AWS Bedrock | LangChain handler + `langchain-aws`, or a custom handler | LangChain handler, or a custom handler | [bedrock-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/bedrock-tracking.md) |
| Gemini / Google GenAI | LangChain handler with a model factory, or a custom handler | Same | [gemini-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/gemini-tracking.md) |
| Strands Agents | Custom handler | Custom handler | [strands-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/strands-tracking.md) |
| Cohere, Mistral, custom HTTP | Custom handler | Custom handler | [anthropic-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/anthropic-tracking.md#tier-3--custom-handler) shows the shape |
| Any provider, streaming | `stream()` | `stream()` | [streaming-tracking.md](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/streaming-tracking.md) |

The LangChain handlers register for every provider (`"*"`) and only apply when no exact-provider handler matches.

### 3. Implement from the matching reference

Open the reference file and follow the pattern. Stop at the first tier that matches the app's shape.

Guardrails that apply to every tier:

1. **Handle disabled configs.** `invoke()` raises a `RuntimeError` when the served variation is disabled or the config can't be reached. If you fall back on `RuntimeError`, first check `(await inspect_config(key, context))["enabled"]` (or `(await inspectConfig(key, context)).enabled` in Node.js) and re-raise when it's `True` — the config was served, so the error came from the run.
2. **Don't add tracking on top.** The SDK already records duration, tokens, success, and errors. A manual `track("$ld:ai:tokens:total", ...)` next to `invoke()` double counts.
3. **Let errors propagate.** When a handler raises, the SDK records the duration and a generation error, then re-raises to the caller. Don't catch and swallow the exception inside a custom handler, or the call is recorded as a success.
4. **Match the provider name exactly.** Handlers route on the config's provider name and mode, case-sensitively: `("OpenAI", "messages")`, `("Anthropic", "messages")`, `("Bedrock", "messages")`, `("Gemini", "messages")`.
5. **Shut down cleanly.** Call `await shutdown()` before the process exits. It flushes pending events, then closes the client. Trailing events are at risk of being lost otherwise, in short-lived scripts and long-running services alike.

### 4. Verify

Confirm the Monitoring tab fills in:

- [ ] Run one real request through the instrumented path.
- [ ] Open the config in LaunchDarkly → **Monitoring** tab. Duration, token counts, and generation counts should appear within 1–2 minutes.
- [ ] Or query the [metrics API](https://skilld.dev/api/skills-raw/launchdarkly/ai-tooling/built-in-metrics/references/metrics-api.md) directly for generation, token, and duration totals.
- [ ] Force an error (bad API key, unreachable endpoint) and confirm the error count increments.
- [ ] If you see generations but no tokens, the handler isn't returning usage. Custom handlers must return `{"output": ..., "usage": {"input": n, "output": n}}`.

## Replacing legacy tracker code

| Legacy | Current |
|---|---|
| `LDAIClient(ldclient.get())` / `initAi(ldClient)` | Lazy init from `LD_SDK_KEY`, or `init_client()` / `initClient()` |
| `completion_config()` / `agent_config()` + provider call | `config(key=..., handler=...).invoke(input, context)` |
| `create_model()` + `model.run()` | `config().invoke()` or a convenience function |
| `tracker.track_metrics_of(extractor, fn)` / `trackMetricsOf` | Delete. Route the call through a handler package or a custom handler. |
| `get_ai_metrics_from_response` / custom `LDAIMetrics` extractors | Delete. Handlers return normalized usage. |
| `track_duration`, `track_tokens`, `track_success`, `track_error` | Delete. Recorded on every call. |
| `track_tool_call(s)` | Delete. Recorded for each tool the SDK dispatches. |
| `track_time_to_first_token` | Not recorded by the current SDKs. |
| `track_feedback({kind})` | `get_client().track("$ld:ai:feedback:user:positive" or ":negative", context, result.track_data, 1)` (Tier 4) |
| `track_judge_result` | Attached judges record their own scores. See `online-evals`. |
| `ldclient.get().flush()` + `close()` | `await shutdown()` |

## Tier 4 — User feedback

The SDK doesn't record thumbs-up / thumbs-down, but the Monitoring tab still reads the feedback events. Keep the `track_data` from the `invoke()` result and send the event when the user responds:

```python
from ldclient import Context
from launchdarkly_ai_server import get_client

result = await config(key="support-agent", handler=handler).invoke(user_input, context)

# Later, when the user clicks thumbs up or down:
event = "$ld:ai:feedback:user:positive" if thumbs_up else "$ld:ai:feedback:user:negative"
get_client().track(event, Context.from_dict(context), result.track_data, 1)
```

```typescript
import { getClient } from '@launchdarkly/ai-node';

const result = await config({ key: 'support-agent', handler }).invoke(userInput, context);

const event = thumbsUp ? '$ld:ai:feedback:user:positive' : '$ld:ai:feedback:user:negative';
getClient().track(event, context, result.trackData, 1);
```

`track_data` identifies the config, variation, version, and model, so the feedback lands on the variation that produced the response. `stream()` doesn't return `track_data` in the current SDKs, so feedback on streamed responses can't be attributed to a variation yet. In Python, convert the context dict with `Context.from_dict()` before calling `track()`.

For business outcomes such as conversions or resolutions, use the `custom-metrics` skill with the same `track_data`.

## Related skills

- `configs-create` — prerequisite if the app doesn't have a config yet
- `custom-metrics` — business metrics (conversion, resolution, retention) layered on top of the agent metrics this skill captures
- `online-evals` — automatic quality scoring (LLM-as-judge) on sampled live requests; complementary to the metrics here
- `tools` — tool implementations and custom handler details
- `migrate` — Stage 4 of the hardcoded-to-AgentControl migration delegates to this skill
