Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/online-evals

@f0ee110 official
Run it yourself

Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/online-evals

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

README.md

≈323 tokens on demand. Your agent reads this file only when SKILL.md points to it.

LaunchDarkly Config Online Evaluations Skill

An Agent Skill for attaching judges to config variations for automatic LLM-as-a-judge evaluation.

Overview

This skill teaches agents how to:

  • Create custom judge configs with evaluation criteria
  • Attach judges to config variations via API
  • Configure sampling rates for cost control
  • Monitor evaluation results in the dashboard

Installation (Local)

Copy skills/agentcontrol/online-evals/ into your agent client's skills path.

Prerequisites

  • LaunchDarkly API access token with ai-configs:write permission
  • Existing config with variations (use configs-create skill)
  • For custom judges: understanding of LLM-as-a-judge methodology

Usage

Attach security and API contract judges to the model-selector config at 100% sampling
Create a custom judge that checks for scope creep in code changes

Structure

online-evals/
├── SKILL.md
└── README.md

Related

License

Apache-2.0

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts11d3 checks · Risk SAFE
  • Gen Agent Trust Hub15d

    This skill provides instructions and tools for configuring LaunchDarkly's LLM-as-a-judge evaluation system. It allows users to create judges that score AI responses based on custom criteria. A low-risk surface for indirect prompt injection is present because the system interpolates untrusted conversation history into judge prompts.

  • Socket11d

    No alerts

  • Snyk15d

    Risk: LOW · No issues

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 weeks ago
metadata
{
  "author": "launchdarkly",
  "version": "0.2.0"
}
Other metadata
compatibility
Requires the remotely hosted LaunchDarkly MCP server. Scripted setup without MCP needs an API access token with ai-configs:write permission. SDK examples use `launchdarkly-ai-server` 0.3.0+ (Python) or `@launchdarkly/ai-node` 0.3.0+ (Node.js).
  • Python
  • launchdarkly
  • evaluation
  • llm-as-a-judge
  • quality-scoring
  • ai-config
  • node.js
  • sdk

README badge

README badge for launchdarkly/ai-tooling/online-evals

Attaches LLM-as-a-judge evaluators to LaunchDarkly config variations for automatic quality scoring. Supports built-in judges (accuracy, relevance, toxicity) and custom judges via the LaunchDarkly API, with configurable sampling rates and automatic metric recording in Python v0.20.0+ and Node.js v0.20.0+ SDKs.

Generated from the current SKILL.md.

What SDK versions are required for automatic judge evaluation?
Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated track_judge_result / trackJudgeResult API.
Can judges be attached to agent mode configs?
No. Judges can only be attached to completion mode configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK instead.
What are the built-in judges LaunchDarkly provides?
Three pre-configured judges: Accuracy (measures correctness and grounding), Relevance (measures how well it addresses the request), and Toxicity (measures harmful phrasing, where lower scores are safer).
Can multiple judges with the same metric key be attached to one variation?
No. You cannot attach multiple judges with the same metric key to a single variation.
What API permissions are required?
Your LaunchDarkly API access token must have ai-configs:write permission.

Generated from the current SKILL.md. These answers refresh after source changes.