Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/launchdarkly-experiment-guide

@f0ee110 official
Run it yourself

Guide a developer from 'I don't know what to test' to a ready-to-create LaunchDarkly experiment. Use when someone wants to run an experiment or A/B test but doesn't have a metric, a flag, or a hypothesis yet, or isn't sure what to measure. Scans the codebase and the connected LaunchDarkly account, proposes a hypothesis and metric from real context, wires up the flag and tracking code, then drafts the experiment and offers to create it. Built for engineers without a data-science or product-analytics background.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/launchdarkly-experiment-guide

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencescreate-experiment.md

≈1.6k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Create the experiment (abstracted: MCP or REST)

This is one logical step with two implementations. Detect what's available and use it. Only run this after the user has confirmed the draft (Step 7).

Detection order (already confirmed up front by the skill's Step 0 preflight, which checks for the specific create tools — a flags-only MCP does not qualify):

  1. An LD MCP exposing the experiment + metric tools (create-experiment, create-metric, start-experiment-iteration, …) → MCP path (preferred).
  2. Else a LaunchDarkly API access token the user pastes for this session → REST path.
  3. Both present → use MCP; fall back to REST only for an operation MCP doesn't expose.
  4. No create capability (flags-only MCP, or nothing) → you should never reach the create step: Step 0 fails fast. If the user chose draft-only mode, present these calls for them to run later and stop.

You need these values, gathered in earlier steps:

  • projectKey, environmentKey (from get-projects, or get-environment for one environment).
  • Primary metric key (reused or newly created).
  • Flag key, the variation IDs for control/treatment, the targeting ruleId (fallthrough rule id is the string "fallthrough"), and current flagConfigVersion.

Prefer delegating the mechanical details to launchdarkly-experiment-setup. It owns the field-level rules (mutableFieldsByStatus, iteration lifecycle, stopping with a winner). This reference is the abstraction contract so this skill can complete the flow even when that skill isn't loaded.


MCP path (preferred)

  1. Create any missing metric — create-metric (or delegate to launchdarkly-metric-create). Skip if reusing an existing metric.
  2. Create the experiment with its first iteration — create-experiment:
{
  "projectKey": "my-project",
  "environmentKey": "production",
  "key": "redesigned-checkout-experiment",
  "name": "Redesigned Checkout Experiment",
  "description": "Test the redesigned checkout against the current flow.",
  "iteration": {
    "hypothesis": "The redesigned checkout will lift completion ~3% for logged-in shoppers.",
    "primarySingleMetricKey": "checkout-completed",
    "metrics": [
      { "key": "checkout-completed" },
      { "key": "checkout-error-rate" }
    ],
    "treatments": [
      { "name": "Control", "baseline": true, "allocationPercent": 50,
        "parameters": [{ "flagKey": "new-checkout", "variationId": "<control-variation-id>" }] },
      { "name": "New Checkout", "baseline": false, "allocationPercent": 50,
        "parameters": [{ "flagKey": "new-checkout", "variationId": "<treatment-variation-id>" }] }
    ],
    "flags": {
      "new-checkout": { "ruleId": "fallthrough", "flagConfigVersion": 7 }
    },
    "randomizationUnit": "user"
  }
}
  1. Verify — get-experiments; confirm the experiment exists with a not_started iteration.

Create the experiment as a draft. Start it only after the flag and tracking code are deployed and the user says yes again. Starting (start-experiment-iteration, with confirm: true) requires the flag to be on, a randomizationUnit, and at least one non-zero allocation.

Rules that matter: exactly one treatment has baseline: true; allocations sum to 100; the iteration object is required on create.


REST fallback

Base URL https://app.launchdarkly.com. Header: Authorization: <token>, where <token> is the API access token the user pasted (plus Content-Type: application/json, and LD-API-Version: beta for experiment endpoints).

Create a metric (skip if reusing):

POST /api/v2/metrics/{projectKey}
{
  "key": "checkout-completed",
  "name": "Checkout completed",
  "kind": "custom",
  "isNumeric": false,
  "eventKey": "checkout-completed",
  "successCriteria": "HigherThanBaseline"
}

Create the experiment (body mirrors the MCP iteration):

POST /api/v2/projects/{projectKey}/environments/{environmentKey}/experiments
{
  "key": "redesigned-checkout-experiment",
  "name": "Redesigned Checkout Experiment",
  "iteration": {
    "hypothesis": "The redesigned checkout will lift completion ~3% for logged-in shoppers.",
    "primarySingleMetricKey": "checkout-completed",
    "metrics": [{ "key": "checkout-completed" }, { "key": "checkout-error-rate" }],
    "treatments": [
      { "name": "Control", "baseline": true, "allocationPercent": 50,
        "parameters": [{ "flagKey": "new-checkout", "variationId": "<control-variation-id>" }] },
      { "name": "New Checkout", "baseline": false, "allocationPercent": 50,
        "parameters": [{ "flagKey": "new-checkout", "variationId": "<treatment-variation-id>" }] }
    ],
    "flags": { "new-checkout": { "ruleId": "fallthrough", "flagConfigVersion": 7 } },
    "randomizationUnit": "user"
  }
}

Start the iteration (only after the flag and tracking code are deployed and the user says yes again): create-experiment already created a not_started iteration — do not POST .../iterations to start it (that creates a second, now-deprecated iteration). Start the existing draft with a semantic-patch PATCH on the experiment:

PATCH /api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}
Content-Type: application/json; domain-model=launchdarkly.semanticpatch
{ "instructions": [{ "kind": "startIteration" }] }

Same preconditions as the MCP path: the flag must be on, the iteration needs a randomizationUnit, and at least one treatment must have a non-zero allocation.

Verify:

GET /api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}

CRUD reality check: metrics support full CRUD (create/read/update/delete). Experiments support create/read/update + iterations, but no delete — you manage an experiment's life through its iterations, not by deleting it.

For exact request/response schemas, see the OpenAPI spec: https://launchdarkly.com/docs/api/other/get-openapi-spec (tags: Metrics, Experiments).


No access at all

Present the draft plus the two POST calls above with the user's real values filled in, and tell them they can either run them with an API token or open the experiment in the LaunchDarkly UI. Point them at launchdarkly-experiment-setup to finish and to run the iteration lifecycle.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts15d3 checks · Risk SAFE
  • Gen Agent Trust Hub15d

    This skill is a guided assistant for setting up A/B tests and experiments using LaunchDarkly. It scans the local codebase and git history to suggest relevant experiments and interacts with LaunchDarkly via MCP or an API token to manage flags and metrics. The skill follows best practices for credential handling and user consent.

  • Socket15d

    No alerts

  • Snyk15d

    Risk: LOW · No issues

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 weeks ago
metadata
{
  "author": "launchdarkly",
  "version": "0.1.0-experimental"
}
Other metadata
compatibility
Uses the LaunchDarkly MCP server when it is configured; otherwise falls back to the LaunchDarkly REST API (needs a LaunchDarkly API token). Reads the local codebase and git history.

README badge

README badge for launchdarkly/ai-tooling/launchdarkly-experiment-guide