Index of every page on this site: llms.txt
All skills
launchdarkly avatar

/migrate

@f0ee110 official
Run it yourself

Migrate an application with hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation on the current AI SDKs in five stages: audit the code, wrap the call, move the tools, verify metrics, attach evaluators. Use when the user wants to externalize model/prompt configuration, move from direct provider calls (OpenAI, Anthropic, Bedrock, Gemini, Strands) to a managed config, or stage a full hardcoded-to-LaunchDarkly migration. If the app already uses a legacy LaunchDarkly AI SDK (`launchdarkly-server-sdk-ai` / `ldai`, `@launchdarkly/server-sdk-ai`), use `legacy-sdk-migrate` instead.

Use this Skill: https://skilld.dev/gh/launchdarkly/ai-tooling/migrate

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈152 tokens for metadata: the name and description. ≈9.3k when used: this file. ≈16k more on demand in 7 files.

Description uses 7.6% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈609 characters, or 7.6%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Migrate to AgentControl

You're using a skill that will guide you through migrating an application from hardcoded LLM prompts to a full LaunchDarkly AgentControl implementation. Your job is to run the migration in five stages, stopping at each stage for the user to confirm:

  1. Audit the code — read-only scan that produces a structured list of everything hardcoded (prompt, model, parameters, tools, app-scoped knobs).
  2. Wrap the call — install the SDK, create the config in LaunchDarkly, and route the provider call through config().invoke(), keeping the original call as the fallback.
  3. Move the tools — extract each tool's JSON schema, attach it to the config, and pass the implementations to the SDK as tool handlers.
  4. Verify metrics — confirm duration, tokens, and success/error reach the Monitoring tab, and add feedback or business metrics if the app needs them.
  5. Attach evaluators — either offline evals via the Playground + Datasets, or judges that score sampled traffic automatically.

⚠️ Three first-run failure modes to avoid.

  1. Placeholders with spaces. The SDK fills {{name}} and {{user.name}} only. {{ name }} with spaces is sent to the model as literal text. Rewrite every placeholder without spaces in Stage 2.
  2. No fallback. invoke() has no fallback argument. It raises RuntimeError (a plain Error in Node.js) when the variation is disabled, the config key doesn't exist, or LaunchDarkly is unreachable. Keep the original hardcoded call and run it from an except RuntimeError branch after checking if (await inspect_config(key, context))["enabled"]: raise (so mid-run handler errors still propagate), or the app goes down with LaunchDarkly.
  3. Fallthrough not flipped after /configs-create. A freshly created config's fallthrough points at an auto-generated disabled variation, so every invoke() raises until /configs-targeting runs. Flip it before Stage 2 verification.

How the current SDK changes the migration

The SDK now owns the provider call. You evaluate a config and invoke it in one step, and a handler calls the provider. Metrics are recorded on every call, so there's no tracker to create or scope. What that means for each stage:

Concern Current SDK
Provider call A handler package (OpenAI, Anthropic, LangChain) or a custom handler that wraps your existing call
Fallback Your own code, run when invoke() raises RuntimeError and inspect_config() reports enabled=False
Template variables Third argument to invoke(); {{name}} placeholders, no spaces
Tools tool_handlers={"tool-key": fn} on config(); the handler runs the tool loop
Metrics Automatic: duration, tokens, success/error, tool calls
Judges Attached judges run inside invoke(); skip_judges=True + run_judge() for background work
Multi-agent graph(), or resolve_graph() + a native runner such as to_lang_graph

If the app already uses the legacy LaunchDarkly AI SDK (ldai, @launchdarkly/server-sdk-ai, completion_config(), create_tracker()) and its configs already exist in LaunchDarkly, use the legacy-sdk-migrate skill instead — it upgrades between the LaunchDarkly AI SDKs without creating new configs.

Coverage — which shapes are well-trodden vs require extrapolation

The skill is optimized for Python and Node.js / TypeScript. The current AI SDKs exist for Python and Node.js only.

Shape Python Node.js Reference
One-shot completion (direct OpenAI / Anthropic call) ✅ Worked example ✅ Worked example before-after-examples.md
Bedrock / Gemini / custom HTTP direct call ✅ Custom handler ✅ Custom handler built-in-metrics per-provider references
LangChain single-call ✅ Handler package ✅ Handler package agent-mode-frameworks.md § LangChain
LangGraph prebuilt agent (Python create_agent, Node createReactAgent) ✅ Worked example ✅ Worked example before-after-examples.md, agent-mode-frameworks.md § LangGraph prebuilt
LangGraph custom StateGraph ✅ Custom agent handler ⚠️ Translate from Python agent-mode-frameworks.md § Custom StateGraph
CrewAI ✅ Custom agent handler — (not a Node framework) agent-mode-frameworks.md § CrewAI
Strands ✅ Custom agent handler ⚠️ BedrockModel + OpenAIModel only agent-mode-frameworks.md § Strands
Custom ReAct loop ✅ Replace with SDK tool loop ✅ Same agent-mode-frameworks.md § Custom ReAct loop
Streaming (SSE / WebSocket) ✅ stream() ✅ stream() built-in-metrics streaming reference
Multi-agent graph (supervisor + workers) ⚠️ Out of main scope ⚠️ Out of main scope agent-graph-reference.md
Other agent frameworks (Pydantic AI, DSPy, AutoGen, LlamaIndex) ⚠️ Custom agent handler ⚠️ Same agent-mode-frameworks.md § Framework-agnostic rules
Go, Ruby, .NET ℹ️ Legacy SDKs only ℹ️ Legacy SDKs only phase-1-analysis-checklist.md § SDK routing table

Reading the key: ✅ = follow the skill verbatim; ⚠️ = the architecture applies but you'll translate idioms or cross-reference another skill; ℹ️ = this skill doesn't cover it.

Prerequisites

This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment, and an application that already calls an LLM provider with hardcoded model, prompt, and parameter values.

Required environment:

  • LD_SDK_KEY — server-side SDK key (starts with sdk-) from the target LaunchDarkly project. The SDK reads this variable, not LAUNCHDARKLY_SDK_KEY.

MCP tools used directly by this skill: none — every LaunchDarkly write happens in a focused sibling skill.

Check the SDK CHANGELOG before applying any pattern. Before you start, skim the latest CHANGELOG for anything that contradicts the pattern you're about to apply:

If a CHANGELOG entry post-dates this skill and changes an API you're about to use, the CHANGELOG wins — and the skill should be updated.

Hand-off model. This skill does not auto-invoke other skills. At each stage that needs a LaunchDarkly write, this skill prepares the inputs (config key, mode, model, prompt, tool schemas, judge keys) and then tells the user to run the next slash-command themselves. After the user finishes that sibling skill, return to the next step here.

Sibling skills the user runs at each stage:

  • projects — pre-Stage 2, only if no project exists yet
  • configs-create — Stage 2 (creates the config and first variation)
  • configs-targeting — right after configs-create (promotes the new variation to fallthrough so the SDK actually serves it)
  • tools — Stage 3 (creates tool definitions and attaches them)
  • online-evals — Stage 5 (attaches judges, creates custom judges)

Core Principles

  1. Inspect before you mutate. Every stage begins with a read-only audit. Do not touch code until Step 1 is confirmed by the user.
  2. Move the source of config, not the business logic. The request handler's logic and return shape stay the same. What changes is where the model, prompt, and parameters come from, and that the provider call runs inside a handler. The call site does change shape: it becomes async and needs a context source such as the user ID. Update every caller, or keep a sync wrapper that calls asyncio.run() for scripts.
  3. The fallback is the current behavior. Keep the original hardcoded call as a function and run it when invoke() raises and inspect_config() reports enabled=False, so the app is unchanged if LaunchDarkly is unreachable.
  4. Stages are ordered. Wrap before you add tools. Add tools before you verify metrics. Verify metrics before you add evals.
  5. Hand off to focused skills, manually. Each stage that needs a LaunchDarkly write tells the user to run a sibling slash-command and waits for them to come back.

Workflow

Minimum viable migration

Stages 1–4 are independently shippable. A migration that stops after Stage 4 is complete, production-ready, and delivers the core value — externalized prompts and model config, targeting, variation A/B testing, and Monitoring-tab metrics. Stage 5 (evaluators) is a quality-of-life addition, not a gate.

Step 1: Audit the codebase (Stage 1)

This stage is read-only — no code writes, no LaunchDarkly resources created. Scan the repo and produce a structured manifest of every hardcoded value that needs to move, then hand it back to the user for confirmation.

Use phase-1-analysis-checklist.md to scan:

  1. Language and package manager — Python (pip/poetry/uv) or TypeScript/JavaScript (npm/pnpm/yarn)
  2. LLM provider and framework — OpenAI, Anthropic, Bedrock, Gemini, LangChain, LangGraph, CrewAI, Strands
  3. Existing LaunchDarkly usage — an LDClient / ldclient initialization to reuse, and any legacy AI SDK usage (route legacy AI SDK upgrades to legacy-sdk-migrate)
  4. Hardcoded model configs — model name string literals, temperature / max_tokens / top_p, system prompts, instruction strings
  5. Template placeholders in prompts — .format() calls, f-strings, JS/TS template literals, %(var)s, hand-rolled str.replace("__VAR__", ...), and existing Mustache with spaces ({{ var }}). Flag each placeholder name and its runtime-value source; all get rewritten to {{var}} in Stage 2.
  6. Externalized prompt files — YAML / JSON / TOML / Markdown / .prompt / .j2 files and prompt-template registries (langchain.hub.pull(...), LangSmith client.pull_prompt(...)). See phase-1-analysis-checklist.md § 4.
  7. Hardcoded app-scoped knobs — search-result limits, retry budgets, tool-timeout overrides, feature toggles. These belong in model.custom on the variation, not model.parameters, because handlers won't pass model.parameters to your code and a custom or older handler would send it to the provider.
  8. Mode decision — completion mode (messages array) or agent mode (single instructions string). The variation's mode picks the handler at runtime.

For each hardcoded target, record:

  • File path and line range
  • Current value (model name, full prompt text, parameter dict)
  • Target config field (model.name, model.parameters.temperature, messages[].content, instructions)
  • Whether the surrounding call uses function calling / tools (drives Stage 3)
  • Whether the surrounding call has retry logic (stays in the handler, around the provider call)

Stage 1 output (return to user as a structured summary):

Language: Python 3.12
Package manager: uv
LLM provider: OpenAI
Existing LD SDK: none
Target mode: completion
Hardcoded targets:
  - src/chat.py:42   model="gpt-4o"
  - src/chat.py:43   temperature=0.7, max_tokens=2000
  - src/chat.py:45   system="You are a helpful assistant..."
Externalized prompt files: none
Prompt-template registries: none
Coverage totals: 3 hardcoded code targets · 0 externalized prompt files · 0 registry pulls
Proposed plan: single config key `chat-assistant`, OpenAI handler package, original call kept as fallback, Stage 3 (tools) skipped (no function calling), Stage 5 attach built-in accuracy judge.

STOP. Present this summary, state the coverage totals out loud (e.g. "I found N hardcoded code targets and M externalized prompt files — does that match what you expected?"), and wait for the user to reply with one of four explicit forms:

  • confirm — proceed to Stage 2.
  • add: <files or paths> — re-run the audit with the new locations and present an updated summary.
  • fix: <correction> — update a target in the list (provider, mode, prompt content, etc.) and ask again.
  • stop — pause the migration here.

Do not interpret any other word — including skip, next, go, ok, proceed — as confirmation; ask the user to pick one of the four forms. This is the most important checkpoint in the workflow.

Step 2: Wrap the call in the AI SDK (Stage 2)

This is the first stage that writes code. It has eight sub-steps.

  1. Delete hand-rolled model wrappers the audit flagged. load_chat_model(f"{provider}/{name}") and similar one-argument init_chat_model(...) wrappers (shipped with langchain-ai/react-agent and its derivatives) drop every variation parameter. Delete the function and its module; the LangChain handler package replaces it. Hand-rolled resolve_tools / TOOL_REGISTRY helpers get replaced by tool_handlers in Stage 3.

  2. Install the AI SDK. Detect the package manager from Step 1, then install the core package plus one handler package per provider and mode:

    • Python: "launchdarkly-ai-server[ld]" (the [ld] extra installs launchdarkly-server-sdk) + launchdarkly-ai-openai-messages, -openai-agents, -claude-messages, -claude-agents, -langchain-messages, or -langchain-agents
    • Node.js/TypeScript: @launchdarkly/ai-node + @launchdarkly/ai-openai-messages, -openai-agents, -claude-messages, -claude-agents, -langchain-messages, or -langchain-agents
    • Bedrock, Gemini, and custom HTTP: no handler package; use the LangChain handler or a custom handler (see built-in-metrics)
  3. Initialize the client. The SDK initializes lazily from LD_SDK_KEY on the first call, so a plain app needs no init code. Call init_client() / initClient() at startup only to fail fast or pass options. If the app already has an LDClient for feature flags, hand it to the SDK instead of creating a second one:

    from launchdarkly_ai_server import init_client, shutdown
    
    await init_client(client=ldclient.get())   # reuse the app's existing client
    # ... on process exit:
    await shutdown()                             # flushes events, then closes the client
    import { initClient, shutdown } from '@launchdarkly/ai-node';
    
    await initClient(ldClient); // reuse the app's existing client
    // ... on process exit:
    await shutdown();

    Without a key, init_client() and every invoke() raise RuntimeError (a plain Error in Node.js). Don't let that crash startup: log a warning and let the fallback in sub-step 6 handle requests.

  4. Hand off to configs-create, then configs-targeting. Print the extracted model, prompt/instructions, parameters, and mode from the Stage 1 manifest, then tell the user: "Run /configs-create with these inputs, then /configs-targeting to serve the new variation, then come back here." Supply the config key the code will call (e.g. chat-assistant). Wait for the user to finish.

  5. Rewrite template placeholders. If the hardcoded prompt interpolates runtime values with .format(), f-strings, JS template literals, or any other syntax ({system_time}, ${userName}, %(topic)s, {{ topic }}), rewrite every placeholder to {{name}} with no spaces, in the prompt you send to /configs-create. Pass the values as the third argument to invoke():

    Before:

    SYSTEM_PROMPT = "You are a helpful assistant. The time is {system_time}."
    prompt = SYSTEM_PROMPT.format(system_time=datetime.now().isoformat())

    After (prompt in the config): You are a helpful assistant. The time is {{system_time}}.

    result = await chat.invoke(user_input, context, {"system_time": datetime.now().isoformat()})

    The renderer supports dot paths ({{user.name}}) and leaves unknown or spaced placeholders as literal text. Every call also gets the evaluation context as {{ldContext.key}}, {{ldContext.name}}, and so on. Prompts written for the legacy SDK's {{ldctx.*}} need renaming.

  6. Keep the original call as the fallback. Rename the existing hardcoded function (e.g. answer → answer_hardcoded) and leave it untouched. It is the fallback, and it already mirrors current behavior exactly. See fallback-defaults-pattern.md for the variants.

  7. Route the call site through config().invoke(). Build the config instance once at module scope, and invoke it per request with the user's context:

    Python — before:

    def answer(user_input: str) -> str:
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            temperature=0.7,
            max_tokens=2000,
            messages=[
                {"role": "system", "content": "You are a helpful assistant..."},
                {"role": "user", "content": user_input},
            ],
        )
        return response.choices[0].message.content

    Python — after:

    from launchdarkly_ai_server import config, inspect_config
    from launchdarkly_ai_openai_messages import create_openai_messages_handler
    
    chat = config(key="chat-assistant", handler=create_openai_messages_handler())
    
    async def answer(user_id: str, user_input: str) -> str:
        context = {"kind": "user", "key": user_id}
        try:
            result = await chat.invoke(user_input, context)
            return result.response
        except RuntimeError:
            if (await inspect_config("chat-assistant", context))["enabled"]:
                raise
            logger.warning("chat-assistant unavailable; using hardcoded fallback")
            return answer_hardcoded(user_input)

    The context is a plain dict ({"kind": "user", "key": ..., "name": ...}), not an ldclient Context. Reuse the attributes the app already uses for feature-flag targeting.

    Note on model parameters. Before 0.5.0, openai-messages forwarded no parameters and claude-messages forwarded only max_tokens. From 0.5.0+ (including 1.0.0), every handler forwards its own fixed list of parameters (mapping max_tokens to max_output_tokens on OpenAI) and drops other keys.

    If the provider has no handler package, or the call has logic you must keep (retries, prompt caching, a proxy, Azure OpenAI), wrap the existing call in a custom handler instead. The handler receives the served config as a dict, so the hardcoded literals become reads from it:

    from openai import AsyncOpenAI
    from launchdarkly_ai_server import config, create_handler, parse_template
    
    openai_client = AsyncOpenAI()
    
    async def call_openai(ld_config, user_input, tool_handlers, variables, history):
        params = ld_config["model"].get("parameters", {})
        messages = [
            {**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", [])
        ]
        response = await openai_client.chat.completions.create(
            model=ld_config["model"]["name"],
            temperature=params.get("temperature"),
            max_tokens=params.get("max_tokens"),
            messages=[*messages, {"role": "user", "content": user_input}],
        )
        return {
            "output": response.choices[0].message.content,
            "usage": {"input": response.usage.prompt_tokens, "output": response.usage.completion_tokens},
        }
    
    chat = config(key="chat-assistant", handler=create_handler(("OpenAI", "messages"), call_openai))

    Custom handlers get the raw prompt, so render placeholders with parse_template / parseTemplate. The provider name in create_handler must match the config's provider exactly ("OpenAI", "Anthropic", "Bedrock", "Gemini").

    Agent mode — for LangGraph, CrewAI, Strands, or any framework that takes an instructions string: use an agent-mode handler package (-openai-agents, -claude-agents, -langchain-agents) or a custom handler registered for (provider, "agent") that reads ld_config["instructions"]. See agent-mode-frameworks.md.

    See before-after-examples.md for full Python OpenAI, Node.js Anthropic, and LangGraph agent-mode paired snippets.

  8. Verify. Run the app with a valid LD_SDK_KEY; confirm the call succeeds and the response matches pre-migration output. Then unset LD_SDK_KEY (or set LD_SDK_KEY=sdk-invalid) and confirm the fallback runs without error. Both paths must work before moving to Stage 3. To check what's being served without invoking, call await inspect_config(key, context) / await inspectConfig(key, context); it returns enabled, the parsed config, and the variation metadata without raising.

Delegate: configs-create and configs-targeting (sub-step 4).

Step 3: Move tools into the config (Stage 3)

Skip this step if the audited app has no function calling / tools. Otherwise:

  1. Enumerate the tools currently registered. Common shapes to look for:

    • openai.chat.completions.create(tools=[...]) — OpenAI direct
    • anthropic.messages.create(tools=[...]) — Anthropic direct
    • create_agent(llm, tools=[...], system_prompt=...) — LangGraph prebuilt (Python, langchain.agents)
    • createReactAgent({ llm, tools: [...] }) — LangGraph.js prebuilt
    • Agent(tools=[...]) — CrewAI or Strands
    • Custom StateGraph — a module-level TOOLS = [...] list referenced in both model.bind_tools(TOOLS) and ToolNode(TOOLS)

    Record each tool's name, description, and JSON schema. For LangChain @tool tools, extract the schema with tool.args_schema.model_json_schema(); for plain callables, StructuredTool.from_function(fn).args_schema.model_json_schema(). Do not hand-write the schema.

  2. Hand off to tools. Print the extracted tool names, descriptions, and schemas, then tell the user: "Run /tools with these tools and the variation key, then come back here." The sibling skill creates tool definitions (create-ai-tool) and attaches them to the variation (update-agent-config with resource: "variation"). Wait for the user to finish before proceeding to sub-step 3. Do not auto-invoke.

  3. Pass the implementations as tool handlers. Map each LaunchDarkly tool key to its function on the config() call, and delete the hardcoded tools array. The handler package converts the config's tool schemas for the provider, runs the tool loop (up to 10 model calls), and records a $ld:ai:tool_call event per call:

    chat = config(
        key="chat-assistant",
        handler=create_openai_messages_handler(),
        tool_handlers={
            "search-kb": lambda args: search_kb(args["query"]),
            "calculator": lambda args: calculate(args["expression"]),
        },
    )
    const chat = config({
      key: 'chat-assistant',
      handler: createOpenAIMessagesHandler(),
      toolHandlers: {
        'search-kb': (args) => searchKb(args.query),
        calculator: (args) => calculate(args.expression),
      },
    });

    Each function receives one argument, the parsed arguments dict, and may be sync or async. Keys must match the tool keys in LaunchDarkly exactly; a tool the model calls with no matching function raises No handler registered for tool "x". For LangChain @tool objects, wrap them: {"search-kb": lambda args: search_kb.invoke(args)}. The tools skill covers the full contract.

    For custom StateGraph shapes and custom handlers, the handler receives tracked tool functions in its tool_handlers argument. Build both the .bind_tools(...) list and the ToolNode(...) from that argument, so the model and the executor see the same tools — see agent-mode-frameworks.md § Custom StateGraph.

  4. Verify. Run the app; confirm the tool flows still execute correctly, get-agent-configs (via the delegate) confirms the tools are attached server-side, and $ld:ai:tool_call events appear for the config.

Delegate: tools (sub-step 2).

Step 4: Verify metrics (Stage 4)

Every invoke() and stream() call records duration, input/output/total tokens, generation success or error, and one event per dispatched tool call, all attributed to the served variation. There is no tracker to create. This stage confirms the data flows and adds the two things the SDK doesn't record.

  1. Check the Monitoring tab. Hit the wrapped endpoint in staging, then open the config in LaunchDarkly → Monitoring. Duration, token, and generation counts should appear within 1–2 minutes. Force an error (bad provider key) and confirm the error count increments; the SDK records the error and re-raises.

  2. If tokens are missing, the handler isn't returning usage. Custom handlers must return {"output": ..., "usage": {"input": n, "output": n}}. For streaming, enable usage on the final chunk (stream_options={"include_usage": True} for OpenAI).

  3. Wire feedback if the app has thumbs-up/down UI. The SDK doesn't record feedback. Store the result's track_data with the message and send the event when the user responds. stream() doesn't return track_data in the current SDKs, so feedback on streamed responses can't be attributed to a variation yet:

    from ldclient import Context
    from launchdarkly_ai_server import get_client
    
    event = "$ld:ai:feedback:user:positive" if thumbs_up else "$ld:ai:feedback:user:negative"
    get_client().track(event, Context.from_dict(context), result.track_data, 1)

    track_data is a plain dict, so it can be persisted and sent from a different process than the one that produced the response.

  4. Time to first token isn't recorded by the current SDKs. Don't hand-roll it.

  5. Business metrics (conversion, resolution) go through the custom-metrics skill, using the same track_data.

Delegate: none required. Point the user at built-in-metrics for per-provider details and at custom-metrics for business metrics. See sdk-ai-tracker-patterns.md for what's recorded and how legacy tracker code maps.

Step 5: Attach evaluations (Stage 5)

  1. Decide between three evaluation paths.

    Path When to use Supports agent mode?
    Offline eval (recommended default for migration) Pre-ship regression: run a fixed dataset through the new variation in the LD Playground and score against baseline. Yes — all modes
    Attached judges Attach judges to a variation; invoke() runs them on sampled requests and records the scores. Zero code changes. Yes, when attached through the API; the UI widget is completion-mode only
    Background judges Same attached judges, but skip_judges=True returns them as judge_tasks so a worker runs them off the request path. Yes

    Most migration users should start with offline eval, then attach judges once the rollout is stable.

  2. Offline eval shape for a migration. Generate a starter dataset CSV from the audit manifest (one representative input per row) and point the user at the Offline Evals guide.

    • Run the variation that mirrors the pre-migration hardcoded behavior against the dataset first — this is the baseline.
    • Clone it into a second variation pointing at a different model family. The comparison is most informative across families, not across siblings.
    • Attach the built-in Accuracy judge with a pass threshold of 0.85, and run both variations against the same dataset.
    • Promote the winner to fallthrough via /configs-targeting only if it beats the baseline on Accuracy and does not regress on Relevance or Toxicity.

    Write this shape into the project's datasets/README.md (or equivalent) so the comparison is reproducible.

  3. Hand off to online-evals for attached judges. Tell the user: "Run /online-evals with these inputs, then come back here." Pass the config key, variation key, the built-in judges or custom judge keys to attach, and the target environment. Every judge's provider needs a handler in the handler list, alongside the config's own.

  4. For background judges, pass skip_judges=True and hand each task to a worker. The online-evals skill has the full worker, including the Python Context.from_dict() conversion track() needs:

    # At module scope — the event loop holds only weak references to tasks, so keep them until they finish
    _background = set()
    
    # Inside the request handler:
    result = await config(key="chat-assistant", handler=handlers, skip_judges=True).invoke(user_input, context)
    for task in result.judge_tasks or []:
        t = asyncio.create_task(run_judge_worker(task))
        _background.add(t)
        t.add_done_callback(_background.discard)
  5. Verify.

    • Attached judges: trigger a request in staging, open the Monitoring tab → "Evaluator metrics". Scores appear within 1–2 minutes at the configured sampling rate. result.judge_results shows the scores in code.
    • Offline eval: run the dataset through the LD Playground, compare baseline vs new-variation scores side by side.

Delegate: online-evals (sub-step 3, optional — offline eval doesn't delegate).

Edge Cases

Situation Action
App already initializes LDClient for feature flags Reuse it — init_client(client=ldclient.get()) / initClient(ldClient). Do not create a second client
App is on the legacy AI SDK (ldai, @launchdarkly/server-sdk-ai) Route to legacy-sdk-migrate instead — the configs already exist in LaunchDarkly, and that skill handles the code-only SDK upgrade
App uses LangChain ChatOpenAI(model=...) Use the LangChain handler package. For providers it doesn't build itself (anything but OpenAI, Anthropic, Bedrock), pass a model factory: create_langchain_messages_handler(lambda cfg: ChatX(model=cfg["model"]["name"], **{k: v for k, v in cfg["model"].get("parameters", {}).items() if k != "tools"})). The factory gets the raw config, so drop tools
Retry wrapper around the provider call Keep it inside a custom handler, around the provider call. The SDK records one generation per invoke(), however many retries happen inside
App has no tools — Stage 3 skipped Move directly from Stage 2 verification to Stage 4
Mode mismatch: user said agent, audit shows one-shot chat Choose completion mode unless the app uses a prebuilt agent (create_agent, createReactAgent), CrewAI, Strands, or a similar goal-driven framework
App uses Strands Agents Agent mode, custom handler registered for each provider the variations serve. See agent-mode-frameworks.md § Strands
Strands app on TypeScript TS SDK ships BedrockModel and OpenAIModel only — cannot serve Anthropic-backed variations
LD_SDK_KEY is missing at startup Log a warning; the except RuntimeError fallback serves every request. Never raise at import time
Multi-agent graph (supervisor + workers) Stop after migrating a single agent. Read agent-graph-reference.md for the graph-level path
Tool with a Pydantic args_schema (LangChain @tool) Extract the schema via tool.args_schema.model_json_schema(); do not hand-write it
App-scoped knob a tool needs per variation Store it in model.custom. A custom handler reads ld_config["model"]["custom"]; with a handler package, read it once per request with await inspect_config(key, context) and close over it in the tool function you pass to tool_handlers
App has a Context dataclass with env-var fallbacks for model and prompt (e.g. react-agent's context.py) Move the values into the config and the original call path. Keep the dataclass only as the fallback's source of values, not as a second runtime source

What NOT to Do

Fallback and config lifetime (most common failure mode)

  • Don't call invoke() without a fallback. It raises RuntimeError for a disabled variation, a missing config key, a missing SDK key, or an unreachable LaunchDarkly. Because handler packages also raise RuntimeError when a run fails after starting (for example hitting the tool-loop step limit), check if (await inspect_config(key, context))["enabled"]: raise in the except RuntimeError block before running the fallback.
  • Don't delete the original hardcoded call. It's the fallback. Rename it, don't remove it.
  • Don't build a config() instance per request. Build it once at module scope; invoke() evaluates targeting fresh on every call.
  • Don't call invoke() more than once per user turn for the same logical request. Each call is a flag evaluation and a recorded generation. For an agent, one turn is one invoke() call; the handler runs the loop.

Templates

  • Don't write placeholders with spaces. {{ name }} is sent literally; write {{name}}.
  • Don't keep {{ldctx.*}} from legacy prompts; the current variable is {{ldContext.*}}.
  • Don't skip parse_template in a custom handler. The handler receives the raw prompt.

LangChain / LangGraph patterns

  • If the repo contains a load_chat_model(f"{provider}/{name}") helper, delete it — don't just avoid using it. It drops every variation parameter. The LangChain handler packages build the model with the served parameters.
  • Same rule for hand-rolled resolve_tools / TOOL_REGISTRY / ALL_TOOLS helpers. Replace them with tool_handlers.
  • Don't put app-scoped knobs in model.parameters. Handlers won't pass model.parameters to your code, and a custom or older handler would send it to the provider. Use model.custom.
  • Don't re-encode tool schemas in the fallback. The fallback is the original call, with whatever tools it had.

Stage / handoff discipline

  • Don't skip Step 1 even when the user says "just wrap it."
  • Don't delegate to configs-create before extracting the prompt and model.
  • Don't try to attach tools while creating the config with create-agent-config. Tool attachment is a separate step owned by tools.
  • Don't claim you "delegated to configs-create" or any other sibling skill. Print the inputs, tell the user to run the slash-command, then wait.
  • Don't skip /configs-targeting after /configs-create. A freshly created config serves a disabled variation, so invoke() raises and every request takes the fallback.
  • Don't attempt a multi-agent graph migration in one pass.

Metrics and evaluations

  • Don't add tracker or track() calls for duration, tokens, success, or tool calls. The SDK records them; manual events double count.
  • Don't use launchdarkly-metric-instrument for agent metrics. That skill is for feature-flag metrics.
  • Don't hardcode built-in judge config keys. Ask the user to copy them from AgentControl > Library in the LD UI.
  • Don't attach judges without a handler for the judge's provider in the handler list.

Related Skills

  • configs-create — called by Stage 2 to create the config
  • configs-targeting — called right after configs-create, and later to roll out new variations
  • tools — called by Stage 3 to create and attach tool definitions
  • built-in-metrics — per-provider handler details and the legacy tracker mapping
  • custom-metrics — business metrics on top of the built-in ones
  • online-evals — called by Stage 5 to attach judges
  • configs-variations — add variations for A/B testing after migration is complete
  • configs-update — modify config properties as your app evolves
  • legacy-sdk-migrate — upgrade an app already on the legacy LaunchDarkly AI SDK (ldai / @launchdarkly/server-sdk-ai) to the current handler-based SDKs

References

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warningtoday3 checks · Risk SAFE
  • Gen Agent Trust Hubtoday

    This skill is a migration guide for LaunchDarkly's AgentControl implementation. It provides instructions and code examples for refactoring hardcoded AI prompts to a managed configuration using official vendor SDKs. It follows security best practices for credential management and identifies common pitfalls during migration.

  • Sockettoday

    No alerts

  • Snyktoday

    Risk: MEDIUM · 1 issue

Signed by skilld at f0ee110. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 days ago
metadata
{
  "author": "launchdarkly",
  "version": "0.2.0"
}
Other metadata
compatibility
Requires the remotely hosted LaunchDarkly MCP server. Code examples use `launchdarkly-ai-server` 1.0.0+ with `launchdarkly-server-sdk` (Python, `pip install "launchdarkly-ai-server[ld]"`) or `@launchdarkly/ai-node` 1.0.0+ (Node.js).
  • Python
  • TypeScript
  • launchdarkly
  • migration
  • llm
  • prompt-management
  • agent
  • openai
  • anthropic
  • bedrock

README badge

README badge for launchdarkly/ai-tooling/migrate

Guides you through migrating a hardcoded LLM application to LaunchDarkly AgentControl in five stages: audit the code, wrap the call, move the tools, add tracking, attach evaluators. Targets Python and Node.js applications calling OpenAI, Anthropic, Bedrock, Gemini, or agent frameworks like LangGraph and CrewAI.

Generated from the current SKILL.md.

Does this skill work with my LLM provider?
The skill has worked examples for OpenAI, Anthropic, Bedrock, Gemini, and Strands. For LangChain, LangGraph, CrewAI, and custom ReAct loops, coverage varies by language — Python has deep examples; Node.js has partial coverage with ⚠️ annotations. See the coverage table in the SKILL.md for your framework.
What if my app uses a framework not listed in the coverage table?
Apply the three framework-agnostic invariants: one `agent_config` per turn, one tracker per turn, and fire at-most-once methods once at turn end. These rules apply regardless of framework; see agent-mode-frameworks.md § Framework-agnostic invariants.
Can I skip stages or do them out of order?
Stages must be ordered — audit, wrap, tools, tracking, then evaluators. You can ship after Stage 4 (tracking) is complete; Stage 5 (evaluators) is optional. Skipping ahead produces configs without traffic or metrics without context.
Does this skill handle streaming responses?
Streaming support is delegated to the built-in-metrics skill's streaming-tracking.md reference. Use `trackStreamMetricsOf` plus manual TTFT tracking.
What environment setup is required?
You need `LD_SDK_KEY` (server-side SDK key) set, the LaunchDarkly MCP server configured, and the target application already calling an LLM provider with hardcoded prompts and model values. Check the SDK CHANGELOG before starting to ensure no breaking changes post-date this skill.

Generated from the current SKILL.md. These answers refresh after source changes.