All skills
github avatar

/eval-driven-dev

@2860790 official
by githubgithub/awesome-copilot40k stars
5,046

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/eval-driven-dev

This session only. Nothing lands on disk.

referencesrunnable-examplescli-app.md

≈506 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Runnable Example: CLI Application

When the app is invoked from the command line (e.g., python -m myapp, a CLI tool with argparse/click).

Approach: Use asyncio.create_subprocess_exec to invoke the CLI and capture output.

# pixie_qa/run_app.py
import asyncio
import sys

from pydantic import BaseModel
import pixie


class AppArgs(BaseModel):
    query: str


class AppRunnable(pixie.Runnable[AppArgs]):
    """Drives a CLI application via subprocess."""

    @classmethod
    def create(cls) -> "AppRunnable":
        return cls()

    async def run(self, args: AppArgs) -> None:
        proc = await asyncio.create_subprocess_exec(
            sys.executable, "-m", "myapp", "--query", args.query,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )
        stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=120)
        if proc.returncode != 0:
            raise RuntimeError(f"App failed (exit {proc.returncode}): {stderr.decode()}")

When the CLI needs patched dependencies

If the CLI reads from external services, create a wrapper entry point that patches dependencies before running the real CLI:

# pixie_qa/patched_app.py
"""Entry point that patches external deps before running the real CLI."""
import myapp.config as config
config.redis_url = "mock://localhost"

from myapp.main import main
main()

Then point your Runnable at the wrapper:

async def run(self, args: AppArgs) -> None:
    proc = await asyncio.create_subprocess_exec(
        sys.executable, "-m", "pixie_qa.patched_app", "--query", args.query,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=120)

Note: For CLI apps, wrap(purpose="input") injection only works when the app runs in the same process. If using subprocess, you may need to pass test data via environment variables or config files instead.

Source: SKILL.md on GitHub

3 warnings17d5 checks · Risk SAFE
  • Gen Agent Trust Hub17d

    This skill facilitates evaluation-driven development for Python AI applications using the pixie-qa framework. It automates the setup of an evaluation pipeline, including package installation, instrumentation, and result analysis. Security considerations include the installation of third-party packages, a self-updating mechanism, and the processing of potentially untrusted project specifications.

  • Socket17d

    1 alert: gptAnomaly

  • Snyk17d

    Risk: MEDIUM · 1 issue

  • Runlayer6mo

    1/2 files flagged

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 2860790. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 13 hours ago.

Activeupdated 5 months ago
compatibility
Python 3.10+
Other metadata
metadata
{
  "version": "0.8.4",
  "pixie-qa-version": ">=0.8.4,<0.9.0",
  "pixie-qa-source": "https://github.com/yiouli/pixie-qa/"
}
  • Python
  • Testing
  • llm
  • evaluation
  • pixie-qa
  • quality-assurance
  • benchmarking
  • agent
  • observability

README badge

README badge for github/awesome-copilot/eval-driven-dev

Defines evaluation criteria, instruments a Python LLM application to inject controlled test data, builds datasets, and runs automated quality checks via the pixie-qa framework. Targets Python projects that call LLMs and need end-to-end testing of application code (routing, prompt assembly, response formatting) with real LLM calls — not mocking — scored by evaluators instead of deterministic assertions.

Generated from the current SKILL.md.

Does this skill work with any Python LLM application or only specific frameworks?
This skill works with any Python application that calls an LLM, regardless of framework. It instruments the app's data boundaries and runs the full application code end-to-end, exercising real routing, prompt assembly, and LLM calls.
Can I mock or stub the LLM during evaluation?
No. The skill requires real LLM calls — mocking or stubbing the LLM makes eval scores meaningless because you control both inputs and outputs. The app's own unit tests may mock the LLM; evals must not.
What is pixie-qa and do I need to install it separately?
pixie-qa is the Python package that powers the eval pipeline. The skill's setup.sh installs it automatically. If installation fails, the workflow cannot proceed and you must ask for help.
Does this skill create evals from scratch or does it assume I already have test data?
The skill builds evals from scratch. It guides you through defining eval criteria, instrumenting the app, capturing reference traces, defining evaluators, and building a golden dataset. If your prompt specifies a dataset or evaluation spec, the skill will use that instead.
What Python versions does this support?
Python 3.10 and later. The skill also requires pixie-qa version 0.8.4 or later.

Generated from the current SKILL.md. These answers refresh after source changes.