All skills
github avatar

/eval-driven-dev

@2860790 official
by githubgithub/awesome-copilot40k stars
5,040

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/eval-driven-dev

This session only. Nothing lands on disk.

referencesrunnable-examplesstandalone-function.md

≈453 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Runnable Example: Standalone Function (No Server)

When the app is a plain Python function or module — no web framework, no server, no infrastructure.

Approach: Import and call the function directly from run(). This is the simplest case.

# pixie_qa/run_app.py
from pydantic import BaseModel
import pixie


class AppArgs(BaseModel):
    question: str


class AppRunnable(pixie.Runnable[AppArgs]):
    """Drives a standalone function for tracing and evaluation."""

    @classmethod
    def create(cls) -> "AppRunnable":
        return cls()

    async def run(self, args: AppArgs) -> None:
        from myapp.agent import answer_question
        await answer_question(args.question)

If the function is synchronous, wrap it with asyncio.to_thread:

import asyncio

async def run(self, args: AppArgs) -> None:
    from myapp.agent import answer_question
    await asyncio.to_thread(answer_question, args.question)

If the function depends on an external service (e.g., a vector store), the wrap(purpose="input") calls you added in Step 2a handle it automatically — the registry injects test data in eval mode.

When to use setup() / teardown()

Most standalone functions don't need lifecycle methods. Use them only when the function requires a shared resource (e.g., a pre-loaded embedding model, a database connection):

class AppRunnable(pixie.Runnable[AppArgs]):
    _model: SomeModel

    @classmethod
    def create(cls) -> "AppRunnable":
        return cls()

    async def setup(self) -> None:
        from myapp.models import load_model
        self._model = load_model()

    async def run(self, args: AppArgs) -> None:
        from myapp.agent import answer_question
        await answer_question(args.question, model=self._model)

Source: SKILL.md on GitHub

3 warnings16d5 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    This skill facilitates evaluation-driven development for Python AI applications using the pixie-qa framework. It automates the setup of an evaluation pipeline, including package installation, instrumentation, and result analysis. Security considerations include the installation of third-party packages, a self-updating mechanism, and the processing of potentially untrusted project specifications.

  • Socket16d

    1 alert: gptAnomaly

  • Snyk16d

    Risk: MEDIUM · 1 issue

  • Runlayer6mo

    1/2 files flagged

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 2860790. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 5 months ago
compatibility
Python 3.10+
Other metadata
metadata
{
  "version": "0.8.4",
  "pixie-qa-version": ">=0.8.4,<0.9.0",
  "pixie-qa-source": "https://github.com/yiouli/pixie-qa/"
}
  • Python
  • Testing
  • llm
  • evaluation
  • pixie-qa
  • quality-assurance
  • benchmarking
  • agent
  • observability

README badge

README badge for github/awesome-copilot/eval-driven-dev

Defines evaluation criteria, instruments a Python LLM application to inject controlled test data, builds datasets, and runs automated quality checks via the pixie-qa framework. Targets Python projects that call LLMs and need end-to-end testing of application code (routing, prompt assembly, response formatting) with real LLM calls — not mocking — scored by evaluators instead of deterministic assertions.

Generated from the current SKILL.md.

Does this skill work with any Python LLM application or only specific frameworks?
This skill works with any Python application that calls an LLM, regardless of framework. It instruments the app's data boundaries and runs the full application code end-to-end, exercising real routing, prompt assembly, and LLM calls.
Can I mock or stub the LLM during evaluation?
No. The skill requires real LLM calls — mocking or stubbing the LLM makes eval scores meaningless because you control both inputs and outputs. The app's own unit tests may mock the LLM; evals must not.
What is pixie-qa and do I need to install it separately?
pixie-qa is the Python package that powers the eval pipeline. The skill's setup.sh installs it automatically. If installation fails, the workflow cannot proceed and you must ask for help.
Does this skill create evals from scratch or does it assume I already have test data?
The skill builds evals from scratch. It guides you through defining eval criteria, instrumenting the app, capturing reference traces, defining evaluators, and building a golden dataset. If your prompt specifies a dataset or evaluation spec, the skill will use that instead.
What Python versions does this support?
Python 3.10 and later. The skill also requires pixie-qa version 0.8.4 or later.

Generated from the current SKILL.md. These answers refresh after source changes.