/eval-driven-dev

Step 5: Run pixie test and Fix Mechanical Issues

Why this step: Run pixie test and fix mechanical issues in your QA components — dataset format problems, runnable implementation bugs, and custom evaluator errors — until every entry produces real scores. This step is NOT about assessing result quality or fixing the application itself.

5a. Run tests

uv run pixie test

For verbose output with per-case scores and evaluator reasoning:

uv run pixie test -v

pixie test automatically loads the .env file before running tests.

The evaluation harness:

1. Resolves the Runnable class from the dataset's runnable field
2. Calls Runnable.create() to construct an instance, then setup() once
3. Runs all dataset entries concurrently (up to 4 in parallel): a. Reads input_data and eval_input from the entry b. Populates the wrap input registry with eval_input data c. Initialises the capture registry d. Validates input_data into the Pydantic model and calls Runnable.run(args) e. wrap(purpose="input") calls in the app return registry values instead of calling external services f. wrap(purpose="output"/"state") calls capture data for evaluation g. Builds Evaluable from captured data h. Runs evaluators
4. Calls Runnable.teardown() once

Because entries run concurrently, the Runnable's run() method must be concurrency-safe. If you see sqlite3.OperationalError, "database is locked", or similar errors, add a Semaphore(1) to your Runnable (see the concurrency section in Step 2 reference).

5b. Fix mechanical issues only

This step is strictly about fixing what you built in previous steps — the dataset, the runnable, and any custom evaluators. You are fixing mechanical problems that prevent the pipeline from running, NOT assessing or improving the application's output quality.

What counts as a mechanical issue (fix these):

What is NOT a mechanical issue (do NOT fix these here):

• Application produces wrong/low-quality output → that's the application's behavior, analyzed in Step 6
• Evaluator scores are low → that's a quality signal, analyzed in Step 6
• LLM calls fail inside the application → report in Step 6, do not mock or work around
• Evaluator scores fluctuate between runs → normal LLM non-determinism, not a bug

Iterate — fix errors, re-run, fix the next error — until pixie test runs to completion with real evaluator scores for all entries.

Output

After pixie test completes successfully, results are stored in the per-entry directory structure:

{PIXIE_ROOT}/results/<test_id>/
  meta.json                           # test run metadata
  dataset-{idx}/
    metadata.json                     # dataset name, path, runnable
    entry-{idx}/
      config.json                     # evaluators, description, expectation
      eval-input.jsonl                # input data fed to evaluators
      eval-output.jsonl               # output data captured from app
      evaluations.jsonl               # evaluation results (scored + pending)
      trace.jsonl                     # LLM call traces (if captured)

The <test_id> is printed in console output. You will reference this directory in Step 6.

If you hit an unexpected error when running tests (wrong parameter names, import failures, API mismatch), read wrap-api.md, evaluators.md, or testing-api.md for the authoritative API reference before guessing at a fix.