Index of every page on this site: llms.txt
All skills
github avatar

/phoenix-evals

@f10fdb8 official
by githubgithub/awesome-copilot41k stars
5,399
Run it yourself

Build and run evaluators for AI/LLM applications using Phoenix.

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/phoenix-evals

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈20 tokens for metadata: the name and description. ≈1.1k when used: this file. ≈22k more on demand in 34 files.

How this Skill uses context

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈79 characters, or 1.0%.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task Files
Setup setup-python, setup-typescript
Decide what to evaluate evaluators-overview
Choose a judge model fundamentals-model-selection
Use pre-built evaluators evaluators-pre-built
Build code evaluator evaluators-code-python, evaluators-code-typescript
Build LLM evaluator evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame evaluate-dataframe-python
Understand experiments experiments-overview
Run experiment experiments-running-python, experiments-running-typescript
Create dataset experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review observe-sampling-python, observe-sampling-typescript
Analyze errors error-analysis, error-analysis-multi-turn, axial-coding
RAG evals evaluators-rag
Avoid common mistakes common-mistakes-python, fundamentals-anti-patterns
Production production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix Description
fundamentals-* Types, scores, anti-patterns
observe-* Tracing, sampling
error-analysis-* Finding failures
axial-coding-* Categorizing failures
evaluators-* Code, LLM, RAG evaluators
experiments-* Datasets, running experiments
validation-* Validating evaluator accuracy against human labels
production-* CI/CD, monitoring

Key Principles

Principle Action
Error analysis first Can't automate what you haven't observed
Custom > generic Build from your failures
Code first Deterministic before LLM
Validate judges >80% TPR/TNR
Binary > Likert Pass/fail, not 1-5

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts23d4 checks · Risk SAFE
  • Gen Agent Trust Hub23d

    This skill provides comprehensive documentation and code examples for using Phoenix to evaluate AI and LLM applications. It covers tracing, error analysis, and the creation of code-based and LLM-based evaluators. The skill follows security best practices by recommending the use of XML boundary markers to protect against indirect prompt injection during the evaluation of production data.

  • Socket23d

    No alerts

  • Snyk23d

    Risk: LOW · No issues

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at f10fdb8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago
Other metadata
compatibility
Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
metadata
{
  "author": "oss@arize.com",
  "version": "1.0.0",
  "languages": "Python, TypeScript"
}

README badge

README badge for github/awesome-copilot/phoenix-evals

Builds and runs evaluators for LLM applications using Phoenix, supporting code-based checks, LLM-as-judge approaches, and human validation workflows. Includes pre-built evaluators for RAG systems, error analysis, experiment tracking, and production monitoring across Python and TypeScript.

Generated from the current SKILL.md.

Does this skill work with Python and TypeScript?
Yes. Python skills require the phoenix and openai packages; TypeScript skills require @arizeai/phoenix-client. Both require a running Phoenix server.
Can I build custom evaluators or only use pre-built ones?
You can build both code-based evaluators (deterministic logic) and LLM-based evaluators (using an LLM as a judge), with templates and validation support for both.
Does this cover RAG system evaluation?
Yes. The skill includes a dedicated RAG evaluators workflow covering retrieval quality and answer faithfulness.
Can I validate that my evaluators are accurate?
Yes. The skill provides validation references to test your evaluators' accuracy (TPR/TNR targets) against human labels.
What should I do before building evaluators?
The skill recommends starting with error analysis and tracing to observe actual failures, then categorizing them before automating evaluation.

Generated from the current SKILL.md. These answers refresh after source changes.