skilld
All skills
github avatar

/phoenix-evals

@4214189 official
by githubgithub/awesome-copilot40k stars
5,040

Build and run evaluators for AI/LLM applications using Phoenix.

Ask your Agent

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/phoenix-evals

This session only. Nothing lands on disk.

SKILL.md

≈20 tokens always: the name and description. ≈1.1k when used: this file. ≈22k more on demand in 34 files.

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task Files
Setup setup-python, setup-typescript
Decide what to evaluate evaluators-overview
Choose a judge model fundamentals-model-selection
Use pre-built evaluators evaluators-pre-built
Build code evaluator evaluators-code-python, evaluators-code-typescript
Build LLM evaluator evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame evaluate-dataframe-python
Understand experiments experiments-overview
Run experiment experiments-running-python, experiments-running-typescript
Create dataset experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review observe-sampling-python, observe-sampling-typescript
Analyze errors error-analysis, error-analysis-multi-turn, axial-coding
RAG evals evaluators-rag
Avoid common mistakes common-mistakes-python, fundamentals-anti-patterns
Production production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix Description
fundamentals-* Types, scores, anti-patterns
observe-* Tracing, sampling
error-analysis-* Finding failures
axial-coding-* Categorizing failures
evaluators-* Code, LLM, RAG evaluators
experiments-* Datasets, running experiments
validation-* Validating evaluator accuracy against human labels
production-* CI/CD, monitoring

Key Principles

Principle Action
Error analysis first Can't automate what you haven't observed
Custom > generic Build from your failures
Code first Deterministic before LLM
Validate judges >80% TPR/TNR
Binary > Likert Pass/fail, not 1-5

Source: SKILL.md on GitHub

Third-party checks

No alerts9d4 checks · Risk SAFE
  • Gen Agent Trust Hub9d

    This skill provides comprehensive documentation and examples for building AI evaluators using Arize Phoenix. No malicious patterns were detected. It follows security best practices by recommending XML delimiters for LLM prompts and uses standard package management for its dependencies.

  • Socket9d

    No alerts

  • Snyk9d

    Risk: LOW · No issues

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 4214189. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 18 hours ago.

Activeupdated 2 months ago

Capability

Other metadata
compatibility
Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
metadata
{
  "author": "oss@arize.com",
  "version": "1.0.0",
  "languages": "Python, TypeScript"
}

Topics

README badge

README badge for github/awesome-copilot/phoenix-evals

What it does

Builds and runs evaluators for LLM applications using Phoenix, supporting code-based checks, LLM-as-judge approaches, and human validation workflows. Includes pre-built evaluators for RAG systems, error analysis, experiment tracking, and production monitoring across Python and TypeScript.

Generated from the current SKILL.md.

Frequently asked

Does this skill work with Python and TypeScript?
Yes. Python skills require the phoenix and openai packages; TypeScript skills require @arizeai/phoenix-client. Both require a running Phoenix server.
Can I build custom evaluators or only use pre-built ones?
You can build both code-based evaluators (deterministic logic) and LLM-based evaluators (using an LLM as a judge), with templates and validation support for both.
Does this cover RAG system evaluation?
Yes. The skill includes a dedicated RAG evaluators workflow covering retrieval quality and answer faithfulness.
Can I validate that my evaluators are accurate?
Yes. The skill provides validation references to test your evaluators' accuracy (TPR/TNR targets) against human labels.
What should I do before building evaluators?
The skill recommends starting with error analysis and tracing to observe actual failures, then categorizing them before automating evaluation.

Generated from the current SKILL.md. These answers refresh after source changes.