skilld
All skills
github avatar

/phoenix-evals

@4214189 official
by githubgithub/awesome-copilot40k stars
5,040

Build and run evaluators for AI/LLM applications using Phoenix.

Ask your Agent

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/phoenix-evals

This session only. Nothing lands on disk.

referencesvalidation.md

≈441 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Validation

Validate LLM judges against human labels before deploying. Target >80% agreement.

Requirements

Requirement Target
Test set size 100+ examples
Balance ~50/50 pass/fail
Accuracy >80%
TPR/TNR Both >70%

Metrics

Metric Formula Use When
Accuracy (TP+TN) / Total General
TPR (Recall) TP / (TP+FN) Quality assurance
TNR (Specificity) TN / (TN+FP) Safety-critical
Cohen's Kappa Agreement beyond chance Comparing evaluators

Quick Validation

from sklearn.metrics import classification_report, confusion_matrix, cohen_kappa_score

print(classification_report(human_labels, evaluator_predictions))
print(f"Kappa: {cohen_kappa_score(human_labels, evaluator_predictions):.3f}")

# Get TPR/TNR
cm = confusion_matrix(human_labels, evaluator_predictions)
tn, fp, fn, tp = cm.ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)

Golden Dataset Structure

golden_example = {
    "input": "What is the capital of France?",
    "output": "Paris is the capital.",
    "ground_truth_label": "correct",
}

Building Golden Datasets

  1. Sample production traces (errors, negative feedback, edge cases)
  2. Balance ~50/50 pass/fail
  3. Expert labels each example
  4. Version datasets (never modify existing)
# GOOD - create new version
golden_v2 = golden_v1 + [new_examples]

# BAD - never modify existing
golden_v1.append(new_example)

Warning Signs

  • All pass or all fail → too lenient/strict
  • Random results → criteria unclear
  • TPR/TNR < 70% → needs improvement

Re-Validate When

  • Prompt template changes
  • Judge model changes
  • Criteria changes
  • Monthly

Source: SKILL.md on GitHub

Third-party checks

No alerts9d4 checks · Risk SAFE
  • Gen Agent Trust Hub9d

    This skill provides comprehensive documentation and examples for building AI evaluators using Arize Phoenix. No malicious patterns were detected. It follows security best practices by recommending XML delimiters for LLM prompts and uses standard package management for its dependencies.

  • Socket9d

    No alerts

  • Snyk9d

    Risk: LOW · No issues

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 4214189. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 19 hours ago.

Activeupdated 2 months ago

Capability

Other metadata
compatibility
Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
metadata
{
  "author": "oss@arize.com",
  "version": "1.0.0",
  "languages": "Python, TypeScript"
}

Topics

README badge

README badge for github/awesome-copilot/phoenix-evals

What it does

Builds and runs evaluators for LLM applications using Phoenix, supporting code-based checks, LLM-as-judge approaches, and human validation workflows. Includes pre-built evaluators for RAG systems, error analysis, experiment tracking, and production monitoring across Python and TypeScript.

Generated from the current SKILL.md.

Frequently asked

Does this skill work with Python and TypeScript?
Yes. Python skills require the phoenix and openai packages; TypeScript skills require @arizeai/phoenix-client. Both require a running Phoenix server.
Can I build custom evaluators or only use pre-built ones?
You can build both code-based evaluators (deterministic logic) and LLM-based evaluators (using an LLM as a judge), with templates and validation support for both.
Does this cover RAG system evaluation?
Yes. The skill includes a dedicated RAG evaluators workflow covering retrieval quality and answer faithfulness.
Can I validate that my evaluators are accurate?
Yes. The skill provides validation references to test your evaluators' accuracy (TPR/TNR targets) against human labels.
What should I do before building evaluators?
The skill recommends starting with error analysis and tracing to observe actual failures, then categorizing them before automating evaluation.

Generated from the current SKILL.md. These answers refresh after source changes.