skilld
All skills
github avatar

/arize-experiment

@4e136f3 official
by githubgithub/awesome-copilot40k stars
5,040

Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.

Ask your Agent

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/arize-experiment

This session only. Nothing lands on disk.

referencesax-setup.md

≈383 tokens on demand. Your agent reads this file only when SKILL.md points to it.

ax CLI — Troubleshooting

Consult this only when an ax command fails. Do NOT run these checks proactively.

Check version first

If ax is installed (not command not found), always run ax --version before investigating further. The version must be 0.14.0 or higher — many errors are caused by an outdated install. If the version is too old, see Version too old below.

ax: command not found

macOS/Linux:

  1. Check common locations: ~/.local/bin/ax, ~/Library/Python/*/bin/ax
  2. Install: uv tool install arize-ax-cli (preferred), pipx install arize-ax-cli, or pip install arize-ax-cli
  3. Add to PATH if needed: export PATH="$HOME/.local/bin:$PATH"

Windows (PowerShell):

  1. Check: Get-Command ax or where.exe ax
  2. Common locations: %APPDATA%\Python\Scripts\ax.exe, %LOCALAPPDATA%\Programs\Python\Python*\Scripts\ax.exe
  3. Install: pip install arize-ax-cli
  4. Add to PATH: $env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"

Version too old (below 0.14.0)

Upgrade: uv tool install --force --reinstall arize-ax-cli, pipx upgrade arize-ax-cli, or pip install --upgrade arize-ax-cli

SSL/certificate error

  • macOS: export SSL_CERT_FILE=/etc/ssl/cert.pem
  • Linux: export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
  • Fallback: export SSL_CERT_FILE=$(python -c "import certifi; print(certifi.where())")

Subcommand not recognized

Upgrade ax (see above) or use the closest available alternative.

Still failing

Stop and ask the user for help.

Source: SKILL.md on GitHub

Third-party checks

No alerts16d4 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    The skill provides a comprehensive interface for managing AI experiments via the Arize 'ax' CLI. It includes templates for inference scripts and data manipulation using 'jq'. No major malicious patterns were detected; however, an indirect prompt injection surface is documented due to the skill's primary function of processing and exporting untrusted model outputs.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: LOW · No issues

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 4e136f3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 5 months ago

Capability

metadata
{
  "author": "arize",
  "version": "1.0"
}
compatibility
Requires the ax CLI and a configured Arize profile.

Topics

  • arize
  • experiments
  • model-evaluation
  • a-b-testing
  • benchmarking
  • ax-cli
  • dataset
  • metrics

README badge

README badge for github/awesome-copilot/arize-experiment

What it does

Creates, runs, and analyzes Arize experiments for evaluating model performance using the ax CLI. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows with support for datasets, evaluations, and metadata collection.

Generated from the current SKILL.md.

Frequently asked

Does this skill require the ax CLI to be installed?
Yes. The skill requires the ax CLI and a configured Arize profile. If the CLI is not found, see references/ax-setup.md.
Can I export experiments with more than 500 runs?
Yes, but you must use the `--all` flag to enable Arrow Flight for bulk export. REST export (the default) is limited to 500 runs per page.
What happens if I don't provide an API key or credentials?
The skill will not fabricate outputs. If credentials are missing or invalid, it will direct you to retrieve your API key from https://app.arize.com/admin > API Keys or use `ax profiles` to configure them.
Can I pipe dataset examples directly into a model inference script?
Yes. Use `ax datasets export DATASET_NAME --space SPACE --stdout` to pipe examples directly into a Python script or other inference tool that reads from stdin and writes runs JSON to stdout.
What file formats does the skill accept for creating experiments?
The skill accepts CSV, JSON, JSONL, and Parquet files. You can also pipe data directly via stdin using `--file -`.

Generated from the current SKILL.md. These answers refresh after source changes.