skilld
All skills
github avatar

/arize-prompt-optimization

@4e136f3 official
by githubgithub/awesome-copilot40k stars
5,040

Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.

Ask your Agent

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/arize-prompt-optimization

This session only. Nothing lands on disk.

referencesax-setup.md

≈383 tokens on demand. Your agent reads this file only when SKILL.md points to it.

ax CLI — Troubleshooting

Consult this only when an ax command fails. Do NOT run these checks proactively.

Check version first

If ax is installed (not command not found), always run ax --version before investigating further. The version must be 0.14.0 or higher — many errors are caused by an outdated install. If the version is too old, see Version too old below.

ax: command not found

macOS/Linux:

  1. Check common locations: ~/.local/bin/ax, ~/Library/Python/*/bin/ax
  2. Install: uv tool install arize-ax-cli (preferred), pipx install arize-ax-cli, or pip install arize-ax-cli
  3. Add to PATH if needed: export PATH="$HOME/.local/bin:$PATH"

Windows (PowerShell):

  1. Check: Get-Command ax or where.exe ax
  2. Common locations: %APPDATA%\Python\Scripts\ax.exe, %LOCALAPPDATA%\Programs\Python\Python*\Scripts\ax.exe
  3. Install: pip install arize-ax-cli
  4. Add to PATH: $env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"

Version too old (below 0.14.0)

Upgrade: uv tool install --force --reinstall arize-ax-cli, pipx upgrade arize-ax-cli, or pip install --upgrade arize-ax-cli

SSL/certificate error

  • macOS: export SSL_CERT_FILE=/etc/ssl/cert.pem
  • Linux: export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
  • Fallback: export SSL_CERT_FILE=$(python -c "import certifi; print(certifi.where())")

Subcommand not recognized

Upgrade ax (see above) or use the closest available alternative.

Still failing

Stop and ask the user for help.

Source: SKILL.md on GitHub

Third-party checks

1 warning16d4 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    The skill is generally safe and implements robust security best practices regarding credential handling, explicitly forbidding the reading of .env files or exposing API keys. It contains a standard surface for indirect prompt injection because it processes external trace logs and data records for optimization purposes, which is addressed using clear structural delimiters.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: MEDIUM · 1 issue

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 4e136f3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 5 months ago

Capability

metadata
{
  "author": "arize",
  "version": "1.0"
}
compatibility
Requires the ax CLI and a configured Arize profile.

Topics

  • prompt-optimization
  • llm
  • arize
  • evaluations
  • production-data
  • ax-cli
  • prompt-engineering

README badge

README badge for github/awesome-copilot/arize-prompt-optimization

What it does

Extracts prompts from production LLM traces using Arize, gathers performance signals from annotations and evals, and generates optimized prompt versions using a structured meta-prompt. Targets teams using the ax CLI with Arize projects who have traced LLM spans and human or automated feedback on output quality.

Generated from the current SKILL.md.

Frequently asked

What does this skill require to run?
The ax CLI and a configured Arize profile with API credentials. If the ax command fails with a 401 error, run `ax profiles show` to check your profile configuration or visit https://app.arize.com/admin to retrieve your API key.
Where does this skill look for prompts in my traces?
It extracts prompts from `attributes.llm.input_messages` for structured chat messages, `attributes.llm.prompt_template.template` for templated prompts, or `attributes.input.value` as a fallback. The skill targets LLM spans specifically and navigates child spans in chains or agents.
What kind of performance data does this skill use to optimize?
It uses human annotations (labels, scores, text feedback), LLM-as-judge evaluations (especially `eval.*.explanation`), and comparisons of actual outputs against ground truth. The skill merges dataset examples with experiment runs to identify failure patterns.
Does this skill modify my prompts automatically?
No. The skill guides you through extracting the current prompt, gathering performance data, and applying an optimization meta-prompt template. You send the analysis to an LLM (GPT-4o, Claude, etc.) to generate the revised prompt, then decide whether to deploy it.
Can I use this with template variables like {input} or {question}?
Yes. The skill preserves template placeholders during extraction and instructs the optimization step to maintain variable syntax so prompts remain dynamic at runtime.

Generated from the current SKILL.md. These answers refresh after source changes.