skilld
All skills
microsoft avatar

/m365-agent-evaluator

@a43d2c6
by microsoftmicrosoft/skills3.1k stars
351

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Ask your Agent

Use this Skill: https://skilld.dev/gh/microsoft/skills/m365-agent-evaluator

This session only. Nothing lands on disk.

examplesiterate-on-changes.md

≈326 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Example: iterate after agent changes

User intent: "I changed my agent instructions. Re-run the evals and compare."

Baseline run

npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\baseline.json

After-change run

npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\after-instructions.json

Keep the dataset, evaluator thresholds, model deployment, and concurrency stable when comparing. If the user intentionally changed the dataset, report that the comparison is not a strict regression comparison.

Compare

  1. Compare items by prompt or conversation name.
  2. Compare only score keys that exist in both runs.
  3. Look for improvements and regressions by evaluator theme.
  4. If a setup/auth/model error appears in only one run, do not call it an agent regression.

Example summary

The instruction change improved grounding on the project-status prompt from fail to pass, but the action-item prompt still fails citations. The next targeted change should require source citations when listing owners, or the eval should be relaxed if the agent cannot expose citations for that data path.

Source: SKILL.md on GitHub

Third-party checks

1 warning3mo3 checks · Risk SAFE
  • Gen Agent Trust Hub3mo

    This skill facilitates the evaluation of Microsoft 365 Copilot declarative agents using official Microsoft tooling. It includes robust security guardrails, such as guidance on managing environment secrets and avoiding the exposure of sensitive data in logs or chat. The skill's operations are aligned with its intended purpose and follow established development best practices.

  • Socket3mo

    No alerts

  • Snyk3mo

    Risk: MEDIUM · 1 issue

Provenance

Signed by skilld at a43d2c6. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago

README badge

README badge for microsoft/skills/m365-agent-evaluator