Index of every page on this site: llms.txt
All skills
Run it yourself

Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage.

Use this Skill: https://skilld.dev/gh/homeassistant-ai/ha-mcp/bat-story-eval

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesregression-protocol.md

≈1.4k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Regression Protocol

When a story's score decreases compared to its baseline, follow this protocol to confirm whether the regression is real and identify its cause.

Overview

Decreased score detected
  |
  +- Step 1: Re-run (up to 3x)
  |   +- Still decreased? -> Continue
  |   +- Passes on retry? -> Flaky, note and move on
  |
  +- Step 2: Control run (baseline version)
  |   +- Baseline also fails? -> Not a regression, environment issue
  |   +- Baseline passes? -> Confirmed regression
  |
  +- Step 3: Cross-agent check
  |   +- Other agent also regressed? -> Code-level regression
  |   +- Other agent fine? -> Agent-specific issue
  |
  +- Step 4: Diagnose
  |   +- git diff between versions
  |   +- Identify likely cause
  |
  +- Step 5: Report

Step 1: Re-Run (Flakiness Check)

Re-run the failed story up to 3 times. Agent behavior is non-deterministic, so a single failure may be a fluke.

# Re-run specific story
uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
  --agents gemini \
  --keep-container

Decision:

  • Passes on any retry: Mark as flaky and move on. Note the pass/fail ratio.
  • Fails all 3 times: Continue to Step 2.

Step 2: Control Run (Baseline Version)

Run the same story against the baseline version to confirm the regression is real.

# Run against baseline version
uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
  --agents gemini \
  --branch v6.6.1 \
  --keep-container

Decision:

  • Baseline also fails: Not a regression - likely environment issue, flaky test, or agent model change. Note and move on.
  • Baseline passes, current fails: Confirmed regression. Continue to Step 3.

Step 3: Cross-Agent Check

Run the same story with a different agent to determine if the regression is code-level or agent-specific.

# Run with different agent
uv run python tests/uat/stories/run_story.py \
  tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
  --agents claude \
  --keep-container

Decision:

  • Both agents regressed: Code-level regression (a tool changed behavior). Continue to Step 4.
  • Only one agent regressed: Agent-specific issue (model update, prompt sensitivity). Note as agent-specific.

Token-Based Regression Detection (Secondary)

Token cost is a secondary metric — it triggers investigation, not automatic failure.

  • Extract billable tokens (non-cached input + output + thoughts) from session files
  • Compare against baseline: >30% increase = investigate (check for KV-cache misses first, see SKILL.md Step 7)
  • Common false positives: KV-cache misses (provider-side), agent exploring differently between runs

Also secondary:

  • Cached tokens / cache hit ratio — useful for diagnosing billable token spikes, but varies provider-side
  • Duration — noisy (network latency, cache misses, server load), only flag large (>2x) outliers

A story can pass functionally but show higher token cost. Investigate the cause before labeling it a regression.

Step 4: Diagnose

Identify the likely cause of the regression.

Git Diff Analysis

# Find the baseline SHA from JSONL
baseline_sha=$(grep '"story":"s01"' local/uat-results.jsonl | grep '"passed":true' | tail -1 | jq -r '.sha')

# Diff src/ between baseline and current
git diff ${baseline_sha}..HEAD -- src/

Focus Areas

Look at changes in these files first:

Changed File Impact
src/ha_mcp/tools/tools_*.py Tool behavior changes
src/ha_mcp/tools/smart_search/ Entity search behavior
src/ha_mcp/tools/device_control.py Device control behavior
src/ha_mcp/client/rest_client.py API interaction changes
src/ha_mcp/errors.py Error message changes

Session Comparison

Compare the session files from the passing baseline run and failing current run:

# Gemini session comparison
baseline_tools = extract_tool_calls(baseline_session)
current_tools = extract_tool_calls(current_session)

# Look for:
# - New error responses
# - Different tool call patterns
# - Changed return values

Step 5: Report

Document the regression finding:

### Regression Report: s01 (Sunset Porch Light)

**Status:** Confirmed regression
**Agent:** gemini
**Baseline:** v6.6.1 (sha: abc123) - PASS
**Current:** uat-stories (sha: def456) - FAIL

**Cause:** `ha_config_set_automation` now returns a different error format
when the automation alias contains spaces, breaking the agent's retry logic.

**Changed files:**
- `src/ha_mcp/tools/tools_config_automations.py` (lines 45-60)

**Suggested fix:** Update error message format to preserve backward compatibility.

Flakiness Tracking

When a story is flaky (passes on retry), track it:

{"story":"s01","agent":"gemini","flaky":true,"attempts":3,"passed":2,"failed":1,"timestamp":"..."}

If a story is consistently flaky (>30% failure rate over multiple evaluation runs), consider:

  1. Making the story prompt more explicit
  2. Adding more setup steps
  3. Increasing the agent timeout
  4. Splitting into simpler sub-stories

Decision Matrix

Re-run Result Control Result Cross-Agent Verdict
Passes on retry N/A N/A Flaky - note and move on
Fails 3x Also fails N/A Environment issue
Fails 3x Passes Also regressed Code regression - diagnose
Fails 3x Passes Only one agent Agent-specific - note

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts1mo3 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    The skill is a specialized tool for testing and evaluating Home Assistant MCP tool behavior. It performs shell command execution to manage Docker containers and git repositories. Potential security concerns include the handling of access tokens in command-line arguments, the execution of code from user-provided git branches, and a surface for indirect prompt injection from the analysis of external session logs.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

Signed by skilld at 955af7b. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
What it can do
Runs commands Reads files Edits files
disable-model-invocation
true
argument-hint
--baseline v6.6.1 [--agents gemini] [--stories s01,s02]
All 6 allowed tools
BashReadWriteGlobGrepTask

README badge

README badge for homeassistant-ai/ha-mcp/bat-story-eval