Regression Protocol
When a story's score decreases compared to its baseline, follow this protocol to confirm whether the regression is real and identify its cause.
Overview
Decreased score detected
|
+- Step 1: Re-run (up to 3x)
| +- Still decreased? -> Continue
| +- Passes on retry? -> Flaky, note and move on
|
+- Step 2: Control run (baseline version)
| +- Baseline also fails? -> Not a regression, environment issue
| +- Baseline passes? -> Confirmed regression
|
+- Step 3: Cross-agent check
| +- Other agent also regressed? -> Code-level regression
| +- Other agent fine? -> Agent-specific issue
|
+- Step 4: Diagnose
| +- git diff between versions
| +- Identify likely cause
|
+- Step 5: ReportStep 1: Re-Run (Flakiness Check)
Re-run the failed story up to 3 times. Agent behavior is non-deterministic, so a single failure may be a fluke.
# Re-run specific story
uv run python tests/uat/stories/run_story.py \
tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
--agents gemini \
--keep-containerDecision:
- Passes on any retry: Mark as
flakyand move on. Note the pass/fail ratio. - Fails all 3 times: Continue to Step 2.
Step 2: Control Run (Baseline Version)
Run the same story against the baseline version to confirm the regression is real.
# Run against baseline version
uv run python tests/uat/stories/run_story.py \
tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
--agents gemini \
--branch v6.6.1 \
--keep-containerDecision:
- Baseline also fails: Not a regression - likely environment issue, flaky test, or agent model change. Note and move on.
- Baseline passes, current fails: Confirmed regression. Continue to Step 3.
Step 3: Cross-Agent Check
Run the same story with a different agent to determine if the regression is code-level or agent-specific.
# Run with different agent
uv run python tests/uat/stories/run_story.py \
tests/uat/stories/catalog/s01_automation_sunset_lights.yaml \
--agents claude \
--keep-containerDecision:
- Both agents regressed: Code-level regression (a tool changed behavior). Continue to Step 4.
- Only one agent regressed: Agent-specific issue (model update, prompt sensitivity). Note as agent-specific.
Token-Based Regression Detection (Secondary)
Token cost is a secondary metric — it triggers investigation, not automatic failure.
- Extract billable tokens (non-cached input + output + thoughts) from session files
- Compare against baseline: >30% increase = investigate (check for KV-cache misses first, see SKILL.md Step 7)
- Common false positives: KV-cache misses (provider-side), agent exploring differently between runs
Also secondary:
- Cached tokens / cache hit ratio — useful for diagnosing billable token spikes, but varies provider-side
- Duration — noisy (network latency, cache misses, server load), only flag large (>2x) outliers
A story can pass functionally but show higher token cost. Investigate the cause before labeling it a regression.
Step 4: Diagnose
Identify the likely cause of the regression.
Git Diff Analysis
# Find the baseline SHA from JSONL
baseline_sha=$(grep '"story":"s01"' local/uat-results.jsonl | grep '"passed":true' | tail -1 | jq -r '.sha')
# Diff src/ between baseline and current
git diff ${baseline_sha}..HEAD -- src/Focus Areas
Look at changes in these files first:
| Changed File | Impact |
|---|---|
src/ha_mcp/tools/tools_*.py |
Tool behavior changes |
src/ha_mcp/tools/smart_search/ |
Entity search behavior |
src/ha_mcp/tools/device_control.py |
Device control behavior |
src/ha_mcp/client/rest_client.py |
API interaction changes |
src/ha_mcp/errors.py |
Error message changes |
Session Comparison
Compare the session files from the passing baseline run and failing current run:
# Gemini session comparison
baseline_tools = extract_tool_calls(baseline_session)
current_tools = extract_tool_calls(current_session)
# Look for:
# - New error responses
# - Different tool call patterns
# - Changed return valuesStep 5: Report
Document the regression finding:
### Regression Report: s01 (Sunset Porch Light)
**Status:** Confirmed regression
**Agent:** gemini
**Baseline:** v6.6.1 (sha: abc123) - PASS
**Current:** uat-stories (sha: def456) - FAIL
**Cause:** `ha_config_set_automation` now returns a different error format
when the automation alias contains spaces, breaking the agent's retry logic.
**Changed files:**
- `src/ha_mcp/tools/tools_config_automations.py` (lines 45-60)
**Suggested fix:** Update error message format to preserve backward compatibility.Flakiness Tracking
When a story is flaky (passes on retry), track it:
{"story":"s01","agent":"gemini","flaky":true,"attempts":3,"passed":2,"failed":1,"timestamp":"..."}If a story is consistently flaky (>30% failure rate over multiple evaluation runs), consider:
- Making the story prompt more explicit
- Adding more setup steps
- Increasing the agent timeout
- Splitting into simpler sub-stories
Decision Matrix
| Re-run Result | Control Result | Cross-Agent | Verdict |
|---|---|---|---|
| Passes on retry | N/A | N/A | Flaky - note and move on |
| Fails 3x | Also fails | N/A | Environment issue |
| Fails 3x | Passes | Also regressed | Code regression - diagnose |
| Fails 3x | Passes | Only one agent | Agent-specific - note |