---
name: bat-adhoc
description: Run bot acceptance tests to validate MCP tools work correctly from a real AI agent's perspective. Use when testing PRs, detecting regressions, or verifying tool changes end-to-end with Claude/Gemini CLIs.
disable-model-invocation: true
argument-hint: [ scenario-description or --help ]
allowed-tools: Bash, Read, Write
title: bat-adhoc
canonical_url: https://skilld.dev/gh/homeassistant-ai/ha-mcp/bat-adhoc
last_updated: 2026-11-12T06:30:57.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> If the user asked to install this Skill, run `npx skilld install homeassistant-ai/ha-mcp/bat-adhoc`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/bat-adhoc`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/homeassistant-ai/ha-mcp/bat-adhoc) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/bat-adhoc`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/bat-adhoc --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/bat-adhoc --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# BAT - Bot Acceptance Testing

Bot acceptance testing validates that MCP tools work correctly from a real AI agent's perspective. You design test scenarios dynamically, run them via `tests/uat/run_uat.py`, and evaluate results.

## When to Use BAT

- **PR validation**: Test that tool changes work correctly from an agent's perspective
- **Regression detection**: Compare behavior between branches
- **Integration verification**: Ensure MCP tools work end-to-end with real agent CLIs

## Workflow

1. **Analyze the change**: Read the diff, identify which tools are affected
2. **Design scenario**: Generate a scenario JSON with setup/test/teardown prompts
3. **Run the script**: Pipe the scenario to `uv run python tests/uat/run_uat.py`
4. **Evaluate summary**: Check `all_passed` per agent. If true, you're done.
5. **Dig deeper on failure**: Read `results_file` for full output, stderr, raw JSON
6. **Regression check**: If test fails, re-run with `--branch master` to compare

## Output Structure

The runner returns a **concise summary** to stdout (saves context when all passes):

```json
{
  "results_file": "/tmp/bat_results_abc123.json",
  "agents": {
    "gemini": {
      "all_passed": true,
      "test": {
        "completed": true,
        "duration_ms": 8100,
        "exit_code": 0,
        "num_turns": 5,
        "tool_stats": { "totalCalls": 4, "totalSuccess": 4, "totalFail": 0 }
      },
      "aggregate": {
        "total_duration_ms": 15300,
        "total_turns": 12,
        "total_tool_calls": 9,
        "total_tool_success": 9,
        "total_tool_fail": 0
      }
    }
  }
}
```

- **Phase stats**: `num_turns`, `tool_stats` (per phase) for fine-grained comparison
- **Aggregate stats**: Total counts across all phases for overall efficiency comparison
- **Output**: every phase includes `output` (plus `tool_trace` when tool calls were logged); a failed phase also includes `stderr` when it is not empty
- **Full results**: raw JSON, complete output always available at `results_file`

## Scenario Design Guidelines

- **setup_prompt**: Create any entities/state the test needs
- **test_prompt**: Exercise the tools being tested, ask the agent to report results clearly
- **teardown_prompt**: Clean up created entities
- Keep prompts focused - each scenario tests ONE behavior
- Ask the agent to report: what succeeded, what failed, any unexpected behavior

## Example: Testing Error Signaling

```bash
cat <<'EOF' | uv run python tests/uat/run_uat.py --agents gemini
{
  "setup_prompt": "Create a test automation called 'bat_error_test' with a time trigger at 23:59 and action to turn on light.bed_light.",
  "test_prompt": "Try to get automation 'automation.nonexistent_xyz'. Report if the tool signaled an error or returned a normal response. Then get automation 'automation.bat_error_test' and report its structure.",
  "teardown_prompt": "Delete automation 'bat_error_test' if it exists."
}
EOF
```

## Regression Comparison Workflow

Run the same scenario twice from the branch checkout and compare stats. `--branch master` installs ha-mcp from `master` on GitHub; omitting `--branch` runs the local code, which also covers unpushed commits:

```bash
# Baseline: master
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --branch master --agents gemini

# Target: local code
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --agents gemini
```

To compare a pushed branch that is not checked out, pass `--branch <branch>` for the target run too.

**Compare these metrics:**

*Primary (decide pass/fail on these):*
- **Task completion**: Did both pass? Any new failures?
- **Accuracy**: Check agent output quality - did it understand the task correctly?
- **Tool success rate**: Compare `aggregate.total_tool_calls` vs `total_tool_fail`

*Secondary (report but don't decide on these alone):*
- **Tool call count**: Compare `aggregate.total_tool_calls`, `aggregate.total_turns` — directional signal, not conclusive (agent exploration varies between runs)
- **Duration**: Compare `aggregate.total_duration_ms` — noisy due to network, cache misses, server load. Only flag large (>2x) regressions.

**Robustness tip:** Ask the same task in different ways (variation testing) to check if results are consistent across phrasings.

## Cost Awareness

Each scenario invocation costs API credits (one per agent per phase). Design scenarios efficiently:

- Combine related checks in a single test_prompt when possible
- Only use setup/teardown when the test needs specific state
- Start with one agent, expand to both only when cross-agent comparison matters

## Handling Arguments

When `/bat-adhoc` is invoked with arguments:

**If arguments contain a scenario description**, generate the JSON scenario and run it:
```
/bat-adhoc test automation create with sunrise trigger then modify to sunset
```
→ Generate appropriate scenario JSON and execute

**If `--help` or no arguments**, show this help text.

**Otherwise**, treat `$ARGUMENTS` as instructions for what to test and design+run the scenario accordingly.

## Full Documentation

For complete CLI reference and output format, see `tests/uat/README.md`.
