All skills
mblode avatar

/ax-audit

@57eb304
by Matthew Blodemblode/agent-skills134 stars
12

Audits agentic products for tool parity, authority, approval payloads, recovery, and trust using 27 rules and a ship verdict. Use when asked for an "AX audit", to review an agent approval flow, or whether an agent can operate the product. For human-facing API ergonomics use dx-audit; for ordinary UI use ui-design.

Use this Skill: https://skilld.dev/gh/mblode/agent-skills/ax-audit

This session only. Nothing lands on disk.

rules-archcomm-no-approval-gate.md

≈1.5k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Orchestrator executes high-stakes tools with no gate on the code path

The tool-use loop calls tool.execute() directly, so nothing in the orchestrator can interpose a confirmation. Whatever the UI renders, a destructive tool registered tomorrow ships ungated by default. Violates Parity: oversight belongs on the execution path both the user and the agent go through.

Scope: this rule asks whether a gate exists in the orchestrator at all, and whether tool definitions carry the risk metadata a gate needs. Which approval treatment each risk level earns (auto-apply, quick confirm, diff, modal) is rules-ax/control-no-approval-gate.

What goes wrong

A chat surface ships a confirmation modal for the three delete tools it knows about, wired at the call site. Someone adds transfer_funds to the registry. The loop calls it the same way it calls list_files, and no reviewer notices, because the gate was never on the path, only on three call sites.

Detection

Surfaces: agent-tool-execution

Static signals:

  1. Find the tool-use loop and every call site that reaches .execute(. More than one path to execution means there is no single gate.
  2. Identify destructive/financial/external operations: send, delete, publish, deploy, charge, transfer.
  3. Check whether an approval handler sits between dispatch and execution on that path, and whether it is reached for every tool or only for named ones.
  4. Check what happens to a tool with no stakes metadata: defaulting to auto-approve is a fail even when today's registry is fully labelled.
  5. Check what the checkpoint receives. Tool name and arguments alone cannot express provenance (did the agent create this object this conversation, does the target leave the workspace), so a gate given only those can be tuned by strictness and nothing else.
  6. Check where the stakes come from. MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) are hints the spec tells clients to treat as untrusted from untrusted servers; a checkpoint that auto-approves on a third-party server's readOnlyHint: true has delegated itself. The spec defaults (destructiveHint: true, readOnlyHint: false) are the fail-closed baseline.
  7. In Claude Agent SDK code, find which step holds the check. Allow rules, acceptEdits, and bypassPermissions resolve a call before canUseTool runs, so a check that exists only in that callback is off the path for every pre-approved tool. A PreToolUse hook runs before every other step and is on it.

Runtime signals: a destructive tool added to the registry with no other change executes without prompting.

Concrete commands:

rg '(name|toolName).*["'"'"'](send|delete|remove|publish|deploy|charge|transfer)' --type=ts src/
rg '(executeTool|callTool|invokeTool)' --type=ts -A 10 src/ | rg -v '(confirm|approve|requireApproval)'
rg '(requireApproval|confirmBefore|approvalGate|stakesLevel)' --type=ts src/
rg '(toolApproval|needsApproval|canUseTool|PreToolUse|permissionMode|bypassPermissions|readOnlyHint|destructiveHint)' --type=ts src/

False-positive guards:

  • Skip files with // ax-audit-ignore:comm-no-approval-gate.
  • Skip read-only operations (get, list, search, fetch).
  • Skip operations marked safe/reversible in tool metadata.
  • Skip test files and fixtures.

Fix

Every tool call passes through one checkpoint, and tools declare their own risk so the checkpoint can classify them without a hardcoded name list. Fail closed: a tool with no declared stakes is treated as high.

// before: no interposition point, gating is per-call-site or absent
async function executeTool(tc: ToolCall) {
  return tools[tc.name].execute(tc.args);
}

// after: single choke point, risk declared on the tool
async function executeTool(tc: ToolCall, onApproval: ApprovalHandler) {
  const tool = tools[tc.name];
  const risk = tool.stakes ?? "high";        // unlabelled tools are not auto-approved
  const decision = await onApproval({ tool: tc.name, args: tc.args, risk, reversibility: tool.reversibility });
  if (decision !== "approved") return { status: "cancelled", reason: decision };
  return tool.execute(tc.args);
}

The onApproval handler owns the treatment per risk level, so it is audited by rules-ax/control-no-approval-gate, not here. This rule passes once the checkpoint exists, covers every tool, and cannot be bypassed by registering a new one. The choke point has a name in each stack: toolApproval on the agent in AI SDK 7, a PreToolUse hook in the Claude Agent SDK, the tools/call dispatcher in an MCP host.

Default tier and overrides

Defaults to: release-blocker

The gap is in shared orchestrator code, so it reaches every surface that can trigger a mutating tool. The rows do not taper the way a presentation rule's do.

Surface Tier
Agent tool execution release-blocker
Agent chat release-blocker

Examples

Anti-pattern (fails): the tool-use loop calls tools[tc.name].execute(tc.args) with no handler parameter threaded in, and grep for requireApproval|approvalGate in the orchestrator returns nothing. Confirmation modals in the chat component do not count; they cannot see a tool call the loop makes on the server.

Applied (passes): one executeTool wrapper, every registration path goes through it, and tool definitions carry stakes / reversibility fields the checkpoint reads.

Suppression

// ax-audit-ignore:comm-no-approval-gate, internal cleanup, operates only on temp files
const cleanupTool = { execute: (args) => fs.rm(args.tempDir) };

Source: SKILL.md on GitHub

No alerts13d3 checks · Risk SAFE
  • Gen Agent Trust Hub13d

    The skill is a specialized auditing framework for AI agent products, focusing on architectural integrity and user trust. It uses standard shell tools for static analysis of codebases. The analysis found no malicious behavior, obfuscation, or data exfiltration risks.

  • Socket13d

    No alerts

  • Snyk13d

    Risk: LOW · No issues

Signed by skilld at 57eb304. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 2 days ago.

Activeupdated 2 weeks ago

README badge

README badge for mblode/agent-skills/ax-audit