Index of every page on this site: llms.txt
All skills
vercel avatar

/plugin-audit

@250769d official
by vercelvercel/vercel-plugin340 stars
82
Run it yourself

Audit vercel-plugin performance on real-world projects. Extracts tool calls from Claude Code conversation logs, tests hook matching against actual inputs, identifies pattern coverage gaps, and checks plugin cache staleness. Use when asked to audit, test, or investigate plugin skill injection on a real project.

  • 3 files
  • 11.1 KB
  • Updated 9 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/vercel/vercel-plugin/plugin-audit

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈81 tokens for metadata: the name and description. ≈651 when used: this file. ≈431 more on demand in 1 file.

Description uses 4.1% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈326 characters, or 4.1%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Plugin Audit

Audit how well vercel-plugin skill injection performs on real-world Claude Code sessions.

Workflow

1. Locate conversation logs

Find JSONL conversation logs for a target project:

ls -lt ~/.claude/projects/-Users-*-<project-name>/*.jsonl

The path uses the project's absolute path with slashes replaced by hyphens and a leading hyphen.

2. Extract tool calls

Parse the JSONL log to extract all tool_use entries. Each line is a JSON object with message.content[] containing type: "tool_use" blocks. Extract name and input fields. Group by tool type (Bash, Read, Write, Edit).

3. Test hook matching

Use the exported pipeline functions directly — do NOT shell out to the hook script for each test. Import from the hooks directory:

import { loadSkills, matchSkills } from "./hooks/pretooluse-skill-inject.mjs";
import { createLogger } from "./hooks/logger.mjs";

Call loadSkills() once, then matchSkills(toolName, toolInput, compiledSkills) for each tool call. This is fast and gives exact match results.

4. Identify gaps

Compare matched skills against what SHOULD have matched based on the project's technology stack. Common gap categories:

  • Path pattern gaps: Files that should trigger a skill but don't (e.g., src/db/schema.ts not matching vercel-storage)
  • Bash pattern gaps: Commands that should trigger but don't (e.g., missing package manager variants)
  • Dedup masking: Skills that matched but were deduped before injection
  • Budget/cap drops: Skills matched but dropped by the 12KB budget or 3-skill ceiling

5. Check plugin cache staleness

Compare the installed plugin cache against the dev version:

# Cache location
~/.claude/plugins/cache/vercel-labs-vercel-plugin/vercel-plugin/<version>/

# Compare skill content
diff <(grep 'pattern' skills/<skill>/SKILL.md) <(grep 'pattern' ~/.claude/plugins/cache/.../skills/<skill>/SKILL.md)

Check ~/.claude/plugins/installed_plugins.json for version and git SHA.

Report Format

Produce a structured report with:

  1. Session summary: Project, date, tool call count, model
  2. Match matrix: Table of tool calls × matched skills (with match type)
  3. Coverage gaps: Unmatched tool calls that should have matched, with suggested pattern additions
  4. Dedup timeline: Order of skill injections and what got deduped
  5. Cache status: Whether installed version matches dev, with specific diffs

References

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 250769d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 9 months ago

README badge

README badge for vercel/vercel-plugin/plugin-audit