Index of every page on this site: llms.txt
All skills
vercel avatar

/vercel-plugin-eval

@250769d official
by vercelvercel/vercel-plugin340 stars
82
Run it yourself

Run live eval sessions against the vercel-plugin to verify hook behavior, skill injection, dedup correctness, and coverage. Launches real Claude Code sessions via WezTerm, monitors debug logs, and produces a structured coverage report.

  • 1 file
  • 4.9 KB
  • Updated 8 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/vercel/vercel-plugin/vercel-plugin-eval

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈64 tokens for metadata: the name and description. ≈1.2k when used: this file.

Description uses 3.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈256 characters, or 3.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Plugin Eval

Launch real Claude Code sessions with the plugin installed, monitor debug logs in real-time, and verify every hook fires correctly with proper dedup.

DO NOT (Hard Rules)

  • DO NOT use claude --print or -p — hooks don't fire, no files created
  • DO NOT use --dangerously-skip-permissions
  • DO NOT create projects in /tmp/ — always use ~/dev/vercel-plugin-testing/
  • DO NOT manually wire hooks or create settings.local.json — use npx add-plugin
  • DO NOT set CLAUDE_PLUGIN_ROOT manually
  • DO NOT use bash -c in WezTerm — use /bin/zsh -ic
  • DO NOT use full path to claude — use the x alias
  • DO NOT write eval scripts — do everything as Bash tool calls in the conversation

Copy the exact commands below. Do not improvise.

Quick Start

Always append a timestamp to directory names so reruns don't overwrite old projects:

# 1. Create test dir & install plugin (with timestamp)
TS=$(date +%Y%m%d-%H%M)
SLUG="my-eval-$TS"
mkdir -p ~/dev/vercel-plugin-testing/$SLUG
cd ~/dev/vercel-plugin-testing/$SLUG
npx add-plugin https://github.com/vercel/vercel-plugin -s project -y

# 2. Launch session via WezTerm
wezterm cli spawn --cwd /Users/johnlindquist/dev/vercel-plugin-testing/$SLUG -- /bin/zsh -ic \
  "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '<PROMPT>' --settings .claude/settings.json; exec zsh"

# 3. Find debug log (wait ~25s for session start)
find ~/.claude/debug -name "*.txt" -mmin -2 -exec grep -l "$SLUG" {} +

What to Monitor

Hook firing (all 8 registered hooks)

LOG=~/.claude/debug/<session-id>.txt

# SessionStart (3 hooks)
grep "SessionStart.*success" "$LOG"

# PreToolUse skill injection
grep -c "executePreToolHooks" "$LOG"        # total calls
grep -c "provided additionalContext" "$LOG"  # injections

# UserPromptSubmit
grep "UserPromptSubmit.*success" "$LOG"

# PostToolUse validate + shadcn font-fix
grep "posttooluse-validate.*provided" "$LOG"
grep "PostToolUse:Bash.*success" "$LOG"

# SessionEnd cleanup
grep "SessionEnd" "$LOG"

Dedup correctness (the key metric)

TMPDIR=$(node -e "import {tmpdir} from 'os'; console.log(tmpdir())" --input-type=module)
CLAIMDIR="$TMPDIR/vercel-plugin-<session-id>-seen-skills.d"

# Claim files = one per skill, atomic O_EXCL
ls "$CLAIMDIR"

# Compare: injections should equal claims
inject_meta=$(grep -c "skillInjection:" "$LOG")
claims=$(ls "$CLAIMDIR" 2>/dev/null | wc -l | tr -d ' ')
echo "Injections: $((inject_meta / 3)) | Claims: $claims"

skillInjection: appears 3x per actual injection in the debug log (initial check, parsed, success). Divide by 3.

PostToolUse validate quality

Look for real catches — API key bypass, outdated models, wrong patterns:

grep "VALIDATION" "$LOG" | head -10

Scenario Design

Describe products and features, never name specific technologies. Let the plugin infer which skills to inject. Always end prompts with: "Link the project to my vercel-labs team so we can deploy it later. Skip any planning and just build it. Get the dev server running."

Coverage targets by scenario type

Scenario Type Skills Exercised
AI chat app ai-sdk, ai-gateway, nextjs, ai-elements
Durable workflow workflow, ai-sdk, vercel-queues
Monorepo turborepo, turbopack, nextjs
Edge auth + routing routing-middleware, auth, sign-in-with-vercel
Chat bot (multi-platform) chat-sdk, ai-sdk, vercel-storage
Feature flags + CRM vercel-flags, vercel-queues, ai-sdk
Email pipeline email, satori, ai-sdk, vercel-storage
Marketplace/payments payments, marketplace, cms
Kitchen sink micro, ncc, all niche skills

Hard-to-trigger skills (8 of 44)

These need explicit technology references in the prompt because agents don't naturally reach for them:

  • ai-elements — say "use the AI Elements component registry"
  • v0-dev — say "generate components with v0"
  • vercel-firewall — say "use Vercel Firewall for rate limiting"
  • marketplace — say "publish to the Vercel Marketplace"
  • geist — say "install the geist font package"
  • json-render — name files components/chat-*.tsx

Coverage Report

Write results to .notes/COVERAGE.md with:

  1. Session index — slug, session ID, unique skills, dedup status
  2. Hook coverage matrix — which hooks fired in which sessions
  3. Skill injection table — which of the 44 skills triggered
  4. Dedup stats — injections vs claims per session
  5. Issues found — bugs, pattern gaps, validation findings

Cleanup

rm -rf ~/dev/vercel-plugin-testing

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 250769d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 8 months ago

README badge

README badge for vercel/vercel-plugin/vercel-plugin-eval