Index of every page on this site: llms.txt
All skills
github avatar

/mcp-release-qa

@f10fdb8 official
by githubgithub/awesome-copilot41k stars
5,399
Run it yourself

Verify an MCP server before release by exercising a real protocol session, comparing runtime capabilities with source and documentation, testing failure paths, and recording reproducible evidence. Use when shipping or reviewing an MCP server, tool, resource, prompt, catalog, or install path.

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/mcp-release-qa

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈77 tokens for metadata: the name and description. ≈1.7k when used: this file.

Description uses 3.9% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈309 characters, or 3.9%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

MCP Release QA

Test the server that users will run. A schema review or a passing unit test is not runtime evidence.

This skill complements security review. It focuses on protocol behavior, published-contract drift, transport correctness, and reproducible release evidence.

Rules

  • Run checks against a fresh server process built from the candidate revision.
  • Keep initialize, notifications/initialized, discovery, and invocation in the same session. A new process is a new STDIO session.
  • Treat source registrations as implementation truth and public documentation as a contract that must match it.
  • Record exact commands and raw responses. Do not replace missing evidence with "looks correct."
  • Do not invoke mutation-capable tools against production data. Use fixtures, a sandbox, or stop and name the missing safe test environment.
  • Derive the expected capability inventory from the candidate source on every run.

1. Establish the release surface

Identify:

  • candidate commit and build command;
  • server entry point and transport: STDIO, Streamable HTTP, or SSE;
  • supported MCP protocol versions;
  • source files that register tools, resources, resource templates, and prompts;
  • generated catalogs, manifests, README tables, and install instructions;
  • existing protocol, integration, and smoke-test commands.

Prefer repository-native commands. Inspect package.json, pyproject.toml, Makefile, CI workflows, and contributor instructions before inventing a test harness.

2. Start a clean server

Build the candidate and start the documented entry point with test-safe configuration. Capture:

  • the exact command;
  • commit SHA;
  • environment variable names, with values redacted;
  • stdout, stderr, and exit status;
  • the endpoint or child-process transport used by the client.

For STDIO, stdout is protocol-only. Logs, banners, and stack traces belong on stderr. For HTTP transports, record the status, relevant MCP headers, and session identifier handling without printing credentials.

If the server cannot start from its documented instructions, report that as a release failure and preserve the startup error verbatim.

3. Exercise one complete session

Run this sequence through a real MCP client or the repository's integration harness:

  1. initialize with a protocol version the server claims to support.
  2. Confirm the negotiated version and advertised capabilities.
  3. Send notifications/initialized.
  4. Call ping.
  5. Call each supported discovery method:
    • tools/list
    • resources/list
    • resources/templates/list
    • prompts/list
  6. Exercise at least one representative read-only item from every advertised capability class.
  7. Follow pagination until no cursor remains when a list method is paginated.

Do not send post-initialization requests through separate one-shot processes. That accidentally tests several incomplete sessions instead of one valid session.

4. Prove inventory parity

Build four inventories from current evidence:

Surface Evidence
Source Registered tool, resource, template, and prompt definitions
Runtime Results from the live discovery methods
Generated metadata Catalogs, manifests, or generated indexes
Documentation README, reference pages, and install output

Compare by stable identifier. Report:

  • source entries missing at runtime;
  • runtime entries absent from metadata or documentation;
  • stale names, descriptions, arguments, URIs, or prompt parameters;
  • documented install commands that do not start the candidate server.

Regenerate derived files with the repository's own build command, then fail if the working tree still contains unexplained generated changes.

5. Check published contracts

For every discovered item, verify the runtime definition against its source:

Tools

  • Name and description are stable and specific.
  • inputSchema defines types, required fields, enums, and bounds where needed.
  • Unknown properties are rejected when the tool contract is closed.
  • Mutation, idempotence, read-only, and open-world annotations match behavior.
  • Successful calls conform to outputSchema when one is published.
  • Errors are protocol errors or structured tool failures, not leaked stack traces.

Resources and templates

  • URIs and MIME types match the registered definitions.
  • Static resources are readable.
  • Template parameters are validated before resolution.
  • Missing or forbidden resources fail explicitly.

Prompts

  • Required and optional arguments match discovery output.
  • prompts/get returns usable messages for valid arguments.
  • Missing required arguments and unknown prompt names fail explicitly.

6. Test failure paths

At minimum, probe:

  • a request before initialization completes;
  • malformed JSON or an invalid JSON-RPC envelope;
  • an unknown method;
  • an unsupported protocol version;
  • repeated initialization;
  • unknown tool, resource, and prompt names;
  • missing, extra, wrong-type, and out-of-bounds arguments;
  • a request at the documented transport-size limit and one beyond it;
  • a controlled internal failure with credentials and stack traces redacted.

Verify that each response has the correct request ID, a useful error message, and no successful side effect. For STDIO, also confirm every stdout line is a complete protocol message and a healthy session leaves stderr clean unless the server explicitly documents diagnostic output.

7. Smoke-test installation

When the project publishes an install command:

  1. Create a temporary destination outside the source checkout.
  2. Run the public install command exactly as documented.
  3. Start the installed artifact without relying on files from the source tree.
  4. Repeat initialization, discovery, and one read-only invocation.
  5. Remove the temporary destination after preserving the command output.

An install string that was only inspected is unverified.

8. Report the evidence

Use this format:

# MCP Release QA

Candidate: [commit]
Transport: [STDIO | Streamable HTTP | SSE]
Verdict: PASS | PASS WITH CAVEATS | FAIL

## Commands and results
- `[exact command]` — [exit status and result]

## Session transcript
- initialize: [result]
- discovery: [result]
- representative calls: [result]
- negative paths: [result]

## Parity
| Identifier | Source | Runtime | Metadata | Docs | Result |
|---|---|---|---|---|---|

## Findings
| Severity | Evidence | Impact | Narrowest fix |
|---|---|---|---|

## Missing evidence
- [check that could not run and why]

Use FAIL for a server that cannot start, complete a valid session, keep the transport parseable, or safely reject invalid input. Use PASS WITH CAVEATS only for bounded documentation or metadata drift that does not misrepresent a dangerous capability. Otherwise use PASS.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning3mo3 checks · Risk SAFE
  • Gen Agent Trust Hub3mo

    This skill provides a structured process for testing MCP servers. It includes essential security practices such as redacting sensitive credentials from test logs and using isolated environments for testing commands.

  • Socket3mo

    No alerts

  • Snyk3mo

    Risk: MEDIUM · 1 issue

Signed by skilld at f10fdb8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago

README badge

README badge for github/awesome-copilot/mcp-release-qa