Index of every page on this site: llms.txt
All skills
microsoft avatar

/scope-cli

@e4b5866
by microsoftmicrosoft/scope89 stars
32
Run it yourself

Interact with the Scope benchmarking platform using the Scope CLI — submit runs, list results, manage criteria, generate reports, and administer agents, skills, and MCP servers.

  • 1 file
  • 9.3 KB
  • Updated last month
  • GitHub
Use this Skill: https://skilld.dev/gh/microsoft/scope/scope-cli

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈48 tokens for metadata: the name and description. ≈2.3k when used: this file.

Description uses 2.4% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈189 characters, or 2.4%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Scope CLI

You have access to the Scope CLI — an AI coding-agent benchmarking platform. Run it with pnpm cli from the repository root (scope-core).

All commands below assume you are in the repo root. Prefix every invocation with pnpm cli.


Authentication and enrollment

The CLI's shared API transport can attach a caller-provided SCOPE_TOKEN containing an IdP access token. Already-enrolled users keep using that bearer unchanged; there is no Scope-token exchange or new CLI login implementation in this milestone. Do not assume the deferred scope auth login/keychain commands exist.

A new identity must explicitly call POST /api/v1/users/me using its IdP bearer before ordinary authenticated commands. Use the configured Scope API URL:

curl --fail-with-body -sS \
  -X POST \
  -H "Authorization: Bearer $SCOPE_TOKEN" \
  -H "Cache-Control: no-store" \
  "${SCOPE_API_URL%/}/api/v1/users/me"

This POST creates/refreshes the user, profile, lastLoginAt, and eligible bootstrap promotion; never prefetch, poll, or automatically use it to recover an ordinary lookup. GET /users/me only checks existing access. 403 user_not_enrolled calls for explicit enrollment; 403 user_disabled is a denial, not a refresh-token prompt. Invalid/expired bearer → 401; required Mongo/JWKS outage → 503.

All non-public bearer calls verify the IdP token before Redis/Mongo resolution. The active-user cache is fixed/non-sliding (300 seconds by default), so DB-only role/disable edits may remain stale until expiry. No raw bearer is cached by the API. Never echo tokens or include them in URLs/debug output. Public and anonymous rollout behavior is unchanged; full RBAC and interactive CLI auth remain deferred.

See Authentication & RBAC.


Quick Reference

In order to get the full updated reference, run pnpm cli --help or pnpm cli <command> --help for specific commands. Below is a summary of the most common commands.

Runs (benchmark executions)

Action Command
List runs pnpm cli run list
List runs (JSON) pnpm cli run list -o json
Filter by worker pnpm cli run list -w coder-acp-copilot
Get run details pnpm cli run get -i <id>
Check status pnpm cli run status -i <id>
Stream logs pnpm cli run logs -i <id>
Stream logs from start pnpm cli run logs -i <id> --from-start
Submit a run pnpm cli run submit -s <scenario.yaml> -w <worker>
Submit with message pnpm cli run submit -m "your task" -w coder-acp-copilot
Submit with model pnpm cli run submit -s <scenario> -w <worker> --model <model>
Submit with skills pnpm cli run submit -s <scenario> -w <worker> --skills <slug...>
Submit with MCP servers pnpm cli run submit -s <scenario> -w <worker> --mcp-servers <slug...>
Delete a run pnpm cli run delete -i <id>
Download artifacts pnpm cli run download -i <id>
Download & extract pnpm cli run download -i <id> -e
Upload archive pnpm cli run upload <path>
Demo (multi-worker TUI) pnpm cli run demo -m "task" -w coder-acp-copilot,coder-acp-claude-code

Reports

Action Command
List reports pnpm cli report list
List for a run pnpm cli report list -r <run-id>
Generate report pnpm cli report generate -i <run-id> --stream
Get report pnpm cli report get -i <report-id>
Get as markdown pnpm cli report get -i <report-id> -o markdown
Stream report logs pnpm cli report logs -i <report-id> --from-start

Report Templates

Action Command
List templates pnpm cli report-template list
Get template pnpm cli report-template get -i <id>
Import templates pnpm cli report-template import <path>

Criteria (evaluation rubrics)

Action Command
List criteria pnpm cli criteria list
Search criteria pnpm cli criteria list -q "search term"
Get criterion pnpm cli criteria get -i <id>
Create criterion pnpm cli criteria create --id <snake_case_id> --prompt "eval prompt"
Update criterion pnpm cli criteria update -i <id> --prompt "new prompt"
Delete criterion pnpm cli criteria delete -i <id>
Show dependency graph pnpm cli criteria graph
Import from YAML pnpm cli criteria import <path>
Dry-run import pnpm cli criteria import <path> --dry-run

Prompt Features

Action Command
List features pnpm cli prompt-feature list
Get feature pnpm cli prompt-feature get -i <id>
Show graph pnpm cli prompt-feature graph
Import from YAML pnpm cli prompt-feature import <path>
Extract from task pnpm cli prompt-feature extract -t "task text"
Extract from scenario pnpm cli prompt-feature extract -s <scenario.yaml>

Agents

Action Command
List agents pnpm cli agent list
Get agent pnpm cli agent get -i <id>
List models pnpm cli agent model list -i <agent-id>
List agent versions pnpm cli agent version list -i <agent-id>
Active versions only pnpm cli agent version list -i <agent-id> --status active

Skills

Action Command
List skills pnpm cli skill list
Search skills pnpm cli skill search -q "query"
Get skill pnpm cli skill get -i <slug>
Import skill pnpm cli skill import --source <owner/repo> --skill-name <name>
Resolve (fetch latest) pnpm cli skill resolve -i <slug>
List revisions pnpm cli skill revisions -i <slug>
Delete skill pnpm cli skill delete -i <slug>

MCP Servers

Action Command
List MCP servers pnpm cli mcp server list
Get MCP server pnpm cli mcp server get -i <slug>
Create MCP server pnpm cli mcp server create --id <slug> --name "Name" --type sse --url <url>
Delete MCP server pnpm cli mcp server delete -i <slug>

Insights

Action Command
List insights pnpm cli insight list
Search insights pnpm cli insight list -q "keyword"
Blocked only pnpm cli insight list --blocked
Get insight (markdown) pnpm cli insight get -i <id> -o markdown
Create insight pnpm cli insight create --title "Title" --description "md body"
Upvote / downvote pnpm cli insight upvote -i <id>

Task Prompts

Action Command
List task prompts pnpm cli task-prompt list
Search prompts pnpm cli task-prompt list -s "search text"
Get prompt pnpm cli task-prompt get -i <uuid>
Create prompt pnpm cli task-prompt create -t "prompt text"
Create from file pnpm cli task-prompt create -f <path>
Extract features pnpm cli task-prompt extract-features -i <uuid>

Provider Secrets

Secret values are never printed; prefer the *-stdin flags.

Action Command
List keys (metadata only) pnpm cli secret list
Get key metadata pnpm cli secret get <key-id>
Store a key printf '%s' "$KEY" | pnpm cli secret create --type anthropic-api-key --value-stdin
Validate without storing printf '%s' "$KEY" | pnpm cli secret preview --type anthropic-api-key --value-stdin
Revalidate pnpm cli secret validate <key-id>
Enable / disable pnpm cli secret update <key-id> --disable
Delete pnpm cli secret delete <key-id>

Output Formats

All list/get commands accept -o, --output <format>:

Format Use case
table Human-readable (default)
tsv Pipe to Unix tools (cut, awk, grep)
json Programmatic access, AI agents
yaml Human-friendly structured data

Examples:

# Get run details as JSON for parsing
pnpm cli run get -i <id> -o json

# List runs as TSV and filter with awk
pnpm cli run list -o tsv | awk -F'\t' '$3 == "completed"'

Common Workflows

Submit a benchmark and watch it

# Submit with streaming
pnpm cli run submit -s config/scenarios/my-scenario.yaml -w coder-acp-copilot

# Or submit without streaming and watch later
pnpm cli run submit -s config/scenarios/my-scenario.yaml -w coder-acp-copilot --no-stream
pnpm cli run logs -i <id> --from-start

Generate and read a report

pnpm cli report generate -i <run-id> --stream
pnpm cli report list -r <run-id>
pnpm cli report get -i <report-id> -o markdown

Import criteria and scenarios

pnpm cli criteria import config/criteria/ --dry-run
pnpm cli criteria import config/criteria/
pnpm cli criteria graph

Tips

  • Use -o json when you need to parse output programmatically.
  • Use --no-stream on run submit to get the ID immediately without blocking.
  • Scenario files are YAML and live in config/scenarios/.
  • Persona files live in config/personas/.
  • Traits files default to config/traits.yaml next to the persona.
  • Skill slugs use the format owner/repo/skill-name.
  • Run IDs are UUIDs — use tab completion or copy from run list output.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at e4b5866. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last month
What it can do
Runs commands
metadata
{
  "author": "scope-core-team",
  "version": "1.0"
}
All 1 allowed tools
Bash

README badge

README badge for microsoft/scope/scope-cli