Skill Benchmark: nvidia-ontology-setup
✅ Overall verdict: PASS — Recommended for publication
Publication Recommendation
Recommended for publication based on the completed evaluation evidence in this report.
Evaluation Metadata
- Skill:
nvidia-ontology-setup - Validation status:
passed - Evaluation date: 2026-11-19
- Evaluator version:
1.6.10 - Agents: Claude Code (
aws/anthropic/bedrock-claude-opus-5-5), Codex (openai/openai/gpt-6.1-sol) - Tasks: 10 evaluation tasks (10 positive)
- Dataset digest:
sha256:dcbcf0848aefb19cd5b010e9a0d69ed8920e61dc3fdba560ca0f88cd48a8da96(skill-evaluator-dataset-snapshot/1) - Attempts per task: 1
- Environment:
k8s-sandbox - Tier 2 evidence: required for publication
- Tier 3 evidence: required for publication
Each task attempt ran in its own isolated sandbox pod.
Tier 1 LLM Provenance
SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan.
| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism |
|---|---|---|---|---|---|---|
| nvidia-ontology-setup | nv_inference | openai | mcp_tool_poisoning | aws/anthropic/bedrock-claude-opus-5-5 |
2.13.5 |
nondeterministic / provider_defaults |
| nvidia-ontology-setup | nv_inference | openai | semantic_developer_intent | aws/anthropic/bedrock-claude-opus-5-5 |
2.13.5 |
nondeterministic / provider_defaults |
| nvidia-ontology-setup | nv_inference | openai | semantic_quality_policy | aws/anthropic/bedrock-claude-opus-5-5 |
2.13.5 |
nondeterministic / provider_defaults |
| nvidia-ontology-setup | nv_inference | openai | semantic_security_discovery | aws/anthropic/bedrock-claude-opus-5-5 |
2.13.5 |
nondeterministic / provider_defaults |
| nvidia-ontology-setup | nv_inference | openai | meta_analyzer | aws/anthropic/bedrock-claude-opus-5-5 |
2.13.5 |
nondeterministic / provider_defaults |
- Controls for
nvidia-ontology-setup: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no)
What This Report Answers
The three-tier evaluation checks whether the skill:
- is safe to use;
- produces correct answers;
- is discovered and activated when needed;
- helps the agent complete the user's goal and expected workflow; and
- avoids wasted skill and tool usage.
Results at a Glance
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---|---|
| Overall | 94.1% — baseline ran, but no comparable score was available; uplift unavailable | 88.8% — baseline ran, but no comparable score was available; uplift unavailable |
| Security | 80.0% → 100.0% (+20.0 points) | 75.0% → 90.0% (+15.0 points) |
| Correctness | 48.0% → 100.0% (+52.0 points) | 58.0% → 98.0% (+40.0 points) |
| Discoverability | 93.5% — baseline ran, but no comparable score was available; uplift unavailable | 91.5% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | 46.4% → 90.0% (+43.6 points) | 55.5% → 78.3% (+22.8 points) |
| Efficiency | 86.8% — baseline ran, but no comparable score was available; uplift unavailable | 86.0% — baseline ran, but no comparable score was available; uplift unavailable |
How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
Comparison details
Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict.
| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation |
|---|---|---|---|---|
| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| claude-code | Efficiency | Token-only comparison | 97.8% → 96.1% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| codex | Efficiency | Token-only comparison | 89.3% → 91.2% (+1.9 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
Token Usage
Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
|---|---|---|---|---|---|---|
| claude-code | All cases | 1,207,654 | 1,514,844 | -307,190 | -20.28% | skill 10/10; base 10/10 |
| claude-code | setup-eval-001-underspecified | 116,199 | 107,379 | +8,820 | +8.21% | skill 1/1; base 1/1 |
| claude-code | setup-eval-002-use-existing-scripts | 75,399 | 146,574 | -71,175 | -48.56% | skill 1/1; base 1/1 |
| claude-code | setup-eval-003-auth-secret-cookies | 117,251 | 143,673 | -26,422 | -18.39% | skill 1/1; base 1/1 |
| claude-code | setup-eval-004-postgres-port-host-only | 115,760 | 142,520 | -26,760 | -18.78% | skill 1/1; base 1/1 |
| claude-code | setup-eval-005-mcp-already-up | 115,272 | 181,282 | -66,010 | -36.41% | skill 1/1; base 1/1 |
| claude-code | setup-eval-006-vault-partial | 107,366 | 142,761 | -35,395 | -24.79% | skill 1/1; base 1/1 |
| claude-code | setup-eval-007-embed-triplet-mismatch | 214,291 | 143,259 | +71,032 | +49.58% | skill 1/1; base 1/1 |
| claude-code | setup-eval-008-connect-source-workflow | 116,719 | 221,039 | -104,320 | -47.20% | skill 1/1; base 1/1 |
| claude-code | setup-eval-009-env-vs-ui-connections | 112,577 | 143,239 | -30,662 | -21.41% | skill 1/1; base 1/1 |
| claude-code | setup-eval-010-helm-install | 116,820 | 143,118 | -26,298 | -18.38% | skill 1/1; base 1/1 |
| codex | All cases | 526,689 | 624,717 | -98,028 | -15.69% | skill 10/10; base 10/10 |
| codex | setup-eval-001-underspecified | 29,618 | 71,920 | -42,302 | -58.82% | skill 1/1; base 1/1 |
| codex | setup-eval-002-use-existing-scripts | 33,946 | 49,034 | -15,088 | -30.77% | skill 1/1; base 1/1 |
| codex | setup-eval-003-auth-secret-cookies | 46,937 | 54,703 | -7,766 | -14.20% | skill 1/1; base 1/1 |
| codex | setup-eval-004-postgres-port-host-only | 51,236 | 44,377 | +6,859 | +15.46% | skill 1/1; base 1/1 |
| codex | setup-eval-005-mcp-already-up | 67,020 | 78,344 | -11,324 | -14.45% | skill 1/1; base 1/1 |
| codex | setup-eval-006-vault-partial | 92,834 | 57,924 | +34,910 | +60.27% | skill 1/1; base 1/1 |
| codex | setup-eval-007-embed-triplet-mismatch | 47,130 | 45,875 | +1,255 | +2.74% | skill 1/1; base 1/1 |
| codex | setup-eval-008-connect-source-workflow | 63,884 | 76,064 | -12,180 | -16.01% | skill 1/1; base 1/1 |
| codex | setup-eval-009-env-vs-ui-connections | 47,432 | 80,059 | -32,627 | -40.75% | skill 1/1; base 1/1 |
| codex | setup-eval-010-helm-install | 46,652 | 66,417 | -19,765 | -29.76% | skill 1/1; base 1/1 |
| ALL AGENTS | Dataset aggregate | 1,734,343 | 2,139,561 | -405,218 | -18.94% | skill 20/20; base 20/20 |
Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. Change is 100 * (with - without) / without; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated.
Tier Status
| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | PASSED WITH OBSERVATIONS | 11 validator(s); 6 finding(s) |
| Tier 2 | Semantic deduplication | PASSED | 2 validator(s); 0 finding(s) |
| Tier 3 | Live agent evaluation | PASS | 2 agent(s); 10 task(s) |
Findings and Observations
<details> <summary>Show detailed findings and successful checks</summary>- MEDIUM QUALITY/quality_correctness: Resource-based skill lacks documentation of available resources (
skills/nvidia-ontology-setup/SKILL.md) - MEDIUM SECURITY/Autonomous Decision Making (EA2): Excessive Agency: Never ask the user (
SKILL.md:65) - MEDIUM SECURITY/External Transmission (E1): Data Exfiltration: curl -s -o /dev/null -w '%{http_code}\n' "$AUTO_ONTOLOGY_API_URL/.well-known/oauth-authorization-server"
`200` is required. Anything else: run the frontend from the checkout
(`pnpm dev`) and poin (`SKILL.md:203`)
- **LOW** QUALITY/quality_discoverability: Skill uses exclusivity language that conflicts with composability (`skills/nvidia-ontology-setup/SKILL.md`)
- **LOW** QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/nvidia-ontology-setup/SKILL.md`)
- 1 additional finding(s) are available in the full evaluation artifacts.
</details>
## Scoring Methodology
<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>
| Dimension | Question | Scored signals |
|---|---|---|
| Security | Is it safe to use? | `security` (100%) |
| Correctness | Is the answer correct? | `accuracy` (100%) |
| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
Signals present in this run:
- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy` (Accuracy): final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
</details>
## Freshness
Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.