Skill Benchmark: doca-telemetry-exporter
✅ Overall verdict: PASS — Recommended for publication
Publication Recommendation
Recommended for publication based on the completed evaluation evidence in this report.
Evaluation Metadata
- Skill:
doca-telemetry-exporter - Evaluation date: 2026-07-29
- Evaluator version:
0.9.0 - Agents: Claude Code (
aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5) - Tasks: 4 evaluation tasks (3 positive, 1 negative)
- Dataset digest:
sha256:bdb053d3eebd3787bba933e197d64385efe318e5e5666e43c6300e91e2c8f42e(skill-evaluator-dataset-snapshot/1) - Attempts per task: 1
- Environment:
k8s-sandbox - Tier 3 evidence: required for publication
Each task attempt ran in its own isolated sandbox pod.
What This Report Answers
The three-tier evaluation checks whether the skill:
- is safe to use;
- produces correct answers;
- is discovered and activated when needed;
- helps the agent complete the user's goal and expected workflow; and
- avoids wasted skill and tool usage.
Results at a Glance
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---|---|
| Overall | 57% → 98% (+40 points) | 64% → 97% (+34 points) |
| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
| Correctness | 40% → 100% (+60 points) | 75% → 100% (+25 points) |
| Discoverability | 62% → 100% (+38 points) | 59% → 92% (+33 points) |
| Effectiveness | 30% → 94% (+64 points) | 54% → 94% (+40 points) |
| Efficiency | 53% → 94% (+41 points) | 30% → 100% (+70 points) |
How to read this table: baseline is the same task attempted without the target skill. Uplift is skill score - baseline score, shown in percentage points.
Example: 47% → 92% (+45 points) means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
Tier Status
| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | PASSED WITH OBSERVATIONS | 1 validator(s); 7 finding(s) |
| Tier 2 | Semantic deduplication | NOT RUN | No result was recorded |
| Tier 3 | Live agent evaluation | PASS | 2 agent(s); 4 task(s) |
Findings and Observations
<details> <summary>Show detailed findings and successful checks</summary>- MEDIUM SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (
skills/libs/doca-telemetry-exporter) - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (
skills/libs/doca-telemetry-exporter/SKILL.md) - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (
skills/libs/doca-telemetry-exporter/SKILL.md) - MEDIUM SCHEMA/author_missing: Author not specified in metadata (
skills/libs/doca-telemetry-exporter/SKILL.md) - LOW SCHEMA/unexpected_file: Unexpected 'CAPABILITIES.md' in skill root (
skills/libs/doca-telemetry-exporter/CAPABILITIES.md) - 2 additional finding(s) are available in the full evaluation artifacts.
Scoring Methodology
<details> <summary>Show dimension definitions, source signals, and thresholds</summary>| Dimension | Question | Scored signals |
|---|---|---|
| Security | Is it safe to use? | security (100%) |
| Correctness | Is the answer correct? | accuracy (100%) |
| Discoverability | Was the right skill loaded when needed? | skill_execution (100%) |
| Effectiveness | Did the skill help complete the task? | goal_accuracy (50%) + behavior_check (50%) |
| Efficiency | Did it avoid wasted tool or skill usage? | skill_efficiency (100%) |
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (
goal_accuracy) and expected workflow adherence (behavior_check). - Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
Signals present in this run:
security(Security): unsafe operations, secret leakage, and unauthorized access.skill_execution(Skill Execution): whether the expected skill was found and executed.skill_efficiency(Efficiency): routing quality, workspace-aware skill reads, and productive tool use.accuracy(Accuracy): final-answer correctness against the reference answer.goal_accuracy(Goal Accuracy): whether the user's goal was achieved.behavior_check(Behavior Check): whether the expected workflow behavior was followed.
Freshness
Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.