Index of every page on this site: llms.txt
All skills
nvidia avatar

/nemo-automodel-recipe-development

@6b75e9e
by NVIDIA Corporationnvidia/skills4.1k stars
547
Run it yourself

Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.

Use this Skill: https://skilld.dev/gh/nvidia/skills/nemo-automodel-recipe-development

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

BENCHMARK.md

≈1.9k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Skill Benchmark: nemo-automodel-recipe-development

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

  • Skill: nemo-automodel-recipe-development
  • Evaluation date: 2026-09-11
  • Evaluator version: 1.5.6
  • Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)
  • Tasks: 3 evaluation tasks (3 positive)
  • Dataset digest: sha256:471b2da53028232964af5684eb94a1c253ca6d6dc90b78082b31b0d0bc2ee6a0 (skill-evaluator-dataset-snapshot/1)
  • Attempts per task: 3
  • Environment: k8s-sandbox
  • Tier 2 evidence: required for publication
  • Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

  • is safe to use;
  • produces correct answers;
  • is discovered and activated when needed;
  • helps the agent complete the user's goal and expected workflow; and
  • avoids wasted skill and tool usage.

Results at a Glance

Measure Claude Code (Baseline → Skill Uplift) Codex (Baseline → Skill Uplift)
Overall 99.1% — baseline ran, but no comparable score was available; uplift unavailable 98.7% — baseline ran, but no comparable score was available; uplift unavailable
Security 100.0% → 100.0% (±0.0 points) 100.0% → 100.0% (±0.0 points)
Correctness 86.7% → 100.0% (+13.3 points) 86.7% → 100.0% (+13.3 points)
Discoverability 100.0% — baseline ran, but no comparable score was available; uplift unavailable 95.0% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness 48.9% → 97.2% (+48.3 points) 46.3% → 100.0% (+53.7 points)
Efficiency 98.4% — baseline ran, but no comparable score was available; uplift unavailable 98.5% — baseline ran, but no comparable score was available; uplift unavailable

How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

Agent Dataset case With skill Without skill Delta Change Coverage
claude-code All cases 192,649 2,739,204 -2,546,555 -92.97% skill 3/3; base 3/3
claude-code nemo-automodel-recipe-development-001-new-finetune-recipe 64,727 375,307 -310,580 -82.75% skill 1/1; base 1/1
claude-code nemo-automodel-recipe-development-002-yaml-target-pattern 64,012 1,833,406 -1,769,394 -96.51% skill 1/1; base 1/1
claude-code nemo-automodel-recipe-development-003-validation-checkpointing 63,910 530,491 -466,581 -87.95% skill 1/1; base 1/1
codex All cases 90,138 192,645 -102,507 -53.21% skill 3/3; base 3/3
codex nemo-automodel-recipe-development-001-new-finetune-recipe 30,297 111,660 -81,363 -72.87% skill 1/1; base 1/1
codex nemo-automodel-recipe-development-002-yaml-target-pattern 30,064 37,718 -7,654 -20.29% skill 1/1; base 1/1
codex nemo-automodel-recipe-development-003-validation-checkpointing 29,777 43,267 -13,490 -31.18% skill 1/1; base 1/1
ALL AGENTS Dataset aggregate 282,787 2,931,849 -2,649,062 -90.35% skill 6/6; base 6/6

Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. N/A means the relevant trajectory counters were not available; coverage is never estimated.

Tier Status

Tier Purpose Status Evidence
Tier 1 Static validation PASSED WITH OBSERVATIONS 11 validator(s); 8 finding(s)
Tier 2 Semantic deduplication PASSED 2 validator(s); 0 finding(s)
Tier 3 Live agent evaluation PASS 2 agent(s); 3 task(s)

Findings and Observations

<details> <summary>Show detailed findings and successful checks</summary>
  • MEDIUM SECURITY/Behavior Manipulation (P4): Prompt Injection: always prefer this over (SKILL.md:329)
  • LOW QUALITY/quality_discoverability: Description doesn't mention WHEN to use this skill (skills/nemo-automodel-recipe-development/SKILL.md)
  • LOW QUALITY/quality_discoverability: No '## Purpose' section (skills/nemo-automodel-recipe-development/SKILL.md)
  • LOW QUALITY/quality_reliability: No prerequisites/requirements documented (skills/nemo-automodel-recipe-development/SKILL.md)
  • LOW QUALITY/quality_reliability: No limitations documented (skills/nemo-automodel-recipe-development/SKILL.md)
  • 3 additional finding(s) are available in the full evaluation artifacts.
</details>

Scoring Methodology

<details> <summary>Show dimension definitions, source signals, and thresholds</summary>
Dimension Question Scored signals
Security Is it safe to use? security (100%)
Correctness Is the answer correct? accuracy (100%)
Discoverability Was the right skill loaded when needed? skill_execution (100%)
Effectiveness Did the skill help complete the task? goal_accuracy (50%) + behavior_check (50%)
Efficiency Did it avoid wasted tool calls and token usage? skill_efficiency (50%) + token_efficiency (50%)
  • Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
  • Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
  • Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
  • The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
  • Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).
  • Efficiency is 50% tool-call productivity (the backward-compatible skill_efficiency wire id) and 50% token_efficiency. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

  • security (Security): unsafe operations, secret leakage, and unauthorized access.
  • skill_execution (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
  • skill_efficiency (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
  • accuracy (Accuracy): final-answer correctness against the reference answer.
  • goal_accuracy (Goal Accuracy): whether the user's goal was achieved.
  • behavior_check (Behavior Check): whether the expected workflow behavior was followed.
  • token_efficiency (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
</details>

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides instructions for developing and modifying NeMo AutoModel recipes using YAML configurations. It is authored by NVIDIA and focuses on technical workflows for machine learning training and evaluation. The skill utilizes standard framework patterns for configuration and local validation. No malicious patterns or security risks were detected.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 6b75e9e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
Other metadata
when_to_use
Creating or modifying training, SFT, or eval recipes, adding new YAML config fields, debugging recipe construction or trainer issues, or understanding the recipe execution flow.
metadata
{
  "author": "NVIDIA",
  "tags": [
    "nemo-automodel",
    "recipe-development"
  ]
}

README badge

README badge for nvidia/skills/nemo-automodel-recipe-development