Index of every page on this site: llms.txt
All skills
nvidia avatar

/nemo-mbridge-multi-node-slurm

@6b75e9e
by NVIDIA Corporationnvidia/skills4.1k stars
547
Run it yourself

Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.

  • 6 files
  • 34.3 KB
  • Apache-2
  • Updated 6 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/nvidia/skills/nemo-mbridge-multi-node-slurm

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

skill-card.md

≈935 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Description: <br>

Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation. <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache-2.0 <br>

Use Case: <br>

Developers and engineers converting single-node training scripts to multi-node Slurm sbatch jobs, scaling distributed training, and debugging common multi-node failures such as NCCL timeouts and OOM errors. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

Skill Output: <br>

Output Type(s): [Shell commands, Configuration instructions] <br> Output Format: [Markdown with inline bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Agents Used: <br>

  • Claude Code (claude-code) <br>
  • Codex (codex) <br>

Evaluation Tasks: <br>

Evaluated against 1 evaluation task (1 positive skill-activation case). <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

  • Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
  • Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
  • Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
  • Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
  • Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>

  • security: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
  • skill_execution: Verifies that the agent loaded the expected skill and workflow. <br>
  • skill_efficiency: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
  • accuracy: Grades final-answer correctness against the reference answer. <br>
  • goal_accuracy: Checks whether the overall user task completed successfully. <br>
  • behavior_check: Verifies expected behavior steps, including safety expectations. <br>
  • token_efficiency: Compares token usage with and without the skill. <br>

Evaluation Results: <br>

Dimension Num claude-code codex
Security 2 100% (+0%) 100% (+0%)
Correctness 2 100% (+0%) 88% (+5%)
Discoverability 2 100% (+0%) 62% (+0%)
Effectiveness 2 97% (+3%) 95% (+8%)
Efficiency 2 92% (-0%) 60% (+1%)

Skill Version(s): <br>

b0f64d72 (source: git SHA, committed 2026-06-02) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 6b75e9e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 6 months ago
Other metadata
when_to_use
Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL/launch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.

README badge

README badge for nvidia/skills/nemo-mbridge-multi-node-slurm