All skills
nvidia avatar

/physical-ai-infrastructure-setup-and-resilient-scaling

@6c6fc09
by NVIDIA Corporationnvidia/skills3.5k stars
424

Use when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local MicroK8s or Azure AKS, including Kubernetes clusters, inference endpoint deployment, OSMO deployment, workload submission readiness, and infrastructure failure recovery. Trigger keywords: physical ai infrastructure, resilient scaling, SDG infrastructure, microk8s, azure aks, NVCF deployment, NIM Operator, OSMO deploy, workflow scaling. Don't trigger for: OSMO log summarization or workload-only operations unless infrastructure setup, scaling, validation, or recovery is requested.

Use this Skill: https://skilld.dev/gh/nvidia/skills/physical-ai-infrastructure-setup-and-resilient-scaling

This session only. Nothing lands on disk.

componentsosmo-cliagentslogs-reader.md

≈886 tokens on demand. Your agent reads this file only when SKILL.md points to it.

OSMO Logs Reader Agent

Spawn a general-purpose subagent and pass these instructions as the prompt.

You are a subagent invoked by the main OSMO agent. Your sole job is to fetch and summarize logs for a specific workflow, then return a concise digest that the main agent can use without holding large raw logs in context.


Inputs

The main agent will tell you:

  • Workflow ID — the OSMO workflow identifier (e.g. my-workflow-abc123)
  • Tasks to read — either:
    • A list of specific task names (e.g. ["train", "eval"])
    • "all" — meaning fetch overall (un-split) logs
    • "auto" — determine the task list yourself by querying the workflow

Step 1: Determine task list (only when told "auto")

If the main agent said "auto", query the workflow to find its tasks:

osmo workflow query <workflow_id> --format-type json

Read the tasks field from the JSON response. If there are ≤ 5 tasks, treat each as an individual target. If there are > 5, fall back to fetching overall logs (treat as "all").


Step 2: Fetch logs

All log-fetching commands stream live output, so run each with a 5-second timeout and use whatever was captured — do not wait for the stream to end.

Overall logs (when tasks = "all" or > 5 tasks):

osmo workflow logs <workflow_id> -n 200

Per-task logs (when you have a specific list of 1–5 task names):

Fetch each task in parallel:

osmo workflow logs <workflow_id> --task <task_name> -n 200

Step 3: Fetch workflow spec (optional, when logs are ambiguous)

If the logs alone don't make it clear what stage the workflow is at — for instance, the log output is sparse, shows only container startup messages, or references config keys you don't recognize — fetch the spec for additional context:

osmo workflow spec <workflow_id> --template

Use the spec to understand what the workflow is supposed to do, so you can interpret partial or noisy logs more accurately. Do not surface the raw spec YAML to the main agent; use it only to inform your summary.


Step 4: Summarize and return

For each task (or for the overall log if un-split), write a compact summary block. Keep each block short — the goal is to preserve context for the main agent, not to reproduce the raw logs.

Return your response in this format:

## Workflow: <workflow_id>

### Task: <task_name>  (or "Overall" if un-split)
- **Stage / progress**: What stage is this task at? (e.g. "downloading dataset",
  "training step 840/1000 — 84% complete", "completed successfully")
- **Key events**: Any notable milestones, warnings, or errors seen in the logs
  (1–3 bullet points, skip if nothing notable)
- **Errors**: If the task has failed or shows error output, include up to 50
  lines of the error logs verbatim. Prefer the most recent and most relevant
  error lines (stack traces, exception messages, fatal log lines). Otherwise
  omit this field.

### Task: <next_task_name>
...

Guidelines:

  • If a task is still at container startup with no meaningful progress yet, say so in one line (e.g. "Container starting, no application output yet").
  • If training is in progress, include the latest step/epoch count and loss if visible.
  • If the task completed, say so and note any output URLs or dataset paths mentioned.
  • Never dump raw log lines except for error output (up to 50 lines).
  • Keep the entire response under ~300 words so the main agent can use it efficiently without hitting context limits.

Source: SKILL.md on GitHub

2 warnings3mo3 checks · Risk SAFE
  • Gen Agent Trust Hub3mo

    This skill from NVIDIA provides a comprehensive set of tools and instructions for setting up Physical AI infrastructure on Azure AKS or local MicroK8s clusters. The analysis found no malicious patterns; all external downloads are from trusted or well-known services (NVIDIA, Microsoft, HashiCorp, Hugging Face), and the orchestration design prioritizes user visibility through a partitioned multi-agent architecture.

  • Socket3mo

    1 alert: gptAnomaly

  • Snyk3mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 6c6fc09. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago
version
1.0.0
tools
[
  "Read",
  "Shell"
]
Other metadata
compatibility
Requires the selected component prerequisites, usually kubectl plus either MicroK8s or Azure CLI/Terraform, and OSMO or inference credentials for the chosen target.
metadata
{
  "author": "NVIDIA Physical AI",
  "tags": [
    "physical-ai",
    "infrastructure",
    "kubernetes",
    "azure",
    "microk8s",
    "osmo",
    "nim-operator",
    "scaling"
  ],
  "domain": "ai-ml",
  "languages": [
    "bash",
    "hcl",
    "yaml"
  ]
}

README badge

README badge for nvidia/skills/physical-ai-infrastructure-setup-and-resilient-scaling