skilld
All skills
nvidia avatar

/physical-ai-infrastructure-setup-and-resilient-scaling

@6c6fc09
by NVIDIA Corporationnvidia/skills3.5k stars
424

Use when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local MicroK8s or Azure AKS, including Kubernetes clusters, inference endpoint deployment, OSMO deployment, workload submission readiness, and infrastructure failure recovery. Trigger keywords: physical ai infrastructure, resilient scaling, SDG infrastructure, microk8s, azure aks, NVCF deployment, NIM Operator, OSMO deploy, workflow scaling. Don't trigger for: OSMO log summarization or workload-only operations unless infrastructure setup, scaling, validation, or recovery is requested.

Ask your Agent

Use this Skill: https://skilld.dev/gh/nvidia/skills/physical-ai-infrastructure-setup-and-resilient-scaling

This session only. Nothing lands on disk.

componentsdriverreference.md

≈2.2k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Infrastructure Driver Notes

Setup the infrastructure needed to run NVIDIA physical AI synthetic data generation (SDG) workflows.

Infrastructure stack

This infrastructure skill covers the Kubernetes stack: OSMO for workflow execution plus an inference provider of choice. Kubernetes deployment may be a proper cluster like Azure Kubernetes Engine, or a single-node deployment like MicroK8s. Docker-only workflows are out of scope for this infrastructure skill; route those to the workload skill directly.

Full stack component selection

Pick exactly one option per component. The sub-skill handles its own layer — this root only sequences them.

1. Kubernetes

Option Sub-skill When
MicroK8s ../cluster-microk8s/reference.md Local dev, single-node CPU+NVCF or GPU+NIM
Azure ../cluster-azure/reference.md Production, multi-node, managed

2. Orchestration (OSMO)

Sub-skill is dictated by component selection in 1. Kubernetes.

Kubernetes Component Sub-skill When
MicroK8s ../osmo-k8s/reference.md MicroK8s cluster
Azure ../osmo-azure/reference.md Azure AKS cluster

Both OSMO install paths leave a persistent port-forward on localhost:9000 — downstream skills (workload) rely on it and will not start their own.

OSMO storage is configured in-band by the upstream deploy script - all options run deploy-osmo-minimal.sh in the main OSMO repo which in turn runs configure-storage.sh.

3. Inference Provider

Option Sub-skill When
NIM Operator (in-cluster) ../inference-nim-operator/reference.md GPU in cluster; low-latency/air-gapped
NVCF (pre-deployed cloud) ../inference-nvcf/reference.md Cloud inference, no cluster GPU budget; needs NVCF_API_KEY
Azure AI Foundry (serverless) ../inference-azure/reference.md Azure cluster, pay-per-token
None — Workflow does not need inference endpoints

4. Workload

Any skill exposing an OSMO workflow YAML can be submitted via osmo workflow submit. OSMO workflow YAMLs are largely portable across environments, with exception to those that may specify an inference provider.

Option Skill Notes
Video Data Augmentation (VDA) skills/physical-ai-video-data-augmentation/SKILL.md End-to-end SDG: augmentation + auto-labeling
Defect Image Generation (AOI) skills/physical-ai-defect-image-generation/SKILL.md PCBA / metal / glass defect image generation: Day 0 from CAD (texture / good-image / structural-defect) or Day 1 from clean inspection images
NRE (Neural Reconstruction) nre (canonical: NVIDIA/nurec-skills) Train / render / export via NRE Docker CLI (covers 26.02 + 26.04)
NCore data conversion ncore (canonical: NVIDIA/nurec-skills) Convert sensor datasets to NCore V4
NuRec carline adaptation skills/carline-adaptation/SKILL.md Adapt USDZ reconstructions to new camera rigs
Asset Harvester asset-harvester (canonical: NVIDIA/nurec-skills) 3DGS asset extraction from NCore clips
Custom / external spec any YAML path osmo workflow submit /abs/path/spec.yaml --pool <pool>

Compatibility matrix

Not every option combines with every other — enforce these when picking:

Cluster NIM Operator NVCF AI Foundry
MicroK8s ✅ ✅ ❌ (Foundry requires Azure identities)
Azure ✅ ✅ ✅

Deployment

Component Dependencies

  • 1. Kubernetes is required and blocks everything else.
  • 2. OSMO and 3. Inference Provider can be deployed in parallel once 1. Kubernetes is deployed. Deploy these two in parallel; gate everything downstream on their readiness.
  • 4. Workload: Workload will specify the compute requirements, the OSMO workflow spec, and any inference endpoints it needs. Ensure the compute requirements are available in OSMO prior to submission, and any inference endpoints are reachable.

Pipeline → inference requirement discovery

Over-deploying wastes GPUs and oversubscribes small pools. Derive the minimum set before running the inference stage:

  1. Scan the chosen pipeline spec's default-values + task args for URL-shaped references:
    • *.osmo-nims.svc.cluster.local → NIMService of that name (NIM Operator)
    • api.nvcf.nvidia.com/* → NVCF function ID
    • *.inference.ai.azure.com / *.cognitiveservices.azure.com → Foundry endpoint
  2. Pass the filtered set to the sub-skill: NIM_SERVICES="<a> <b>", the matching *_URL / function ID envs (NVCF), or install.sh --endpoint-name <name> (Foundry).
  3. If a required capability isn't in the chosen backend's catalog, STOP and surface — never substitute.

Decision prompt

Prompt for stages 1, 3, and 4 (stage 2 resolves from Kubernetes choice):

  1. Kubernetes: MicroK8s | Azure
  2. OSMO: Resolves based on 1. Kubernetes
  3. Inference Provider: NIM Operator | NVCF | Azure AI Foundry
  4. Workload: NuRec Carline Adaptation | Video Data Augmentation | Defect Image Generation | Custom spec (path)

Reject invalid cluster/inference pairs per the matrix. Custom spec → osmo workflow submit <path>.

Prerequisites

Each sub-skill owns its own prerequisites. Before provisioning anything, read the Prerequisites section of the SELECTED components, enumerate every selected preflight, run them (Azure targets first run components/azure-access/reference.md), then compile a single implementation plan. Resolve everything up front - don't prompt the user mid-deploy. Derive what you can (caller IP for allowed_cidr, subscription ID from az account show); TF outputs are deploy-time inputs, not preflight inputs.

Preflight is before flight: no cluster API, Terraform outputs, Helm releases, OSMO pools, or workflow state are expected. Stage deploy/verify gates check those after resources exist.

Workflow submit/query requires components/osmo-cli/reference.md; run its scripts/preflight.sh with the resolved prerequisites.

Prior to provisioning Kubernetes, collect compute requirements from the SELECTED workload skill. Check these compute requirements against the SELECTED environment before proceeding.

Prompt the user only for values you truly can't derive such as API keys.

Secrets should be stored in ${REPO_ROOT}/.env. Stage scripts source it via ${REPO_ROOT}/.env where REPO_ROOT is this repo's root.

Runtime Routing

If running under OpenClaw and any selected Azure stage needs az auth, read ../openclaw-azure-login/reference.md before resolving Azure prerequisites.

Verification gates (mandatory, per stage)

Each sub-skill has a Verify section — run it to completion before moving to the next stage.

After stage Must run + confirm
1. Kubernetes kubectl cluster-info, all nodes Ready. GPU paths: GPU capacity advertised (kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:'.status.allocatable.nvidia\.com/gpu'). CPU+NVCF paths: kubectl get runtimeclass nvidia -o jsonpath='{.handler}' returns runc.
2 Inference Every endpoint YOUR pipeline references is reachable. NIM /v1/health/ready should return 200. NVCF preflight treats any HTTP response other than 000 as endpoint reachability, while authenticated worker calls must still satisfy the task-specific response check. Foundry endpoint reachable with az cognitiveservices account keys list credential.
3 Orchestration helm status -n osmo-minimal osmo-minimal, kubectl get pods -n osmo-minimal, osmo pool list default ONLINE, port-forward watchdogs up (pgrep -f 'osmo-pf-watchdog:'), OSMO storage configured (osmo config show WORKFLOW, osmo config show DATASET, osmo credential list)
3 Smoke The upstream deploy-osmo-minimal.sh runs verify.sh (verify-hello workflow) as its final step — verify-hello-N COMPLETED is the gate. CPU instances pass SKIP_GPU=1 to the deploy script. Re-run via "$OSMO_DIR/deployments/scripts/verify.sh" (where $OSMO_DIR=$HOME/.cache/physical-ai/osmo). This catches backend-operator / storage mis-wires before any pipeline runs — do not skip.
4 Workload osmo workflow query <id> → COMPLETED with every task green. FAILED/CANCELLED/TERMINATED → osmo workflow events + osmo workflow logs, NOT "retry and hope".

Failing gate → AGENTS.md rule 5 (Stop on red gate) + rule 4 (Config > Script > Skill).

Source: SKILL.md on GitHub

Third-party checks

2 warnings3mo3 checks · Risk SAFE
  • Gen Agent Trust Hub3mo

    This skill from NVIDIA provides a comprehensive set of tools and instructions for setting up Physical AI infrastructure on Azure AKS or local MicroK8s clusters. The analysis found no malicious patterns; all external downloads are from trusted or well-known services (NVIDIA, Microsoft, HashiCorp, Hugging Face), and the orchestration design prioritizes user visibility through a partitioned multi-agent architecture.

  • Socket3mo

    1 alert: gptAnomaly

  • Snyk3mo

    Risk: MEDIUM · 1 issue

Provenance

Signed by skilld at 6c6fc09. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago

Capability

version
1.0.0
tools
[
  "Read",
  "Shell"
]
Other metadata
compatibility
Requires the selected component prerequisites, usually kubectl plus either MicroK8s or Azure CLI/Terraform, and OSMO or inference credentials for the chosen target.
metadata
{
  "author": "NVIDIA Physical AI",
  "tags": [
    "physical-ai",
    "infrastructure",
    "kubernetes",
    "azure",
    "microk8s",
    "osmo",
    "nim-operator",
    "scaling"
  ],
  "domain": "ai-ml",
  "languages": [
    "bash",
    "hcl",
    "yaml"
  ]
}

README badge

README badge for nvidia/skills/physical-ai-infrastructure-setup-and-resilient-scaling