Azure Cluster

Prerequisites

Azure CLI >= 2.60.0 with components/azure-access/reference.md completed
PIM/RBAC active for the target subscription (Owner, or Contributor plus User Access Administrator)
Terraform >= 1.9.8, < 2.0
kubectl >= 1.31.0
helmfile >= 0.165.0
curl >= 7.68.0

Security Model

AKS API server is public but restricted to allowed_cidr (your IP/32). All deployment tools (kubectl, helm, osmo CLI) run from your local machine. No jumpbox needed.

Supporting files

Path	Use	When
`scripts/terraform.tfvars.example`	Read/copy	Template for local-only `deploy.tfvars`.
`scripts/{main,variables,outputs,versions}.tf`	Runtime config	Active Terraform root used by all `terraform -chdir=.../scripts` commands below.
`scripts/preflight.sh`	Run first	Checks `az` subscription/provider read access, CLI versions, binaries, the tfvars template, and completed local tfvars when present.
`scripts/setup.sh`	Run	Installs GPU Operator and swaps the default RWX StorageClass after AKS is reachable.
`scripts/helmfile.yaml`	Runtime config	Consumed by `scripts/setup.sh`; do not run directly unless debugging setup.
`scripts/storage-class-nfs.yaml`	Runtime config	Applied by `scripts/setup.sh` for the `azurefile-nfs` default StorageClass.
`scripts/system_node_capacity_test.sh`	Run	Optional post-deploy capacity check for system node sizing.
`terraform/`	Legacy/read-only	Older Terraform layout kept for compatibility notes and prerequisites. Do not use for new applies unless explicitly migrating an old state.

Deployment

Run preflight

REPO=$(git rev-parse --show-toplevel)
"$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts/preflight.sh"

Generate deploy.tfvars

deploy.tfvars is local-only (gitignored); terraform.tfvars.example is the tracked template. On a fresh clone, cp terraform.tfvars.example deploy.tfvars and fill in subscription_id (from az account show --query id -o tsv) and allowed_cidr (list containing your public IP /32). Other values (region, VM sizes, GPU min/max, priority, pg SKU, K8s version) are edited here when workload needs change — defaults live in variables.tf. Post-apply IP drift is handled by setup.sh, which re-applies TF against the live AKS resource.

User-overridable values (ask before assuming; defaults in variables.tf):

Variable	Decide when
`location`	Region pinned by quota/data residency
`system_vm_size`	Default D16; D8 is too small
`gpu_vm_size`	Pipeline's GPU model fit + quota (H100 for cosmos, T4/A10 for text)
`gpu_priority`	`Regular` vs `Spot` (dev can use Spot)
`gpu_min` / `gpu_max`	≥ number of standing NIMServices; peak workload
`kubernetes_version`	AKS-supported; match helm chart requirements
`pg_sku`	OSMO load

Check quotas

Check CPU + GPU quota for the chosen location + SKUs in deploy.tfvars before terraform apply. Skill matches name.value (Azure's SKU codename) against the localized usage list — az vm list-usage returns hundreds of rows, the filter must pick the family of system_vm_size / gpu_vm_size:

REPO=$(git rev-parse --show-toplevel)
LOCATION=$(awk -F'"' '/^location/{print $2}' "$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts/deploy.tfvars")
GPU_SKU=$(awk -F'"'  '/^gpu_vm_size/{print $2}' "$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts/deploy.tfvars")

az vm list-usage -l "$LOCATION" -o table \
  --query "[?contains(name.value, 'standardDDSv5')].{name:name.localizedValue, used:currentValue, limit:limit}"

az vm list-usage -l "$LOCATION" -o table \
  --query "[?contains(name.value, 'NCadsH100') || contains(name.value, 'NVADSA10') \
           || contains(name.value, 'NCASv3_T4') || contains(name.value, 'NCADSA100')] \
           .{name:name.localizedValue, used:currentValue, limit:limit}"

name.value filters above cover the SKU families in variables.tf; add the family for any new gpu_vm_size you introduce. Request increases via Azure Portal → Subscriptions → Usage + quotas. STOP before applying if the total limit - used is below what TF will request.

Apply the Terraform

Absolute paths only — no cd. -chdir makes Terraform cwd-agnostic.

REPO=$(git rev-parse --show-toplevel)
terraform -chdir="$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts" init
terraform -chdir="$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts" plan  -var-file=deploy.tfvars
terraform -chdir="$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts" apply -var-file=deploy.tfvars

Connect to AKS cluster

REPO=$(git rev-parse --show-toplevel)
RG=$(terraform -chdir="$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts" output -raw resource_group)
AKS=$(terraform -chdir="$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts" output -raw aks_name)
az aks get-credentials -g "$RG" -n "$AKS"
kubectl get nodes

Install GPU Operator + RWX default StorageClass on AKS cluster

"$(git rev-parse --show-toplevel)/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/cluster-azure/scripts/setup.sh"

RBAC propagation can take 1–2 minutes; a fresh cluster's first NFS PVC may retry once before binding.

Verify

Check general Kubernetes state. Pods should be healthy and running.

kubectl get pods -A

Check GPUs are available and allocatable under nvidia.com/gpu for all nodes.

kubectl get nodes
kubectl describe node <node-name>

/physical-ai-infrastructure-setup-and-resilient-scaling

componentscluster-azurereference.md