/azure-kubernetes

AKS Spot Node Pools

Recommend and create Spot VM node pools for batch, dev/test, or fault-tolerant workloads (60-90% cost reduction vs regular nodes).

Check Existing Node Pools

az aks nodepool list \
  --cluster-name "<CLUSTER_NAME>" --resource-group "<RESOURCE_GROUP>" \
  --query "[].{name:name, vmSize:vmSize, priority:scaleSetPriority, count:count, mode:mode}" \
  -o table

Identify Spot-Suitable Workloads

Before creating a Spot pool, identify which workloads can tolerate interruptions:

# List deployments without PodDisruptionBudgets (single-replica or no PDB = higher eviction risk)
kubectl get deployments --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.replicas == 1) | "\(.metadata.namespace)/\(.metadata.name)"'

# Check which pods already have spot tolerations
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.tolerations[]?.key == "kubernetes.azure.com/scalesetpriority") | "\(.metadata.namespace)/\(.metadata.name)"'

Use the suitability table below to decide which workloads to migrate.

Mixed Node Pool Pattern (Spot + Regular)

For workloads that need resilience but want cost savings, use a mixed approach:

# Keep existing regular node pool as fallback (min 1-2 nodes)
az aks nodepool update \
  --cluster-name "<CLUSTER_NAME>" --resource-group "<RESOURCE_GROUP>" \
  --name "<REGULAR_POOL>" \
  --enable-cluster-autoscaler --min-count 1 --max-count 3

# Add Spot pool for the majority of workload capacity
# -1 means pay up to on-demand price (no cap); set e.g. 0.05 to cap hourly spend
az aks nodepool add \
  --cluster-name "<CLUSTER_NAME>" --resource-group "<RESOURCE_GROUP>" \
  --name "<SPOT_POOL_NAME>" \
  --priority Spot --eviction-policy Delete --spot-max-price -1 \
  --node-vm-size "<VM_SIZE>" \
  --node-count 3 --min-count 0 --max-count 10 \
  --enable-cluster-autoscaler \
  --node-taints "kubernetes.azure.com/scalesetpriority=spot:NoSchedule" \
  --labels "kubernetes.azure.com/scalesetpriority=spot"

Pods that tolerate Spot but don't require it (no nodeSelector or required node affinity pinning them to the Spot pool) will be rescheduled onto the regular pool after eviction. Pods pinned to Spot via nodeSelector cannot reschedule and will remain pending until a Spot node is available again.

Workload Toleration (add to Deployment YAML)

tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
  operator: "Equal"
  value: "spot"
  effect: "NoSchedule"
nodeSelector:
  kubernetes.azure.com/scalesetpriority: spot

Suitability

Risk: Low for batch/dev. High for production stateful workloads. Spot VMs evict with 30-second notice. Eviction policy Delete is recommended for AKS.

Handling Eviction Gracefully

Configure workloads to handle the 30-second eviction notice:

# Add to Deployment spec — terminationGracePeriodSeconds should be < 30s for Spot
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 25
      containers:
      - name: <CONTAINER_NAME>
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]  # Drain in-flight requests

Set a PodDisruptionBudget to limit simultaneous evictions:

kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: <APP_NAME>-pdb
  namespace: <NAMESPACE>
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: <APP_NAME>
EOF