/gke-compute-classes

ComputeClass: Lifecycle, Drift & Updates

Consolidation (Scale-down)

Controlled via spec.autoscalingPolicy.

• consolidationDelayMinutes: Floor is 1 minute.
• consolidationThreshold: CPU utilization % (0 = always candidate).
• gpuConsolidationThreshold: Accelerator utilization %.
• Note: Maintenance windows do not block consolidation. Use PDBs to suppress disruption.
• Blockers: Local storage, 'safe-to-evict: false', and bare pods block scale-down. DaemonSets do NOT block scale-down by default. If empty nodes aren't scaling down, look for other blocking system pods.
• System Pod Segregation: Since non-DaemonSet system pods (e.g., in kube-system namespace) often have tight PDBs or single replicas that block scale-down, always recommend segregating system pods into a dedicated system ComputeClass/node pool (using system-pool-compute-class.yaml). This isolates the system pods and keeps them from blocking the consolidation of Spot/workload nodes.

ActiveMigration (Drift Reconciliation)

Reconciles pods back to higher-priority rules (e.g. moving from fallback On-Demand back to preferred Spot or C3) when capacity becomes obtainable again.

• optimizeRulePriority: true: Enables the drift controller.
• Drift Reconciliation Loop: Periodically evaluates running pods against the ComputeClass priorities[] ladder. When higher-priority capacity becomes available (e.g. 5-minute stockout cooldown expires or new Spot capacity is energized), the controller provisions preferred nodes and initiates voluntary evictions to pull workloads up the ladder.
• Voluntary Disruption Contract: Active Migration calls the Kubernetes Eviction API and strictly respects PodDisruptionBudgets (PDBs). Without a PDB, evictions occur as fast as new nodes become ready.
• Blockers to Active Migration:
  • Tight PDBs (maxUnavailable: 0 or minAvailable: 100%) block eviction entirely.
  • Non-DaemonSet system pods in kube-system without PDBs (DaemonSets are stripped via podutils.FilterRecreatablePods and do NOT block drain).
  • cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation blocks node removal.
  • cluster-autoscaler.kubernetes.io/safe-to-evict: "on-completion" defers active migration until the pod terminates naturally.

Blue-Green & Canary Rollout Protection Against Active Migration Churn

During phased blue-green or canary deployments (e.g., 10% -> 30% -> 60% -> 100% traffic shifts), activeMigration can cause severe rollout churn by voluntarily evicting newly scheduled Green pods to optimize node placement while the pods are warming up or receiving test traffic.

Why PDBs are Superior to Pod Template Annotations

• Pod Template Annotations (safe-to-evict: false): Adding or modifying cluster-autoscaler.kubernetes.io/safe-to-evict: "false" inside spec.template.metadata.annotations mutates the PodTemplateSpec hash. This triggers an immediate rolling restart of the entire Deployment, causing unwanted downtime.
• Rollout-Scoped PodDisruptionBudgets (maxUnavailable: 0): Managing a standalone PDB scoped specifically to the Green version (matchLabels: version: green) operates completely out-of-band via admission control in memory:
  • Causes zero pod restarts (no modification to Deployment or ReplicaSet spec).
  • Blocks voluntary activeMigration and node consolidation during the release window.
  • Can be dynamically relaxed to standard production budgets (e.g., maxUnavailable: 25%) via kubectl patch once cutover reaches 100%.

3-Stage Rollout Lifecycle Pattern

1. Stage 1: Deploy Green + Guard PDB
   • Apply Green rollout PDB with maxUnavailable: 0 targeting version: green.
   • Deploy my-service-green Deployment.
   • Green pods schedule and initialize; activeMigration is blocked from evicting them.
2. Stage 2: Phased Traffic Shift (10% -> 30% -> 60% -> 100%)
   • Shift traffic incrementally across validation windows.
   • Green pods remain stable on their provisioned nodes without eviction churn.
3. Stage 3: Cutover & Relax PDB
   • 100% traffic routed to Green; Blue Deployment and Blue PDB decommissioned.
   • Patch Green PDB to operational budget (maxUnavailable: 25%) to re-enable background activeMigration node shape optimization.

PDB Safety and Edge Cases

# 1. Rollout Guard PDB (Stage 1)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-service-green-pdb
  namespace: default
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      app: my-service
      version: green
---
# 2. Green Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service-green
  namespace: default
spec:
  replicas: 10
  selector:
    matchLabels:
      app: my-service
      version: green
  template:
    metadata:
      labels:
        app: my-service
        version: green
    spec:
      nodeSelector:
        cloud.google.com/compute-class: "general-workloads"
      containers:
      - name: app
        image: gcr.io/my-project/my-service:v2.0.0
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"

CRITICAL K8S DISTINCTION: PDBs and safe-to-evict: false ONLY protect against voluntary disruptions (ActiveMigration, scale-down, upgrades). They DO NOT prevent involuntary Spot VM preemptions. Spot nodes can be reclaimed at any time, regardless of PDBs.

Updating a ComputeClass

• No Retroactive Change: Updating a ComputeClass does not change existing nodes.
• New Nodes Only: Only nodes created after the update use the new spec.
• Drift Behavior:
  • Without ActiveMigration: Old-spec nodes persist until rescheduled (rollout, drain, preemption).
  • With ActiveMigration: Controller drifts pods toward nodes matching the updated (higher-priority) spec.
• Disruption-Sensitive: For training/stateful roles, schedule updates for maintenance windows or drain manually.