/aks-cluster-architecture

Cluster Foundations Bundle

Load this bundle for AKS Automatic vs Standard, version and OS currency, deprecations, permanent cluster decisions, CNI/IPAM, CIDR planning, API server exposure, DNS, and outbound connectivity. If multiple clusters or fleet operations are in scope, also load fleet-management before finalising network, hub, and upgrade-ring decisions.

Assume new AKS projects. Do not preserve legacy defaults unless the task is explicitly a migration.

• Freshness First
• Retirement and Deprecation Checks
• Default Baseline for New Production Clusters
• Decision Classification
• AKS Automatic vs AKS Standard
• Networking Defaults
• Storage and Stateful Data
• Azure Landing Zone Placement Checks
• Version and OS Currency
• Pricing Tier and SLA
• Hyperscale Control Plane Scaling Profile (Public Preview)
• Stop Conditions
• Validation Checklist
• Compliance Alignment
• Threat Model
• Use With

Freshness First

Before giving final guidance for versions, feature state, retirements, or regional availability, verify current status:

az aks get-versions --location <region> --output table
az feature list --namespace Microsoft.ContainerService --output table
az provider show --namespace Microsoft.ContainerService --query "resourceTypes[?resourceType=='managedClusters'].apiVersions" --output table

Use current Microsoft Learn and AKS release notes for feature GA/preview status. Do not hard-code patch versions into long-lived instructions or IaC.

Retirement and Deprecation Checks

Before recommending a networking, OS, ingress, or policy baseline for new clusters, explicitly check for retirements and support deadlines.

Required checks for new designs:

• Do not choose Azure Linux 2.0 for new AKS node pools. Prefer Azure Linux 3 or Ubuntu 24.04 where supported, and verify current node-image support.
• Do not choose kubenet for new production architecture. Prefer Azure CNI Overlay Powered by Cilium unless a documented exception requires another supported model.
• Do not choose Azure Network Policy Manager (NPM) for new Linux production clusters. Azure NPM is being deprecated. Microsoft has announced intent to deprecate NPM and recommends all customers transition to Cilium Network Policy. For existing clusters using Azure NPM, plan migration to Azure CNI Powered by Cilium. Prefer Cilium Network Policy for all new Linux workloads.
• Do not choose an ingress or Gateway pattern without checking support window, GA/preview status, TLS/DNS ownership, controller lifecycle, and migration path.

Active retirement deadlines (verified July 2026):

Build'26 GA announcements (June 2026):

• Kubernetes 1.36 — now GA and LTS
• Managed system node pools — GA for AKS Automatic (preconfigured by default)
• Azure Linux 3.0 — GA default node OS for --os-sku AzureLinux (K8s 1.32–1.36). AzureLinuxOSGuard (hardened/immutable variant) is also an osSku [VERIFY] region/version support.
• Fleet Manager for Arc-enabled clusters — GA (single fleet now supports up to 1,000 member clusters, up from 200)
• Gateway API application routing — GA (replaces NGINX)
• Windows Server 2025 — GA (v1.32+, CLI 2.87.0+)
• Confidential VMs with Azure Linux — GA
• FIPS on Ubuntu 22.04 — GA (FIPS 140-3)

Build'26 Preview announcements:

• AKS on bare metal — direct hardware access (NVLink, RDMA)
• Anyscale on Azure — managed Ray on AKS
• In-place node pool resize — resize VM size without migration
• Automatic PDB management — auto-creates PDBs, scales up to unblock drain
• LocalDNS auto-enabled on k8s 1.36+ node pools
• Monitor the AKS Security Bulletins feed for active node-image advisories. The current worked example is AKS-2026-0003 (illustrative - verify the actual bulletin, CVE IDs, and patched VHD build against the current AKS Security Bulletins feed) (Copy Fail - algif_aead kernel LPE, CVE-2026-31431); bulletins identify the patched VHD build and any interim host mitigation. Designs must include a CVE-response runbook tied to this feed (see operations-resilience and fleet-management).
• If a customer is migrating from a deprecated component, separate the target-state architecture from the migration runbook and call out staged validation.

Post-Build'26 additions (verified 2026-08-11 against AKS release 2026-07-17 and Azure Updates MCP):

Managed system node pools (AKS Automatic): security restrictions to design around (AKS 2026-05-29/06-19): new Automatic clusters preconfigure a managed system node pool by default, which blocks customer-supplied SSH keys, denies kubectl port-forward onto the managed pool, blocks kube-system secret reads (except trusted identities), enforces a ValidatingAdmissionPolicy against Service spec.externalIPs (aligned with the upstream K8s 1.36 deprecation), and restricts mutating admission bindings (all mutating admission resources blocked pre-1.36; controlled subset on 1.36+). LocalDNS mode defaults to Required. Plan workloads against these constraints rather than trying to work around them.

Node OS currency (verified 2026-07-21):

• Azure Linux 3.0 is the GA default for --os-sku AzureLinux (K8s 1.32–1.36). The dated (mariner) label reflects lineage only: Azure Linux 3.0 is CBL-Mariner-derived, but Azure Linux 4.0 (Public Preview, Fedora-derived, dnf5 replaces tdnf) is not yet an AKS node osSku (VM/VMSS/container images only; AKS support "coming soon"), do not select it for node pools yet. [VERIFY] AKS AzureLinux4 osSku availability. The versioned AzureLinux3 SKU blocks upgrades past 1.36, pin --os-sku AzureLinux (not AzureLinux3) so the cluster can advance to 1.37+.
• Ubuntu 24.04 is GA and first-class, not legacy. It is the default for --os-sku Ubuntu on K8s 1.35+; Ubuntu node pools auto-migrate 22.04 → 24.04 on the 1.35 upgrade. The Ubuntu2404 versioned SKU is supported K8s 1.32–1.38. FIPS 140-3 is not supported on Ubuntu 24.04, use Azure Linux or Ubuntu 22.04 FIPS if a FIPS baseline is required. [VERIFY] current SKU/version gates.
• Azure Container Linux (ACL). GA (K8s 1.34+). A lightweight, container-optimised OS maintained by Microsoft, separate from Azure Linux. Key properties: reduced configuration drift, simplified fleet maintenance, and in-place OS SKU migration from Azure Linux or Ubuntu. Use --os-sku AzureContainerLinux (osSku value AzureContainerLinux). Designed for teams that want a minimal surface; does not carry all Azure Linux / Ubuntu package sets. Entra ID SSH is rejected on ACL pools (the extension is incompatible with immutable OS nodes and can make nodes unreachable) - use another access path on ACL. [VERIFY] current supported K8s range, region availability, and feature-parity gaps (GPU, FIPS, Confidential VMs) against Azure Container Linux overview before selecting for production.
• Azure Files NFS Encryption-in-Transit × Azure Linux interaction. Some Azure Linux 3.0 node VHDs shipped without the AZNFS mount helper (/sbin/mount.aznfs), which breaks encryptInTransit NFS mounts (Azure/AKS#5772). If mandating Azure Linux together with the July-2026 Azure Files NFS EiT GA baseline, confirm the node image includes the AZNFS helper before rollout.

containerd 2.x is the default runtime, a major bump that arrives with the OS/K8s upgrade

containerd 2.x is now the default runtime on both default node-OS SKUs: Azure Linux 3.0 since K8s 1.33 and Ubuntu 24.04 (K8s 1.35+). Because it ships inside the node image, a Kubernetes minor upgrade silently crosses a container-runtime major version, treat the 1.33 (Azure Linux) or 1.35 (Ubuntu) hop as a containerd 2.0 migration, not a routine upgrade. This is easy to miss precisely because nobody opted into a runtime change. [VERIFY] the exact containerd build in the current AKS node image against the release-notes changelog.

What breaks, kubelet itself is fine (it uses CRI v1); the breakage is in tooling, so it fails quietly:

• CRI v1alpha2 API removed. Third-party monitoring/security agents and DaemonSets that open the containerd socket with the v1alpha2 client (older node_exporter / node-problem-detector builds, vendored crictl, custom eBPF or CRI DaemonSets) silently lose data or CrashLoop. Earliest signal: CRI-derived node metrics go flat, or the agent pod errors, only after a node moves to containerd 2.x. Fix: upgrade the tooling to a build using CRI v1 (baseline since K8s 1.23). Source: containerd 2.0 migration guide, tracked for AKS in Azure/AKS#4700.
• Registry-mirror config trap (silent bypass or hard fail). The CRI plugin in containerd 2.x does not honor a colon-joined config_path (e.g. '/etc/containerd/certs.d:/etc/docker/certs.d'), it silently ignores the mirror and pulls straight from the public/ACR endpoint with no log or error, which then trips the ACR image-pull throttling storm (one silent config typo becomes a cluster-wide pull storm). Separately, a legacy [plugins."...".registry.mirrors] block makes the CRI plugin fail to load and the node never reaches Ready. Fix: use per-registry hosts.toml under a single existing certs.d tree; remove all legacy [registry.mirrors] / registry.configs stanzas. (containerd issues #12808, #12612, #12636.)
• Docker Schema 1 image support dropped. Pods referencing legacy Schema 1 images fail to pull after the hop (rare in 2026, re-push affected images as OCI/Schema 2).
• Ubuntu 24.04 also tightens unprivileged user namespaces. The 22.04→24.04 migration enables AppArmor's kernel.apparmor_restrict_unprivileged_userns restriction, which blocks unprivileged unshare(CLONE_NEWUSER), rootless build tooling and some sandboxed runtimes fail with EPERM/AppArmor userns_create denials only on 24.04 nodes. Mitigation: ship a per-workload AppArmor profile that allows userns, or set the sysctl via node customization (do not blanket-disable cluster-wide without security review). [VERIFY] whether the AKS 24.04 node image keeps this restriction enabled by default on a live node before relying on either behavior.

Pre-upgrade validation: on a canary node pool pinned to the target OS at the current K8s version (AKS rolls node-OS version independently of K8s), run crictl version to confirm containerd 2.x, enumerate any CRI-v1alpha2 clients, and verify registry-mirror config resolves via hosts.toml. See operations-resilience, pre-upgrade runbook for the full sequence.

Default Baseline for New Production Clusters

IaC surface anchors. OIDC issuer enablement is --enable-oidc-issuer (CLI), properties.oidcIssuerProfile.enabled (Bicep/ARM), oidc_issuer_enabled (Terraform azurerm). Workload Identity enablement is --enable-workload-identity (CLI), properties.securityProfile.workloadIdentity.enabled (Bicep/ARM), workload_identity_enabled (Terraform azurerm). Both must be enabled at cluster creation or via update - confirm regional and version support.

Decision Classification

AKS Automatic vs AKS Standard

Decision rule: choose AKS Automatic when Microsoft-managed defaults are acceptable and reduce operational burden. Choose AKS Standard when network topology, node pool control, GPU/AI, Windows, compliance exceptions, or migration constraints require deeper control.

Structural limits to confirm before choosing AKS Automatic (verified 2026-05-14 against Microsoft Learn - recheck before commitment):

• Availability zones in West US. AKS Automatic clusters in the West US region cannot use availability zones - a regional limitation. Other zonal regions are unaffected. If multi-zone is mandatory and the target region is West US, choose AKS Standard or relocate to a zonal region.
• Deployment Safeguards level is fixed. AKS Automatic enforces Deployment Safeguards at a level that cannot be relaxed via Azure Policy exemption. Workloads that violate the bundled policies must be remediated, not exempted - design for the policy set, or choose AKS Standard.

Migration readiness assessment. Before deciding to move an existing AKS Standard workload to AKS Automatic, run an automated readiness check against the current cluster. KubeBuddy (see SKILL.md - External validation tools) generates a dedicated *-aks-automatic-action-plan.html report when run with the --aks flag, showing migration blockers, warning-driven actions, and a suggested migration sequence with Microsoft Learn links. Use this as a pre-migration gate: resolve all blockers before committing to the migration path. [VERIFY] the current AKS Automatic migration guidance and tooling support against Microsoft Learn, as the migration tooling surface evolves.

Networking Defaults

Azure CNI Overlay Powered by Cilium

Default to Azure CNI Overlay Powered by Cilium for new general-purpose clusters when supported. For production NetworkPolicy and namespace examples, load ../production-workload-controls/guide.md.

Use this wording deliberately:

• Azure CNI Overlay is the IPAM/routing model.
• Powered by Cilium is the eBPF data plane and policy implementation.
• Do not describe Cilium as a standalone CNI choice unless specifically discussing BYO CNI or upstream Cilium outside AKS-managed Azure CNI.
• For production workload namespaces, pair Cilium with default-deny NetworkPolicy and explicit DNS/ingress/east-west/egress allow rules.

Choose alternatives only when justified:

eBPF kube-proxy Replacement (Cilium)

When Azure CNI Overlay Powered by Cilium is the chosen CNI, evaluate enabling Cilium's kube-proxy-free mode, which replaces kube-proxy with a native eBPF data plane for Service load balancing and NodePort handling:

• Performance benefit. Removes the iptables/ipvs layer entirely. Service lookups and DNAT are handled in the kernel eBPF hook, reducing per-packet overhead and improving connection setup latency, particularly significant at high connection rates or many Services.
• AKS support. Enable via --kube-proxy-config when creating or updating a cluster with the Cilium data plane. [VERIFY] the current AKS surface (networkProfile.kubeProxyConfig.mode: IPVS vs the Cilium-native path) and whether kube-proxy can be disabled via the AKS-managed Cilium configuration before recommending in IaC.
• When to choose. New clusters targeting high-connection-rate workloads (APIs, proxies, gateways) where iptables rule-chain scaling is a concern, or clusters adopting Cilium L7 policy (ACNS) where the eBPF path is the preferred integration point.
• When NOT to choose. Clusters with a confirmed requirement for kube-proxy-specific behaviour, or where the kube-proxy-replacement feature is not yet GA for the target AKS minor. Mark as Difficult: removing kube-proxy after workloads are running requires a node-pool replacement path.

eBPF Host Routing (ACNS Container Network Performance, GA)

eBPF Host Routing is the ACNS Container Network Performance feature (GA June 2026): Cilium eBPF programs take over host-side routing and SNAT, bypassing iptables/netfilter processing in the host network namespace for lower pod-to-pod latency, higher throughput, and modest CPU savings. Use for performance-critical high-throughput or latency-sensitive workloads on Azure CNI Powered by Cilium clusters.

• Enable: az aks {create,update} --enable-acns --acns-datapath-acceleration-mode BpfVeth. Disable independently with --acns-datapath-acceleration-mode None. Requires Azure CLI ≥ 2.71.0 and Kubernetes ≥ 1.33.
• Cluster-wide only. Applies to every node; hybrid node scenarios are not supported. Nodes are labeled kubernetes.azure.com/ebpf-host-routing=true, and enabling triggers rolling node-pool upgrades (drain-aware) - treat enablement as a Difficult decision with a maintenance window.
• iptables semantics change. Existing host-network-namespace iptables rules are bypassed, AKS blocks clusters that rely on such rules from enabling the feature (detection at enable time), and once enabled new iptables rule installation in the host namespace is blocked by an init container. Host-firewall/NVA agents that inject host iptables rules are incompatible with this mode. The ip-masq-agent keeps running but its rules are ignored while eBPF routing is active (Cilium performs BPF masquerade).
• Incompatible surfaces (verified against Learn limitations): Static Egress Gateway, Confidential VM node pools, Pod Sandboxing (Kata), Windows nodes, and Istio Ambient via self-managed install combined with ACNS performance mode. Dual-stack connectivity IS supported.
• Node OS gate: Azure Linux 3.0 or Ubuntu 24.04 only.

LocalDNS (DNS Performance)

Enable LocalDNS on AKS node pools to improve DNS performance and reduce load on CoreDNS pods. LocalDNS deploys a DNS proxy as a systemd service on each node that handles DNS queries locally.

Benefits:

• Reduces network hops and resolution latency for service-to-service communication
• Supports serving stale cached responses when upstream DNS is unavailable (configurable duration)
• Improves reliability during transient CoreDNS outages

When to enable:

• Production clusters with >50 pods or >20 services
• Latency-sensitive workloads where DNS resolution time matters
• Clusters where CoreDNS has been observed as a bottleneck

# Enable LocalDNS on node pool
az aks nodepool update --resource-group <rg> --cluster-name <name> \
  --name <pool-name> --dns-scope <dns-scope> --nodepool-mode System

Uptime SLA and Pricing Tier

[VERIFY] exact SLA percentages and per-cluster pricing against current Azure pricing, the numbers change and the detailed figures live in Pricing Tier and SLA below. Note that choosing Long-Term Support routes the cluster to the Premium tier once its minor version exits community support, a version you keep for stability quietly becomes a billing event.

Production rule: Always use at least Standard tier. The uptime SLA guarantees API server availability separate from the node pool SLA. Without it, a control plane outage is your operational responsibility.

The SLA covers the Kubernetes API server endpoint, it does NOT cover your workloads. Workload availability depends on your node pool configuration (availability zones, replicas, PDBs).

Defender for Containers

Enable Microsoft Defender for Containers on production AKS clusters for:

• Threat detection for cluster, container, and workload activity
• Vulnerability assessment for container images in ACR and running in clusters
• Runtime protection for suspicious process execution in containers
• Integration with Microsoft Sentinel for centralized alerting

# Enable Defender for Containers
az security pricing create -n Containers --tier Standard
az aks update --resource-group <rg> --name <name> --defender true

Cost consideration: Defender for Containers charges per node per month. Evaluate against your compliance requirements, mandatory for regulated workloads, recommended for all production.

Networking Decision Framework

Before choosing networking options, classify the decision:

Default recommendations for new production clusters:

1. CNI: Azure CNI Overlay Powered by Cilium
2. API server: Private cluster with VNet integration
3. Network policy: Cilium Network Policy (the eBPF data plane of Azure CNI Powered by Cilium), not Azure NPM, which is deprecated for new Linux clusters
4. Egress: Azure Firewall via UDR (for inspection) or NAT Gateway (for simplicity)
5. DNS: LocalDNS enabled on system node pool
6. Ingress: Application Gateway for Containers (AGC) or App Routing with Gateway API, not upstream/self-managed ingress-nginx as a new baseline (two-stage EOL cliff: OSS maintenance ended March 2026, AKS managed-NGINX security-patch-only through Nov 2026). See operations-resilience.

For NAP/Karpenter designs, verify networking support. Current Microsoft guidance recommends Azure CNI with Cilium and does not support Calico network policy or Dynamic IP Allocation for NAP.

CIDR Planning

Never allow overlap between:

• cluster VNet CIDRs
• pod CIDR
• service CIDR
• DNS service IP
• peered VNet ranges
• on-premises ranges over VPN/ExpressRoute
• other AKS clusters expected to communicate or join a fleet

Document CIDRs in an ADR and an environment registry before provisioning.

Service CIDR sizing rule. The --service-cidr is fixed at cluster creation and cannot be changed without a cluster rebuild. Under-sizing is one of the most common causes of Service resource creation failures on established clusters. Use a minimum of /16 (65 534 service IPs) for production clusters, and /12 for large-scale or multi-namespace platforms. The DNS service IP must fall within this range (default 10.0.0.10 from a 10.0.0.0/16 service CIDR). Document the chosen range in the ADR alongside pod and node CIDRs.

Dual-Stack (IPv4 + IPv6)

Dual-stack AKS is a Permanent decision - the IP family (--ip-families) is fixed at cluster creation and cannot be changed afterwards. Treat it as an architecture-level choice, not a configuration toggle.

Use this wording deliberately:

• Azure CNI Overlay is the required IPAM model for dual-stack on AKS (both Azure CNI Overlay and Azure CNI Overlay Powered by Cilium support it).
• --ip-families ipv4,ipv6 is the enablement flag; single-stack IPv6-only is not supported for nodes or pods (services can be single-stack on either family).
• Cilium is the recommended data plane for dual-stack because its NetworkPolicy engine can control IPv6 traffic; Azure NPM and Calico network policies are not supported in dual-stack.
• Kubernetes 1.29+ is required for dual-stack with Cilium; 1.26.3+ for Azure CNI Overlay dual-stack.

Choose dual-stack only when there is a concrete driver:

Dual-stack constraints to document in the ADR:

• NAT Gateway is not supported in dual-stack (Azure CNI Overlay dual-stack limitation). Outbound for IPv6 uses the standard LoadBalancer outbound path; use outboundType: loadBalancer or UDR with an IPv6 default route. [VERIFY] managedNATGatewayV2 IPv6 egress support if the cluster requires a single static IPv6 egress IP, as NAT GW V2 IPv6 support is evolving.
• Azure Linux node pools in dual-stack only support services with externalTrafficPolicy: Local.
• Virtual nodes add-on is not supported in dual-stack.
• The IPv6 service CIDR is capped at a /108 (significantly smaller than the IPv4 /16 default).
• --pod-cidrs and --service-cidrs must each list an IPv4 and an IPv6 range, in matching order to --ip-families.

For most new production clusters, single-stack IPv4 remains the simpler default; choose dual-stack deliberately when a driver above applies.

API Server Exposure

Do not recommend --enable-private-cluster=false with --api-server-authorized-ip-ranges unset for new production clusters - that combination exposes the API server to the public internet without IP allowlisting.

API Server Hardening

IP allowlisting and the private/public exposure choice are necessary but not sufficient. Production clusters must also pin down authentication, admin elevation, and audit:

• Anonymous auth disabled. Verify the cluster's API server does not accept anonymous requests. Recent AKS releases default --anonymous-auth=false; verify per release and document the value explicitly so the choice does not silently regress on upgrade.
• No standing cluster-admin. Grant cluster-admin and high-impact roles only through Microsoft Entra Privileged Identity Management (PIM) with JIT activation, an approver, and time-bound assignments. Day-to-day operators run with bounded roles; emergency elevation goes through PIM with audit.
• Conditional Access on the sign-in path. az aks get-credentials and kubectl use Entra interactive sign-in. Require device compliance, MFA, and named-location enforcement via Conditional Access policies on the AKS Entra app and the operator group. Block legacy authentication.
• Break-glass account. Maintain a single, fully documented break-glass identity with an offline credential, exclusion from Conditional Access blocks, separate audit alerting on every sign-in, and time-bound activation procedure. Test the activation path on a defined cadence so it works when needed.
• Audit policy. Configure the AKS audit policy so secrets, exec, attach, portforward, and rbac.authorization.k8s.io/* verbs emit at RequestResponse level. Forward audit logs to Microsoft Sentinel (or equivalent SIEM) on an immutable retention tier. See operations-resilience - Audit policy and SIEM forwarding for audit policy levels, the forwarding path, and starter Sentinel detection rules.
• CI runner reachability - Difficult decision. A private API server requires an explicit network path for CI: self-hosted runners in a peered spoke, JIT bastion / Cloud Shell, or ephemeral allowlist updates (--api-server-authorized-ip-ranges). Each option has different blast radius, cost, and audit posture; pick one explicitly and document it as an ADR.
• Admission webhook failurePolicy and timeout hardening. Admission webhooks with failurePolicy: Fail and a short (or misconfigured) timeoutSeconds are among the most dangerous cluster-wide failure modes. A webhook server that becomes unavailable will block all pod creation (and other mutations/validations) cluster-wide until the timeoutSeconds expires per request. Production hardening rules:
  • Set timeoutSeconds ≥ 10 for failurePolicy: Fail webhooks (the API server default is 10 s; many operators default to 5 s which is too tight under load).
  • Prefer failurePolicy: Ignore for any webhook whose failure should not block the admission path (e.g., non-critical telemetry enrichers, advisory labellers). Reserve Fail for security-gating webhooks (image verification, PSA policy enforcement).
  • Set namespaceSelector to exclude kube-system and operator namespaces from non-system webhooks to avoid circular failures during node bootstrapping.
  • Audit all installed ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources during pre-upgrade checks (see operations-resilience - Admission and Conversion Webhook Pre-Upgrade Checklist).
  • kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml | grep -A2 failurePolicy is the fast audit command.

Secrets, etcd Encryption, and Data-at-Rest

Kubernetes Secrets are base64-encoded only - they are not encrypted at rest in etcd unless additional controls are configured. Anyone with get secrets permission can read them in plain text via kubectl get secret -o yaml. For regulated workloads (financial services, healthcare, government, PCI-DSS, HIPAA/HITRUST), this is not acceptable on its own.

Production defaults for new AKS clusters:

• Enable customer-managed key (CMK) encryption for etcd at rest via Azure Key Vault when the workload handles regulated or confidential data. Verify current support status, region, HSM-backed vs software keys, and whether the cluster must be created with the flag enabled (KMS configuration on AKS is normally a Permanent decision because rekeying touches every encrypted resource).
• Prefer Azure Workload Identity + direct Key Vault retrieval over storing application secrets as Kubernetes Secrets. Use Key Vault CSI Driver only when sidecar/init-time retrieval or runtime rotation into the pod filesystem is required; document its sync interval and rotation behaviour.
• Restrict get/list on Secrets to a small set of operator and workload identities. Audit Secret reads via Kubernetes audit logs forwarded to ContainerLogV2 or your SIEM.
• Track cluster encryption status as an ADR-worthy decision; record key vault, key URI, and rotation cadence.

Secrets lifecycle and Workload Identity hygiene:

• One managed identity per workload (or per bounded component). Do not back multiple unrelated workloads with a single shared "platform" MI - it collapses blast radius and prevents per-workload revocation.
• Rotation cadence. Azure-issued tokens for Workload Identity are auto-rotated by the platform; you do not rotate the federated credential trust. For Key Vault references retrieved into pods, define a key-rotation cadence (e.g., 90 days for high-sensitivity keys, 180 days otherwise) and a pod-restart strategy that picks up new key versions - Reloader, controller restart on key-version change, or an explicit rollout when the CSI driver syncs.
• Federated-credential audit. Review federatedIdentityCredentials on every workload managed identity at least quarterly. Confirm subject, issuer, audience, and namespace/service-account still match the live workload; remove orphaned credentials immediately.
• Revocation / break-glass. To revoke a workload identity quickly, remove the federated credential first (cuts new token issuance), then remove the role assignments (cuts data-plane access), and disable the managed identity if needed. Document blast radius: any pod still holding a non-expired token retains access until expiry, so couple revocation with key/secret rotation on the protected resource where the exposure window matters.

IaC surface anchors (verify against current provider docs):

• Azure CLI: az aks create ... --enable-azure-keyvault-kms --azure-keyvault-kms-key-id <kv-key-uri> --azure-keyvault-kms-key-vault-network-access Private|Public
• Bicep / ARM: properties.securityProfile.azureKeyVaultKms.{enabled,keyId,keyVaultNetworkAccess,keyVaultResourceId}
• Terraform (azurerm): key_management_service { key_vault_key_id, key_vault_network_access } on azurerm_kubernetes_cluster

Validation:

az aks show --resource-group <rg> --name <cluster> \
  --query "securityProfile.azureKeyVaultKms" --output yaml

Treat enabling/disabling KMS as Permanent - plan it before provisioning, not as a retrofit.

Confidential Compute

⚠ Retirement notice (verified 2026-07-11; retirement effective March 2026). Microsoft retired AKS Confidential Containers (Kata + AMD SEV-SNP) - the per-pod TEE pattern below - in March 2026 (node images removed 2026-03-31; see Azure/AKS#3781). Do not adopt confidential containers for new AKS projects. For "data in use" requirements on AKS, use Confidential VM node pools (AMD SEV-SNP) - the whole-node pattern - which remains supported; verify current support status against Microsoft Learn before committing. Any remaining confidential-container workload needs a migration plan to confidential VM node pools or off-AKS confidential compute.

Confidential compute on AKS protects data in use by running workloads inside hardware-backed TEEs. Two patterns matter for new clusters:

• Confidential VM node pools (AMD SEV-SNP) - the entire node runs as a confidential VM. Lift-and-shift compatible with most pod workloads.
• Confidential containers (Kata + AMD SEV-SNP) (retired March 2026, see banner above; do not adopt) - was a per-pod TEE isolation pattern; finer-grained but more operational complexity, and no longer available on AKS.

When to consider it:

• Regulated workloads handling restricted data classes (financial transaction processing, healthcare PHI under strict residency, sovereign/government workloads).
• Customer-managed attestation evidence is required by the regulator, ISV contract, or zero-trust controls - the workload owner must be able to prove the runtime state to a third party.
• Explicit "data in use" protection mandates (e.g., contractual or framework-driven requirements that go beyond at-rest and in-transit encryption).

When NOT to use it:

• Most internal applications, dev/test, and workloads with no third-party-key or attestation requirement. The pricing premium, VM family constraints, and operational overhead (attestation pipeline, image preparation for confidential containers) are real.
• Workloads whose threat model is already satisfied by managed identity, network isolation, and etcd KMS.

Validation:

• Attestation flow goes through Microsoft Azure Attestation (MAA); design the policy, signer, and consumer of the attestation token before the cluster is built.
• Confirm region availability of confidential VM SKUs (DCasv5/DCadsv5, ECasv5/ECadsv5, or current equivalents) and AKS support per region; gaps are common.
• Verify VM family constraints (memory, GPU, accelerated networking) match the workload - confidential SKUs do not yet cover every family.

Mark confidential compute as a Difficult-to-reverse decision: switching a node pool in or out of confidential mode requires node-pool replacement and workload migration, not an in-place toggle.

Outbound Connectivity

Default outbound access retired (effective March 31, 2026). Azure no longer provides default outbound access for VMs. New AKS clusters using an AKS-managed VNet deploy their subnets as private subnets (defaultOutboundAccess = false); nodes have no internet egress unless you configure one of the outbound types below. BYO-VNet clusters remain responsible for their own egress. Decide the outbound type at cluster creation: a cluster with no explicit egress will fail node bootstrapping (image pulls, component downloads) unless it is a network-isolated cluster using none/block with a private ACR artifact source. Source: Azure update and AKS egress outbound types.

IaC surface anchors. --outbound-type (CLI), properties.networkProfile.outboundType (Bicep/ARM), network_profile.outbound_type (Terraform azurerm). Valid values: loadBalancer, managedNATGateway, managedNATGatewayV2, userAssignedNATGateway, userDefinedRouting, none, block. The managedNATGatewayV2 outbound type requires the ManagedNATGatewayV2Preview feature flag registered on Microsoft.ContainerService and the aks-preview CLI extension. StandardV2 public IPs are required for customer-defined IPs; existing Standard SKU IPs will not work.

[!IMPORTANT] The managedNATGatewayV2 outbound IP model (Azure-managed vs customer-defined) is a permanent decision set at cluster creation. You cannot switch between the two after creation. Before provisioning a cluster with managedNATGatewayV2, determine whether external systems need to allowlist your egress addresses. If yes, bring your own StandardV2 public IPs at creation time. If no, let Azure manage them.

outboundType can now be updated after cluster creation (verified 2026-07-11). AKS supports defined migration paths between outbound types on both managed and BYO VNets — this was previously a create-only decision. Supported managed-VNet migrations include loadBalancer → any, managedNATGateway → managedNATGatewayV2/none/block, and none/block → any. Migrating away from managedNATGatewayV2 to any other outbound type is "Not Supported" on a managed VNet. managedNATGatewayV2 is effectively a one-way door: you can migrate into it from loadBalancer, managedNATGateway, none, or block, but once on it you cannot migrate off it in place. Migrating to userAssignedNATGateway, userDefinedRouting, or managedNATGatewayV2 changes the cluster's outbound public IP addresses; if API server authorized IP ranges are enabled, append the new outbound range first. Changing outbound type is disruptive (node egress re-plumbing); schedule a maintenance window. Source: AKS egress outbound types. [VERIFY] current supported paths before planning a migration.

Static Egress Gateway (per-namespace egress IP, GA)

When a subset of workloads needs a fixed, predictable source IP for outbound traffic, per-tenant firewall allowlisting, partner integrations, or EgressIP-style parity when migrating from OpenShift, Static Egress Gateway assigns a stable egress IP per namespace without standing up a separate node pool + subnet + NAT Gateway for each one (the pre-feature pattern, which does not scale past a handful of namespaces).

• Enable on the cluster (--enable-static-egress-gateway), add a dedicated gateway-mode node pool (--mode gateway --gateway-prefix-size <28-31>), then bind namespaces with a StaticGatewayConfiguration CRD (egressgateway.kubernetes.azure.com/v1alpha1); pods opt in via annotation. Public-IP and private-IP modes are both GA (private-IP mode requires Kubernetes 1.34+ and --vm-set-type VirtualMachines on the gateway node pool).
• Constraints: not supported with Azure CNI Pod Subnet; the gateway node pool is egress-only (no general workloads) and does not autoscale (size it to the IP prefix); Windows and hostNetwork pods are excluded; pods must be in the same namespace as the StaticGatewayConfiguration. Kubernetes NetworkPolicy does not apply to traffic leaving via the gateway, keep egress controls on the annotated pods in mind. Mutually exclusive with ACNS eBPF Host Routing - do not combine a Static Egress Gateway design with --acns-datapath-acceleration-mode BpfVeth (see eBPF Host Routing). Source: Microsoft Learn. Configure Static Egress Gateway.
• This is a workload-egress overlay on top of the cluster's outbound type (NAT Gateway / UDR), not a replacement for it. Decide it alongside the egress design when per-namespace source-IP identity is a real requirement.

DNS

• Use Azure Private DNS Zones for Private Link and private cluster designs.
• Decide private DNS mode up front for enterprise landing zones.
• Keep CoreDNS customisation minimal and version-controlled.
• Use local DNS caching features only after confirming current AKS support and cluster compatibility.

CoreDNS at scale

Default CoreDNS sizing is fine for small clusters but is a common source of production incidents at scale. Design for DNS load before it bites:

• ndots:5 amplification. The default pod dnsConfig appends search domains, so every external name (api.example.com) is first tried as api.example.com.<namespace>.svc.cluster.local, then several more suffixes, before the real lookup, up to 5+ failed queries per external call. For DNS-heavy workloads calling external endpoints, set dnsConfig.options ndots: 2 (or use FQDNs with a trailing dot) to cut query volume dramatically.
• CoreDNS overload / throttling. Under high QPS, CoreDNS pods saturate and lookups start timing out cluster-wide, frequently misdiagnosed as application latency. Scale CoreDNS (it runs with an autoscaler on AKS; verify replica count vs node/pod count), watch the coredns_dns_request_duration_seconds and SERVFAIL/throttle metrics, and avoid heavy per-request DNS in hot paths.
• NodeLocal DNSCache / LocalDNS. A per-node DNS cache absorbs the bulk of repeat lookups, removes a per-query hop, and shields CoreDNS from spikes. On AKS this surfaces as LocalDNS (on by default for AKS Automatic, see SKILL.md). For Standard clusters, evaluate enabling node-local caching once support and cluster compatibility are confirmed.
• Custom Corefile = a Reversible-but-risky edit. Custom forwarders, rewrites, or stub zones (including the CoreDNS split-DNS trap for NAT Gateway + Private Endpoint designs) live in a ConfigMap; keep them version-controlled and test in non-prod, a bad Corefile takes out cluster-wide name resolution.

Identity and Access

Spot Node Pools for Cost Optimization

Spot node pools use Azure's surplus capacity at 60-90% discount. Essential for cost-optimized batch, CI/CD, and dev/test workloads.

az aks nodepool add \
  --name spot-workers \
  --cluster-name my-cluster \
  --resource-group my-rg \
  --mode User \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --node-taints kubernetes.azure.com/scalesetpriority=spot:NoSchedule \
  --node-count 1 --min-count 0 --max-count 10 \
  --enable-cluster-autoscaler

Key patterns:

• Taint + toleration: Only schedule interruptible workloads on spot nodes
• Eviction policy: Delete (remove node) or Deallocate (stop but keep node)
• Max price: -1 means pay up to on-demand price; set a cap for budget control
• Cluster autoscaler: Required for spot pools, scales from 0 to max based on demand

When to use spot: Batch processing, CI/CD runners, data engineering jobs, dev/test environments, fault-tolerant microservices with replica redundancy.

When NOT to use spot: Databases, stateful services, single-replica workloads, latency-sensitive APIs without replica redundancy.

Fallback scheduling rule: Prefer Spot, do not hard-require Spot, unless pending-on-eviction is explicitly accepted. Use preferredDuringSchedulingIgnoredDuringExecution + Spot toleration so workloads can fall back to regular nodes when Spot capacity is unavailable. Avoid strict Spot-only selectors for critical execution paths.

Pod Security Admission (PSA) Migration

Kubernetes 1.25+ removed PodSecurityPolicy (PSP). AKS uses Pod Security Admission (PSA) with built-in pod security standards.

# Namespace with restricted PSA
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Migration checklist from PSP:

1. Identify all PSP policies in use
2. Map PSP restrictions to PSA standards (baseline vs restricted)
3. Test workloads against new PSA labels in non-production
4. Add exemptions for system workloads (monitoring, logging, CNI)
5. Remove PSP resources after PSA is verified

Node Image Auto-Upgrade Channels

az aks nodepool update \
  --name mypool --cluster-name mycluster --resource-group myrg \
  --node-os-upgrade-channel SecurityPatch

Tip: Combine NodeImage channel with maintenance windows for predictable patching. Don't use Unmanaged unless you have a custom image pipeline.

AKS Cost Estimation Patterns

AKS has multiple identity surfaces. Getting them wrong at creation time is expensive to retrofit.

Kubelet identity vs control-plane identity

AKS creates two managed identities (or uses one you specify), do not confuse them in IaC:

Pitfall (production-verified 2026-06-23). When assigning AcrPull to the kubelet identity in Bicep, use aksCluster.properties.identityProfile.kubeletIdentity.objectId as the role assignment principalId , NOT aksCluster.identity.principalId. The control-plane identity cannot pull images from ACR.

Pitfall (Bicep BCP120). The kubelet objectId is a runtime property: it only exists after the managedClusters resource is provisioned. Derive the role assignment's name from something static such as cluster.id, never from the kubelet objectId itself; using the runtime property as the name anchor fails Bicep validation (BCP120).

Workload Identity (pods → Azure resources)

Enable OIDC issuer and Workload Identity at cluster creation. See identity-managed-identity: AKS Workload Identity for the full federated credential + service account pattern.

Ordering pitfall. The federated credential subject must match the exact service account the pod uses. If a third-party operator (e.g. Drasi) creates per-resource service accounts, you must annotate the correct SA after it is created but before (or with a restart after) the pod needs the token. A pod started before its SA is annotated will not get the injected AZURE_CLIENT_ID / AZURE_FEDERATED_TOKEN_FILE env vars and will fall back to a credential chain that fails.

ACR integration

Prefer managed identity for ACR pull, not admin credentials or Kubernetes image pull secrets:

1. Create or reference a user-assigned identity with AcrPull on the ACR.
2. Assign it to the kubelet identity (above) or to a dedicated workload SA with Workload Identity.
3. For zone-redundant registries serving multi-region clusters, use ACR geo-replication.

Storage and Stateful Data

Stateful workloads on AKS fail in ways stateless ones do not, and the failure modes are usually decided at provisioning time. Pick the storage class and topology deliberately, several of these choices are hard to reverse once data exists.

StorageClass selection

• Default to CSI drivers (disk.csi.azure.com, file.csi.azure.com, blob.csi.azure.com); the in-tree drivers are retired. Set allowVolumeExpansion: true on production StorageClasses so volumes can grow without a migration.
• Pin a reclaim policy intentionally: Retain for anything whose loss is unacceptable, Delete only for ephemeral data.

The zonal-disk trap (Permanent-by-data)

Azure managed disks are zonal. A Disk-backed PersistentVolume is created in one availability zone, and a pod bound to it can only be scheduled onto a node in that same zone. This silently defeats the multi-zone topologySpreadConstraints recommended for stateless workloads: if the zone holding the disk has no capacity (or is in outage), the pod stays Pending.

• Use volumeBindingMode: WaitForFirstConsumer (the default on AKS-managed StorageClasses) so the disk is provisioned in the same zone the scheduler picks for the pod, never Immediate for zonal disks, which can place the disk and the only schedulable node in different zones.
• For zone-resilient stateful services, prefer application-level replication across zones (each replica with its own zonal disk) over trying to move a single disk between zones. Consider ZRS (zone-redundant) disks where the workload and region support them so a single volume survives a zonal outage, [VERIFY] ZRS disk SKU/region availability.
• Azure Files and Blob are not zone-pinned in the same way and are the simpler choice for RWX, cross-zone access.

Stateful workload rules

• Use StatefulSets (stable identity + per-replica PVC via volumeClaimTemplates), not Deployments, for quorum/stateful systems.
• Set fsGroup (and fsGroupChangePolicy: OnRootMismatch for large volumes) so the container can write to the mounted volume under a non-root securityContext.
• Decide data residency and backup per volume: PV snapshots, Azure Backup for AKS, and restore testing belong in the DR plan (operations-resilience - Backup, Restore, and DR). A managed-disk snapshot is zonal too, plan cross-zone/region copy if the DR target is elsewhere.
• For ephemeral scratch that must be writable under readOnlyRootFilesystem: true, use an emptyDir rather than a PVC.

IaC surface anchors. StorageClasses are cluster resources (GitOps-owned), not an az aks flag. CSI drivers are enabled via storageProfile.diskCSIDriver / fileCSIDriver / blobCSIDriver (Bicep/ARM) and the storage_profile block (Terraform azurerm). Verify Azure Container Storage enablement (--enable-azure-container-storage) and supported pool types against current Microsoft Learn.

Azure Landing Zone Placement Checks

For enterprise environments, confirm where each AKS dependency lives before provisioning:

• cluster resource group, node resource group strategy, and subscription ownership
• spoke VNet/subnet ownership and delegation model
• private DNS zone ownership and links to hub/spoke VNets
• firewall/NVA route ownership, FQDN rules, and egress logging
• Log Analytics / Azure Monitor Workspace / Managed Prometheus / Managed Grafana placement
• ACR, Key Vault, managed identities, Private Endpoints, and Defender for Cloud scope
• policy assignment scope, exemptions, and rollout mode

Stop if the architecture assumes platform-owned resources but the subscription or landing zone does not provide them.

Version and OS Currency

• Prefer the latest supported GA minor version available in the target region unless an organisation-wide LTS policy says otherwise.
• Use LTS intentionally; it is not automatically the best choice for every new cluster.
• Plan minor upgrades while the cluster is still within supported N-2/N-3 windows. Do not wait for platform-forced upgrades.
• For node OS, prefer Azure Linux 3 or Ubuntu 24.04 where supported. Avoid Azure Linux 2.0 and plan migration from Ubuntu 22.04 before its support deadline.
• Windows node pools require explicit lifecycle planning; do not default to Windows unless the workload requires Windows containers.

LTS (Long-Term Support) Trade-offs

AKS LTS is not a free upgrade-free button. Concrete trade-offs to weigh before adopting it:

• Pricing and tier. LTS adds a per-cluster pricing premium and requires the Premium AKS tier. Budget approval must include both the Premium tier delta and the LTS uplift.
• Support window. LTS extends a given minor version's support to approximately two years vs the standard rolling N-2/N-3 window. The extension applies only while the minor stays inside the published LTS list.
• Feature lag. New AKS and upstream Kubernetes features land on the Stable channel first; LTS clusters trail by one or more minors. Workloads dependent on recent APIs, alpha-to-beta promotions, or new add-on capabilities will hit gaps.
• When to choose LTS: regulated workloads with a multi-year change-management cycle and constrained upgrade windows; ISV products with long customer support tails; customers unable to absorb minor upgrades on the Stable cadence.
• When NOT to choose LTS: platform teams following modern continuous-upgrade practices, workloads that benefit from recent platform features, or any case where the LTS premium is not justified by an explicit change-management constraint.
• Verify before committing. The supported LTS minors and the pricing model change. Run the Freshness Gate against AKS release notes, Microsoft Learn, and current pricing before locking the decision into IaC or contracts.

Pricing Tier and SLA

The AKS pricing tier is not just a cost lever, it changes the control-plane reliability guarantee. Choose it as an explicit production decision, not an implicit default.

• [VERIFY] the exact SLA percentages (with vs without zones) and current tier pricing against Microsoft Learn and the AKS pricing page before locking them into a contract or SLO, the numbers change.
• IaC surface: --tier Free|Standard|Premium (CLI), sku.tier (Bicep/ARM), sku_tier (Terraform azurerm). Tier is changeable in place but is an approval-gated decision because it affects SLA, support, and the LTS path.
• Some features gate on tier: AKS Cost Analysis and LTS require Standard/Premium; do not assume a Free cluster can later enable them without a tier change.

Hyperscale Control Plane Scaling Profile (Public Preview)

For clusters whose bottleneck is the control plane itself (API server concurrency, pod scheduling throughput, or etcd pressure at thousands of nodes), AKS exposes a control plane scaling profile (controlPlaneScalingProfile.scalingSize) that replaces the default dynamically scaled control plane with preprovisioned, guaranteed-capacity tiers. Source: Hyperscale configuration for AKS (preview). Preview: [VERIFY] regional availability and H-tier pricing before designing around it; neither is published yet.

Concurrency doubles per tier; scheduling throughput grows linearly. Etcd is partitioned into six clusters by resource type (Events, Leases, Nodes, Pods, Secrets, Default for everything else). The 8 GiB per partition is a hard limit: design for 2 GiB or less per partition to keep latency low and startup times short.

Classification: Permanent decision. The profile can only be set at cluster creation, cannot be added to an existing cluster, and cannot be removed; reverting to the standard control plane requires delete-and-recreate. Resizing between H2/H4/H8 after creation is supported (az aks update --control-plane-scaling-size), so tier choice within hyperscale is Difficult, but hyperscale membership itself is Permanent. Decide at design time with an ADR.

Requirements and preview constraints:

• Pricing tier: Standard or Premium only; not supported on Free. Factor the tier fee plus the (unpublished) hyperscale cost into budget approval.
• Kubernetes version: 1.33.0 or later.
• Tooling: Azure CLI (aks-preview extension 21.0.0b8+), REST, and ARM/Bicep only: Terraform (azurerm) and the Azure SDKs do not support the profile yet. Terraform-standard organisations need a CLI/ARM sidecar path or must wait; do not fake the property in HCL.
• Feature flag: ControlPlaneScalingProfilePreview registered on the subscription before cluster creation; if the flag is not Registered at create time the profile is silently absent (az aks show --query controlPlaneScalingProfile returns null).
• Quota: one hyperscale cluster per subscription per region during preview; pick the test environment deliberately.
• Provisioning time: roughly 10 minutes for H2, 20 minutes for H4/H8, longer than a standard cluster.

When to use / not use:

• Use for: large-scale AI/ML training with high pod churn (thousands of nodes), high-throughput multitenant SaaS platforms, anticipated surge events (preprovision before launches/live events), staging-production control-plane parity, DR failover clusters that must absorb production-scale API load on activation, and large clusters with high peak-to-average API demand.
• Do not use by default: Microsoft's guidance is explicit that the default dynamically scaled control plane remains the recommended, cost-optimised option for most production, dev, and test workloads. Hyperscale is for mission-critical large-scale clusters with demonstrated or predicted API-server/scheduling pressure , not a general production baseline.

Operational notes:

• Monitoring: three platform metrics (also in Azure Managed Prometheus) indicate whether the tier holds up: apiserver_flowcontrol_executing_seats (concurrency approaching the tier's guaranteed seats = ceiling), scheduler_schedule_attempts_rate (rising unschedulable rate = scheduler falling behind), etcd_database_usage_percentage (per-partition utilisation; Microsoft's guidance: review resource counts in any partition approaching 50% before you hit storage limits - don't wait for 100%). Set alerts on all three before production.
• Azure Policy add-on: avoid on large hyperscale clusters; it increases API server load and affects control-plane performance at scale. This is a documented carve-out from the usual "Azure Policy from day one" baseline (see the SKILL.md non-negotiable); Deployment Safeguards is a different mechanism and remains recommended.
• Networking: Azure CNI Overlay is the recommended pod networking for high-pod-count hyperscale clusters (IP conservation).
• Workload hygiene: guaranteed capacity is not a licence for API abuse; excessive list/watch operations still consume the guaranteed seats.

CRD Lifecycle and Operator Governance

When introducing Kubernetes operators (Flux, Argo CD, KEDA, cert-manager, Kyverno, NVIDIA GPU Operator, etc.) or custom CRDs:

• Scope CRDs deliberately. Cluster-scoped CRDs are visible to all namespaces and all operators by default. Name-scope CRDs carefully (use the operator's domain prefix, e.g., clusterqueues.kueue.x-k8s.io) to avoid collisions with upstream or other operator schemas.
• Version upgrade ordering. CRD storage-version migrations must happen before or alongside the Kubernetes minor upgrade that removes the old served version. Failure to migrate leaves existing objects in a schema version the new API server does not serve. Add a CRD storage-version check (kubectl get crds -o json | jq '.items[] | select(.status.storedVersions | length > 1) | .metadata.name') to the pre-upgrade checklist.
• CRD version pruning risk. When an operator upgrades a CRD and removes a served apiVersion, any manifests in GitOps or CI pipelines using the old version fail immediately. Pin apiVersion references in manifests to the current served version and update them during the operator upgrade change window.
• Operator ownership. For every operator installed, document: who owns upgrades, what its compatibility matrix with the cluster's Kubernetes minor is, and whether it has its own CRD migrations. Treat operator upgrades as approval-gated changes, not routine config drift.
• Validate CRD health post-upgrade. After a Kubernetes minor or operator upgrade, run kubectl get crds and kubectl api-versions to confirm all expected CRDs and versions are present; a missing CRD is often silent until a controller tries to list it.

Stop Conditions

Stop and ask for architecture confirmation before provisioning if any of these are unknown:

• target region and data residency requirements
• hub-spoke/VNet/on-prem CIDR ranges
• private API server requirement
• outbound inspection requirement
• AKS Automatic vs Standard decision
• production availability target and zone support
• identity/RBAC ownership model
• GPU/Windows/confidential compute requirement
• dual-stack (IPv4/IPv6) requirement, and if yes, that Azure CNI Overlay (or Cilium overlay) is selected, Cilium data plane is chosen for IPv6 NetworkPolicy, and NAT Gateway exclusion / IPv6 egress path is designed
• stateful storage class, access mode, and zone/replication model for any persistent data
• AKS pricing tier (and therefore control-plane SLA) for the environment, and whether a hyperscale control plane scaling profile is required: creation-only and irremovable, so it must be answered before provisioning

Validation Checklist

# Region versions and previews
az aks get-versions --location <region> --output table
az feature list --namespace Microsoft.ContainerService --query "[?properties.state!='Registered']" --output table

# Cluster upgrades after creation
az aks get-upgrades --resource-group <rg> --name <cluster> --output table

# Node image / CVE posture per pool
az aks nodepool list --resource-group <rg> --cluster-name <cluster> \
  --query "[].{name:name,k8s:orchestratorVersion,nodeImage:nodeImageVersion}" --output table
az aks nodepool get-upgrades --resource-group <rg> --cluster-name <cluster> \
  --nodepool-name <pool> --output table

# Network profile after creation
az aks show --resource-group <rg> --name <cluster> --query "networkProfile" --output yaml

# Identity and OIDC
az aks show --resource-group <rg> --name <cluster> --query "{oidc:oidcIssuerProfile, workloadIdentity:securityProfile.workloadIdentity, identity:identity}" --output yaml

Compliance Alignment

For regulated workloads, map AKS architecture decisions to the relevant compliance framework before provisioning. The skill is framework-agnostic, but these anchors help reviewers locate the controls a baseline expects:

Recommended actions before production sign-off:

• Run the CIS AKS Benchmark scan (or an equivalent compliance scanner such as Kubescape, Trivy, or Defender for Cloud regulatory compliance) and remediate findings or document exceptions in an ADR.
• Map each ADR to one or more control IDs from the target framework so auditors can trace decisions back to controls.
• Confirm Defender for Cloud regulatory compliance dashboards cover the cluster, image registry, and supporting Azure resources.
• Treat unmapped controls as open questions; do not silently accept a framework gap.

This skill does not replace a formal control mapping or attestation activity, but production architecture should be defensible against the listed controls before any regulated workload is onboarded.

Threat Model

The following is a starting threat catalogue for new AKS clusters - it is not a substitute for a workload-specific threat-modelling exercise. Use it to confirm the baseline controls in this bundle cover the obvious AKS attack surfaces, then layer workload-specific analysis on top.

STRIDE applied to AKS surfaces:

The STRIDE rows above flag Spoofing, Tampering, Elevation of Privilege, and Information Disclosure but under-represent two letters. Denial of Service is addressed at two layers: API server flood is mitigated by --api-server-authorized-ip-ranges or a private endpoint fronted by Azure Front Door / WAF in front of ingress; pod-level resource exhaustion is mitigated by ResourceQuota and LimitRange per namespace plus PodDisruptionBudget for critical workloads. Repudiation is mitigated by RequestResponse-level audit logging for sensitive verbs (secrets access, RBAC mutations, exec/attach) forwarded to Microsoft Sentinel with immutable WORM retention - cross-link to the operations-resilience guide's audit section.

MITRE ATT&CK for Containers - map controls to these techniques explicitly:

• T1611 - Escape to Host. Mitigate via pod security standards (restricted), drop CAP_SYS_ADMIN, no hostPath/hostNetwork/hostPID outside system namespaces, Pod Sandboxing (Kata) or Confidential VM node pools for high-risk tenants, current node image with kernel patches.
• T1610 - Deploy Container. Mitigate via Azure Policy / Deployment Safeguards admission, signed-image enforcement, RBAC restricting create pods/create deployments to known service accounts, audit on workload creation in sensitive namespaces.
• T1552.007 - Unsecured Credentials: Container API. Mitigate via Workload Identity instead of mounted SP secrets, IMDS egress block (above), Secrets via Key Vault rather than Kubernetes Secrets where possible, restricted get secrets RBAC.
• T1613 - Container and Resource Discovery. Mitigate via least-privilege RBAC (no broad list on cluster-scoped resources), NetworkPolicy preventing lateral discovery, audit on list/watch against sensitive resource types, namespace isolation.
• T1525 - Implant Internal Image. Mitigate via signed-image policy, private ACR with quarantine, image scanning in CI and at admission, deny :latest and unpinned digests, periodic registry audit.

Map each ATT&CK technique above to a corresponding Sentinel analytics rule - see operations-resilience - Audit policy and SIEM forwarding for the starter detection rule set.

Use this catalogue as the input to an ADR-level threat model per workload - do not treat it as the final analysis.

Residual risks not covered by this catalogue

The STRIDE/ATT&CK baseline above is deliberately scoped to the AKS attack surface and does not mitigate several adjacent risk classes. Workload-specific threat modelling must layer additional controls on top to cover:

• Insider abuse by privileged operators (cluster admins, on-call SREs with break-glass access) - mitigated by Entra PIM with approval workflows, just-in-time elevation, and four-eyes review on production changes.
• Undisclosed zero-day vulnerabilities in containerd, kubelet, or Cilium - mitigated by EDR / Defender for Containers runtime detection and rapid CVE-response runbooks, since signature-based controls cannot catch the unknown.
• Base-image supply-chain compromise upstream of ACR (e.g., a compromised distroless source or a poisoned upstream package feed) - mitigated by ISV vetting, SBOM diffing across rebuilds, and provenance attestation (SLSA) on the source registry.
• Physical and datacentre-level risk in the underlying Azure platform - mitigated by Azure platform attestation, the Microsoft shared-responsibility model, and immutable audit retention outside the tenant's blast radius.

Treat these as explicit out-of-scope items for the cluster-foundations baseline and confirm coverage in the workload's own threat model before regulated onboarding.

Use With

• Top-level router: ../../SKILL.md
• Production workload controls: ../production-workload-controls/guide.md
• Full reference: ../../references/full-reference.md