Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP)
Table of Contents
- 1. Control-Plane Health (
kcp_healthchecks) & Maintenance Windows (Lines 10–65) - 2. Node Health, Registration Failures & Persistent Node Labels (Lines 67–99)
- 3. Pod Startup Timeline, Mass Evictions & Autopilot Preemption (Lines 101–128)
- 4. Multi-Burn-Rate SLO Alerting & GMP Troubleshooting (Lines 130–204)
1. Control-Plane Health (kcp_healthchecks) & Maintenance Windows
To check whether Google considers a GKE cluster's control plane healthy, use the
kubernetes.io/cluster/kcp_healthchecks system metric (on the k8s_cluster
monitored resource, included in the default SYSTEM metrics package):
Metric semantics & rate-ratio query:
kubernetes.io/cluster/kcp_healthchecksis an incremental (DELTA/CUMULATIVEcounter) metric sampled every 60s that counts Google-internal control-plane probes, labeled by booleansuccess("true"or"false"). Chart or alert on control-plane availability by dividing the rate of successful checks (success="true") by the overall rate of checks across allsuccessvalues:sum by (project_id, location, cluster_name) (rate(kubernetes_io:cluster_kcp_healthchecks{monitored_resource="k8s_cluster", success="true"}[5m])) / sum by (project_id, location, cluster_name) (rate(kubernetes_io:cluster_kcp_healthchecks{monitored_resource="k8s_cluster"}[5m]))Ruling out a GKE control-plane outage (
kcp_healthchecksinnocence): Checkkubernetes.io/cluster/kcp_healthchecks(rate ratio) alongsideapiserver_request_duration_secondsandapiserver_request_total(code="429"/5xx) in Cloud Monitoring. Whenkcp_healthchecksand API server latency are healthy with no429throttling, slow pod deletions or client delays originate from client-side request concurrency, pod finalizers/graceful termination, or the customer's own pod-creating service rather than a GKE control-plane fault.Maintenance windows, zonal resize downtime & cluster access errors:
- Maintenance window metrics & policy limits: Query
kubernetes.io/cluster/in_maintenance_window(booleanGAUGEonmonitored_resource="k8s_cluster", included in defaultSYSTEMmetrics; PromQL:kubernetes_io:cluster_in_maintenance_window{monitored_resource="k8s_cluster"}) in Cloud Monitoring and correlate withgcloud container operations list(or Cloud Audit Logs forRESIZE_CLUSTERorUPGRADE_MASTER) to verify whether brief API unavailability, blocked updates, or lease-loss alerts occurred during scheduled maintenance. Move the maintenance window to a lower-traffic period or usekubernetes.io/cluster/in_maintenance_windowto suppress transient alerts. Ensure maintenance windows/exclusions do not violate GKE limits (see https://docs.cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions#limitations-maint-exclusions, requiring at least 48 hours of total maintenance availability in any 32-day (and 92-day) rolling window, in contiguous blocks of at least 4 hours) so routine control-plane upgrades and repairs are not blocked. - Zonal resize downtime vs. access errors (
EOF/CLOUD_KMS_KEY_ERROR): Correlatekubernetes.io/cluster/kcp_healthchecksandkubernetes.io/cluster/in_maintenance_windowwithgcloud container operations list(RESIZE_CLUSTERorUPGRADE_MASTER): zonal clusters have a single control-plane replica that experiences expected 15–30 minute API downtime during resizes and upgrades (mitigate by migrating to a regional cluster). Ifkubectlfails withEOForCLOUD_KMS_KEY_ERROR, checkgcloud container clusters describe {cluster_name}(masterAuthorizedNetworksConfig,databaseEncryption, and cluster conditions) and project IAM/APIs to resolve client IP allowlist blocks (EOF), disabled Cloud KMS keys (CLOUD_KMS_KEY_ERROR), a disabledcontainer.googleapis.comAPI, or missing GKE service agent IAM bindings.
- Maintenance window metrics & policy limits: Query
2. Node Health, Registration Failures & Persistent Node Labels
When proposing node health charts, combine both metrics below and state both metrics' limitations explicitly (neither metric is complete alone):
kubernetes.io/node/status_condition(onmonitored_resource="k8s_node", filtered bycondition="Ready"/status_condition="Ready", grouped bystatus=True/False/Unknown; PromQL:count by (status) (kubernetes_io:node_status_condition{monitored_resource="k8s_node", condition="Ready"})): Tracks healthy vs. unhealthy registered nodes, but only reports values for nodes that have successfully bootstrapped (missing VMs stuck failing registration).compute.googleapis.com/instance_group/size(ongce_instance_group, filtered byinstance_group_name="gke-<cluster_name>-.*"): Tracks the total number of GCE nodes in the cluster, but does not differentiate between healthy and unhealthy nodes (so comparinginstance_group/sizeagainststatus_condition="Ready"is required to spot unbootstrapped VMs).- Node registration failures (
AUTO_REPAIR_NODES): When new node pools fail registration before reportingkubernetes.io/node/status_condition, query the consumer project's Cloud Audit Logs (cloudaudit.googleapis.com/activity) for GCE instance creation denials (403/constraints/gcp.resourceLocations), verify Private Google Access (privateIpGoogleAccess) is enabled on the cluster subnet, and check VPC routes/firewall rules so traffic to*.googleapis.com,*.gcr.io, and*.pkg.devis not blackholed into a custom VPN or firewall. - Node labels overwritten on upgrade (
kubectl label nodevs. Node Pool API):gcp-controller-managerreconciles node labels declared in the GKE Node Pool API specification during control-plane upgrades and node repairs (visible assystem:gcp-controller-managernode patches in Cloud Audit Logs), reverting manualkubectl label node {node_name} {key}={val} --overwriteedits on GKE-managed label keys. Update node pool labels persistently viagcloud container node-pools update {pool_name} --node-labels=...(or Terraformnode_config.labels).
3. Pod Startup Timeline, Mass Evictions & Autopilot Preemption
- End-to-end pod startup lifecycle & crashing pods: To pinpoint whether a
slow-starting pod was delayed during controller creation, scheduling,
kubeletpickup, or container startup, correlate Kubernetes Events (resource.type="k8s_pod"ork8s_event) with Cloud Audit Logs (cloudaudit.googleapis.com/activity), optional control-plane component logs (kube-scheduler,kube-controller-manager,kube-apiserverunderresource.type="k8s_control_plane_component"), and nodekubeletlogs (resource.type="k8s_node"). For crashing pods, runkubectl logs {pod_name} --previous(or-p),kubectl describe pod {pod_name}, and query Cloud Logging (resource.type="k8s_container"). tolerationSeconds: 0mass pod eviction: If pods are simultaneously evicted during a brief control-plane hiccup when nodes briefly enterNodeStatusUnknown, inspect workload Pod/Deployment specs (kubectl get deploy -o yaml) for customtolerationsonnode.kubernetes.io/unreachable(NoExecute) andnode.kubernetes.io/not-ready(NoExecute) withtolerationSeconds: 0(which overrides Kubernetes's default 300s eviction grace period), and remove thetolerationSeconds: 0override.- Autopilot single-node bin-packing & system-addon preemption: If all
replicas of a Deployment pack onto a single node at its
maxPodslimit (32/32on Autopilot), surgingsystem-cluster-criticaladdons (priority2000000000) during a control-plane upgrade trigger scheduler preemption, which bypassesPodDisruptionBudgetwhen no other capacity is immediately available. ConfiguretopologySpreadConstraintsorpodAntiAffinityacross nodes/zones so replicas never co-locate on a single node, and assign an appropriatePriorityClass.
4. Multi-Burn-Rate SLO Alerting & GMP Troubleshooting
Prometheus / Grafana / PagerDuty Control-Plane SLO Alerting
When generating Grafana dashboards and multi-burn-rate Prometheus alert rules
for GKE control-plane health (routed to PagerDuty or Slack via Alertmanager or
Cloud Monitoring notification channels; output YAML/JSON directly inline without
scratch generator scripts), use GlobalRules (monitoring.googleapis.com/v1,
required when evaluating against Cloud Monitoring / GMP system metrics) covering
apiserver_request_duration_seconds (e.g., fast 5m/1h 14.4x + slow
30m/6h 6x burn rates), apiserver_admission_webhook_admission_duration_seconds,
apiserver_flowcontrol_rejected_requests_total, and storage utilization
(kubernetes_io:cluster_storage_used_bytes / kubernetes_io:cluster_storage_limit_bytes
or apiserver_storage_size_bytes), queried in Grafana via the GMP data source
(GET HTTP method and two-argument label_values(<metric>, <label>)).
Always state explicitly that GKE control-plane apiserver_* metrics are NOT
enabled by default, and instruct how to enable them (gcloud container clusters update {cluster_name} --region {region} --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,... --enable-managed-prometheus, noting --monitoring replaces the existing list).
Troubleshooting Managed Service for Prometheus (GMP)
Stay read-only (kubectl get / describe / logs) and propose config changes:
- First: split ingestion-side vs. query-side: Query
upin the Metrics Explorer PromQL tab in Cloud Monitoring. Ifupreturns data, ingestion works and the issue is query-side (Grafana / PromQL / permissions). Ifupis empty, the issue is ingestion-side (collectors, scrape config, or write IAM). - Ingestion-side checks:
- GMP system pods: Verify GMP is active via
gcloud container clusters describe {cluster_name}and inspect pods ingmp-system(Standard) orgke-gmp-system(Autopilot) (gmp-operator,collectorDaemonSet, andrule-evaluator):
Akubectl get pods -n gmp-system # gke-gmp-system on Autopilot kubectl logs -n gmp-system -l app.kubernetes.io/name=collector -c prometheuscollectorinCrashLoopBackOffwithOOMKilledindicates high cardinality—drop unneeded series/labels viametricRelabelingor apply a VPA. PodMonitoringvs.ClusterPodMonitoring: Always writePodMonitoringmanifests withapiVersion: monitoring.gke.io/v1(kind: PodMonitoringis namespace-scoped;ClusterPodMonitoringis cluster-wide), settingspec.selector.matchLabelsto match pod labels andspec.endpointswithport(container port name or number),path(e.g.,/metrics), andinterval(e.g.,30s).- Target status debugging: Temporarily patch
OperatorConfigingmp-publicwithfeatures.targetStatus.enabled: true, runkubectl describe podmonitoring {name}(Active Targets,Unhealthy Targets,Last Error), then disable to prevent operator OOMs on large clusters.
- GMP system pods: Verify GMP is active via
- Permissions (
403/PermissionDenied): GMP inherits the node service account. Ingestion requiresroles/monitoring.metricWriter(Permission monitoring.timeSeries.create deniedin collector logs);rule-evaluatorand query paths (including Grafana via Workload Identity) requireroles/monitoring.viewer. - Rule & alert evaluation (
Rules/ClusterRules/GlobalRules): UseGlobalRulesto evaluate rules against Cloud Monitoring system/control-plane metrics (RulesandClusterRulessilently return no data for them). Checkrule-evaluatorlogs (-c evaluator). - Query-side (Grafana / PromQL):
- Point the Grafana data source at the GMP frontend query proxy (not
localhost:9090) and set HTTP Method toGET(POSTfails withno match[] parameter provided). - Use the two-argument
label_values(<metric>, <label>)in Grafana template variables (single-argumentlabel_values(<label>)is unsupported by GMP). - Include a
monitored_resourcelabel matcher when querying Cloud Monitoring metrics that exist across multiple resource types.
- Point the Grafana data source at the GMP frontend query proxy (not
- Cost, cardinality & quota (
429/RESOURCE_EXHAUSTED): Inspect Cloud Monitoring Metrics Management to find top billable/high-cardinality metrics. Prune withmetricRelabeling(action: droporaction: labeldropfor unbounded labels likeuser_id/request_id) or raise the scrapeintervalbefore requesting quota increases.