Control-Plane Metrics, Storage, Webhooks & Capacity
Table of Contents
- 1. Control-Plane Metrics Prerequisites & KCP VM Boundaries (Lines 13–34)
- 2. Etcd-Compatible Storage Backend & Remaining-Capacity Queries (Lines 36–52)
- 3. Real-Time Storage Quota Alerts, Per-Resource Footprint & Cleanup (Lines 54–92)
- 4. Admission Webhooks, Exclusions, Discovery & Recommendations (Lines 94–159)
- 5. Watch Activity & Watch Storms (Lines 161–167)
- 6. Pod Churn, Controller-Manager Workqueues & Scheduler Lag (Lines 169–187)
- 7. Control-Plane Headroom, Network Egress, Connections & Protobuf (Lines 189–224)
1. Control-Plane Metrics Prerequisites & KCP VM Boundaries
Control-plane (KCP) VM CPU and RAM/memory usage are not publicly exposed: GKE manages control-plane VMs internally and exposes no KCP VM CPU or RAM/memory usage metrics in Cloud Monitoring (
kubernetes.io/cluster/*,kubernetes.io/master/*, or worker-node-onlykubernetes.io/node/*/container_*) or Managed Service for Prometheus. Instead of KCP VM CPU, monitorkubernetes.io/cluster/kcp_healthchecks,apiserver_request_duration_seconds, and APF concurrency saturation (apiserver_flowcontrol_priority_level_seat_utilizationandapiserver_current_inflight_requests). Instead of KCP VM RAM/memory, monitorapiserver_resource_size_estimate_bytes,apiserver_storage_size_bytes(kubernetes.io/cluster/storage/used_bytesandkubernetes.io/cluster/storage/limit_bytes), andkubernetes.io/cluster/kcp_healthchecks.
Whenever recommending any apiserver_*, scheduler_*, workqueue_*, or
job_controller_* metric below, state that it is a Managed Service for
Prometheus (GMP) control-plane metric that is not enabled by default
(requires --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,... and
--enable-managed-prometheus; see SKILL.md for CLI vs. API enum spellings)
and link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics.
2. Etcd-Compatible Storage Backend & Remaining-Capacity Queries
In self-managed Kubernetes, operators check remaining etcd space via etcdctl endpoint status or raw etcd metrics (etcd_mvcc_db_total_size_in_bytes,
etcd_server_quota_backend_bytes). GKE uses a managed etcd-compatible
storage backend for cluster state and does not expose etcdctl or raw
etcd_* server metrics—always state explicitly that GKE uses an
etcd-compatible storage backend when answering cluster etcd storage
questions. Instead, query the default Cloud Monitoring SYSTEM metrics
kubernetes.io/cluster/storage/used_bytes and
kubernetes.io/cluster/storage/limit_bytes on monitored_resource="k8s_cluster"
(enabled out-of-the-box without API_SERVER or GMP opt-in):
# Remaining control-plane storage bytes (divide used_bytes / limit_bytes for utilization ratio)
kubernetes_io:cluster_storage_limit_bytes{monitored_resource="k8s_cluster"} - kubernetes_io:cluster_storage_used_bytes{monitored_resource="k8s_cluster"}3. Real-Time Storage Quota Alerts, Per-Resource Footprint & Cleanup
- Real-time storage utilization vs. 6h quota delay: In Cloud Monitoring,
kubernetes.io/cluster/storage/used_bytesandkubernetes.io/cluster/storage/limit_bytessample every 60s (avoiding the 6-hour delay of GCP quota metrics; opt-in GMP also exposesapiserver_storage_size_bytes). Configure an alert whenused_bytes / limit_bytesapproaches capacity on GKE's etcd-compatible control-plane datastore so leaked Pods, Jobs, or Secrets are pruned beforeetcdserver: mvcc: database space exceededwrite locks occur. - Per-resource byte footprint & GKE scalability limits: Query
apiserver_resource_size_estimate_bytes(orkubernetes.io/master/apiserver/resource_size_estimate_bytes) andapiserver_storage_objectsgrouped bygroupandresourceto find which resource types or CRDs consume the most storage bytes. Compare against GKE scalability limits (https://cloud.google.com/kubernetes-engine/docs/concepts/planning-large-clusters: keep total Pod or ReplicaSet state below ~800 MiB per resource type, keep individual ConfigMaps/Secrets below 1 MiB vs. the 1.5 MiB etcd hard limit, and stay below ~30,000 Secrets/ConfigMaps). - Quota exhaustion (
mvcc: database space exceeded) & CRD bloat cleanup: When creations, updates, or Lease renewals fail withetcdserver: mvcc: database space exceeded, delete bloated objects in small paced batches:- Version behavior on failing deletes: State explicitly that while
some earlier GKE versions blocked all edits (including deletes) on
hitting storage quota, this is no longer the case in newer GKE versions
(
kubectl deleteremains permitted). - Finished
batch/v1Jobs and CronJob pods (concurrencyPolicy: Allow): set.spec.ttlSecondsAfterFinishedon Jobs andsuccessfulJobsHistoryLimit/failedJobsHistoryLimiton CronJobs; retained ReplicaSets (lower.spec.revisionHistoryLimit), Kyvernoadmissionreports, oversized Secrets/ConfigMaps, or stuck finalizers.
- Version behavior on failing deletes: State explicitly that while
some earlier GKE versions blocked all edits (including deletes) on
hitting storage quota, this is no longer the case in newer GKE versions
(
- Excessive
VolumeSnapshotobjects hangingLIST(<50,000 limit): When newVolumeSnapshotobjects take 30+ minutes to become ready or deleted snapshots stay stuck inTerminating, checkapiserver_storage_objectsandapiserver_resource_size_estimate_bytes(pluskube-apiserverslowLISTTrace[logs) forvolumesnapshotsandvolumesnapshotcontentscausingsnapshot-controllerLISTtimeouts; prune old snapshots and enforce a retention policy keeping the total count well below 50,000.
4. Admission Webhooks, Exclusions, Discovery & Recommendations
- Measuring webhook latency & errors: When API calls or pod creations time
out (
context deadline exceeded), queryapiserver_admission_webhook_admission_duration_seconds(grouped byname,type,operation, andrejected) andapiserver_admission_webhook_rejection_countalongsidekube-apiserverslow-requestTrace[logs, GKE Webhook Insights (Verify webhook with unavailable endpoints), andkubernetes.io/cluster/kcp_healthchecks(to verify the GKE control plane itself is healthy before requesting an upsize). Scale/fix backing webhook pods, tightentimeoutSeconds(e.g., 2–5s), scopenamespaceSelector, or setfailurePolicy: Ignorewhere appropriate. - System namespace & Lease exclusions (preventing controller leadership loss
& CNI
NotReadynodes): InspectValidatingWebhookConfigurationandMutatingWebhookConfigurationrules. Excludekube-systemandkube-node-leasevianamespaceSelectoron third-party webhooks (Kyverno, Datadog, Vault, OpenTelemetry, or OPA Gatekeeper'sadmission.gatekeeper.sh/ignorelabel) and removecoordination.k8s.ioleases(coordination.k8s.io/leases) from webhook rules so slow/crashed webhooks never block controller Lease renewals orkube-systemCNI pods (netd,anetd, leaving new nodes stuckNotReadywithCNI plugin not initialized). - Control-plane upgrade failures & addon deadlocks: When upgrades fail and
roll back or Cluster Autoscaler stops initializing after an upgrade with
failurePolicy: Failwebhooks, check Cloud Audit Logs or webhook metrics/logs during the upgrade window for denied/timed-out requests fromsystem:apiserver(onrbac.authorization.k8s.io/clusterrolebindings) orsystem:addon-manager(on systemcustomresourcedefinitionssuch ascapacitybuffers.autoscaling.x-k8s.io). Exemptrbac.authorization.k8s.iobootstrap bindings and GKE system CRDs (*.gke.io,autoscaling.x-k8s.io) from webhook rules or setfailurePolicy: Ignore. - Unavailable webhook endpoints, OOMKilled pods,
APIService& discovery:- Inspect
MutatingWebhookConfiguration/ValidatingWebhookConfigurationservice references, verifyEndpointSliceaddresses match running pod IPs, inspect CRD conversion webhooks (spec.conversion.webhook), and check backing pod health and OOM kills (jsonPayload.reason="OOMKilling",kubectl describe pod {pod_name},kubectl logs {pod_name} --previous). Increase memory limits on OOMKilled webhook pods (e.g., Gatekeeper or Kubeflowcache-server), fix the deployment, or scope rules/failurePolicy. - When every
kubectlcommand takes 15–30s, runkubectl get apiservicesandkubectl --v=6alongsideapiserver_response_sizesandapiserver_request_duration_secondsto spot unavailableAPIServicebackends or multi-megabyte discovery documents from aggregated APIs (custom.metrics.k8s.io,external.metrics.k8s.io). Disable or filter unbounded custom metrics exporters (e.g., Airflow or Prometheus adapters exporting thousands of series into API discovery). - When custom scripts, cron jobs, or CLI invocations flood
/apis/custom.metrics.k8s.ioand/apis/metrics.k8s.iowith thousands of requests per minute (visible in Cloud Audit Logs /httplog.gouserAgentandapiserver_request_total), clients are polling API discovery without a persistent local cache; enable client-side discovery caching (client-godisk.NewCachedDiscoveryClientForConfig/memory.NewMemCacheClientor a persistent sharedkubectl --cache-dirrather than ephemeral per-run directories).
- Inspect
- Proactive GKE Control-Plane & Webhook Insights / Recommendations: Check
GKE Insights and Recommendations in the Cloud Console or via
gcloud recommender insights listandgcloud recommender recommendations list(--insight-type=google.container.DiagnosisInsight/--recommender=google.container.DiagnosisRecommender). Warn users against ignoring GKE control-plane recommendations—especially admission webhook findings (unavailable endpoints, high latency, missingkube-systemexclusions) or CRDs with an invalidcaBundle—because they can overload or destabilize the control plane. Link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/optimize-webhooks and https://docs.cloud.google.com/kubernetes-engine/docs/how-to/crd-with-invalid-caBundle.
5. Watch Activity & Watch Storms
When a misconfigured agent or DaemonSet is suspected of causing a watch storm or
repeatedly re-establishing broken watches, query GMP apiserver_longrunning_requests
(verb="WATCH", grouped by group, resource, scope) for active watchers,
apiserver_watch_events_total (grouped by group, kind, version) for watch
event volume/rate, and apiserver_watch_events_sizes for watch payload sizes.
6. Pod Churn, Controller-Manager Workqueues & Scheduler Lag
When batch job bursts (tens of thousands of concurrent Jobs) delay pod
creation/scheduling by 10–20 minutes—saturating kube-controller-manager and
kube-scheduler QPS/workqueues, often exacerbated by crashing DaemonSets—or
when custom controllers fall behind:
- Enable
SCHEDULERandCONTROLLER_MANAGERcontrol-plane metrics. - Query
workqueue_depth,workqueue_queue_duration_seconds,workqueue_adds_total,workqueue_retries_total, andjob_controller_*metrics (such asjob_controller_job_sync_duration_seconds) onkube-controller-manager. - Query
scheduler_pending_pods,scheduler_schedule_attempts_total, andscheduler_scheduling_attempt_duration_secondsonkube-scheduler, plus mutatingapiserver_request_total(CREATE,DELETE,PATCH,UPDATE) to measure object churn. - Correlate workqueue growth (e.g.,
pod-garbage-collectororreplicasetlag) with API server latency and high churn; prune orphaned objects, tune controller concurrency, and pace or batch job submissions.
7. Control-Plane Headroom, Network Egress, Connections & Protobuf
- Saturation, headroom & control-plane VM autoscaling: Monitor
apiserver_flowcontrol_priority_level_seat_utilization,apiserver_current_inflight_requests, andapiserver_request_duration_seconds(plusapiserver_flowcontrol_request_wait_duration_secondsandapiserver_flowcontrol_rejected_requests_total). Always cover three operational guidance points when discussing control-plane headroom or preventing overload:- Correlate spikes with workload changes (new DaemonSets, unpaginated
LISTcalls, bursty controllers) to roll back or tune workloads before saturatingkube-apiserver. - Automatic control-plane VM scaling: Highlight that GKE control-plane VMs are autoscaled to match observed load, so control-plane capacity adjusts dynamically as cluster activity grows.
- Proactive resize via node count: Explain that the GKE control plane uses cluster node count as one of its autoscaling signals, so adding more nodes upfront (for example, ahead of a planned batch or traffic surge) is a relatively simple way to force a proactive control-plane resize—while noting that the exact scaling rules change frequently, so there is no reliable fixed threshold that GKE can share.
- Correlate spikes with workload changes (new DaemonSets, unpaginated
- Control-plane network egress, connections (
65k), & Protobuf: State explicitly out loud that GKE does not offer a per-replica control-plane connection count metric (so active watch/long-running stream metrics serve only as an indirect proxy). Check for heavy response/watch payload egress, connection limits (65k), or high JSON serialization overhead viaapiserver_response_sizes,apiserver_watch_events_sizes,apiserver_longrunning_requests, andapiserver_request_totalbycontentType(application/jsonvs.application/vnd.kubernetes.protobuf). Always advise both mitigations: (1) configure custom controllers and high-volume clients to use Protobuf serialization (application/vnd.kubernetes.protobuf) instead of JSON, and (2) scopeWATCH/LISTselectors (labelSelector,fieldSelector, and namespace-scoped informers) to reduce control-plane network egress and keep active connections within safe scalability limits.