GKE Observability
This reference covers monitoring, logging, and metrics configuration and troubleshooting for Google Kubernetes Engine (GKE). The golden path enables comprehensive observability including control-plane metrics and Managed Service for Prometheus (GMP).
MCP Tools:
get_cluster,list_k8s_events,get_k8s_logs,get_k8s_cluster_info,describe_k8s_resource. CLI-only:gcloud container clusters update --monitoring=...,gcloud logging read
Usage Attribution
Do not run live kubectl, gcloud, or curl commands unless the user names a
target cluster/project or asks to inspect a live cluster (output commands and
queries with placeholders, and skip Usage Attribution).
Before running any live command, read Usage Attribution,
prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)",
and append agent-skills/1.0 (skill:gke-observability) to User-Agent headers
(never in generated scripts, Terraform, or user-facing code).
Sub-Skill Reference Routing (Progressive Disclosure)
Complete Reference Index (no
scripts/directory or unlisted reference files exist — do not runlson the skill folder; read only the matching reference below when answering deep diagnostic, metric, log, or GMP questions):
- Control-Plane Metrics, Storage, Webhooks & Capacity:
Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not
publicly exposed) and customer-visible alternatives; cluster etcd /
etcd-compatible storage left, quota & per-resource scalability limits;
admission webhook latency, system namespace/Lease/upgrade exclusions,
APIServicediscovery bloat, client discovery caching bursts (/apis/custom.metrics.k8s.io,/apis/metrics.k8s.io) & proactive GKE Recommender insights; watch storms; controller-manager workqueue & scheduler lag; or control-plane headroom, VM autoscaling & node-count pre-scaling, network egress, connections (65k, no per-replica metric) & Protobuf. - Cloud Logging, Audit Attribution, APF & API Server Traces:
Read for
kube-apiserverslow-requestTrace[logs (trace.go& KEP-647);httplog.goclient load (user_agent,verb,apf_iseats); Cloud Audit Log noisy-neighbor, heavyLIST& unindexed CRDfieldSelectorscan tail-latency attribution; API Priority and Fairness (APF)429/504/404throttling;409 Conflicthot objects (ConfigMap->Leaseheartbeats) &RenewDeadline / 2leader-election timeouts;pods/execsaturation, Konnectivity (No agent available), disabled Cloud KMS keys, or in-cluster GCP IAM OAuth token webhook authorizer saturation. - Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP):
Read for control-plane availability (
kubernetes.io/cluster/kcp_healthchecksrate ratio & ruling out KCP outages); maintenance windows (in_maintenance_window), zonal resize downtime & cluster access errors (EOF/masterAuthorizedNetworksConfig); node readiness, registration (AUTO_REPAIR_NODES, Private Google Access) & persistent node-pool labels; pod startup timelines, crashing pod logs,tolerationSeconds: 0mass eviction & Autopilot preemption; or multi-burn-rate SLO alerting (GlobalRules) & GMP / Grafana /PodMonitoringmanifests & troubleshooting.
Golden Path Observability Defaults
| Setting | Golden Path Value | Notes |
|---|---|---|
loggingConfig components |
SYSTEM_COMPONENTS, WORKLOADS | Full workload logging |
monitoringConfig components |
SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane |
managedPrometheusConfig.enabled |
true |
Google-managed Prometheus |
advancedDatapathObservabilityConfig.enableMetrics |
true |
Dataplane V2 flow metrics |
loggingService |
logging.googleapis.com/kubernetes |
Cloud Logging |
monitoringService |
monitoring.googleapis.com/kubernetes |
Cloud Monitoring |
Enabling Full Monitoring & Logging
Include all four caveats whenever providing a --monitoring or --logging
command so users avoid accidental metric loss, surprise billing, or silent IAM
failures:
- Control-plane metrics and logs are NOT enabled by default. State this
outright:
API_SERVER,SCHEDULER, andCONTROLLER_MANAGERare off on every new cluster (as areDCGM,CADVISOR,KUBELET, and kube-statePOD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,STORAGE,JOBSET). OnlySYSTEMis enabled by default. Link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics and https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs. - The flag replaces, it does not append. Passing
--monitoringor--loggingoverrides the existing component list; pass the full desired list includingSYSTEM. - Control-plane and kube-state metrics bill per sample ingested via
Managed Service for Prometheus (GMP). Mention the cost impact on large
clusters; use
--monitoring=SYSTEMin non-production to reduce cost. - Verify APIs and IAM permissions (
PermissionDenied). Checkgcloud container clusters describe {cluster_name} --region {region} --format="yaml(loggingConfig,monitoringConfig)", ensurelogging.googleapis.comandmonitoring.googleapis.comare enabled, and verify the node service account / GKE service agent hasroles/monitoring.metricWriterandroles/container.defaultNodeServiceAgent.
CLI flag vs. API enum spelling (mixing them fails the command):
Component gcloud --monitoring=monitoringConfigAPI enumSystem SYSTEMSYSTEM_COMPONENTSAPI server API_SERVERAPISERVERController mgr CONTROLLER_MANAGERCONTROLLER_MANAGER
# Enable golden path monitoring and control-plane logging suite
gcloud container clusters update {cluster_name} --region {region} \
--monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
--logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \
--quiet
# Enable Managed Prometheus and Dataplane V2 flow/policy observability
gcloud container clusters update {cluster_name} --region {region} \
--enable-managed-prometheus \
--enable-dataplane-v2-flow-observability \
--quietCore GKE Metrics & Alerting Baseline
Explicitly name Google Cloud Monitoring when proposing dashboards and
alerts, and cover cluster/node CPU and memory capacity, pod restarts
(CrashLoopBackOff), volume space exhaustion, and API server latency
(apiserver_request_duration_seconds). Direct control-plane VM CPU/RAM metrics
are not publicly exposed — read
Control-Plane Metrics, Storage, Webhooks & Capacity
whenever asked about control-plane VM CPU/memory or deep control-plane metrics.
| Metric | Source | Use / Alert Threshold |
|---|---|---|
apiserver_request_duration_seconds |
API Server (GMP) | Control-plane latency (Alert: P99 > 5s) |
scheduler_scheduling_attempt_duration_seconds |
Scheduler (GMP) | Current GKE scheduling latency (scheduler_schedule_attempts_total{result="error"} > 0) |
container_cpu_usage_seconds_total |
cAdvisor (Node) | Worker-node container CPU usage |
container_memory_working_set_bytes |
cAdvisor (Node) | Worker-node container memory usage |
kubernetes.io/node/cpu/core_usage_time |
Cloud Monitoring | Node CPU usage (kubernetes.io/node/memory/used_bytes for memory) |
kube_pod_status_phase |
kube-state-metrics | Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics) |
kube_pod_container_status_restarts_total |
kube-state-metrics | Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics) |
kube_node_status_condition |
kube-state-metrics | Node readiness (condition="Ready", status!="True"; or kubernetes.io/node/status_condition) |
kubelet_volume_stats_used_bytes / capacity |
Kubelet | PVC capacity utilization (Alert: > 85%) |
DCGM_FI_DEV_GPU_UTIL |
DCGM (GMP) | GPU utilization (Alert: > 95% sustained) |
Live Resource Usage & Cloud Logging Queries
Use kubectl top for live usage (no MCP or gcloud equivalent):
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
kubectl top pods --containers -n {namespace}Use gcloud logging read (or Cloud Logging Log Explorer) with LQL filters:
# Control-plane component logs (requires API_SERVER / SCHEDULER / CONTROLLER_MANAGER enabled)
gcloud logging read 'resource.type="k8s_control_plane_component" AND resource.labels.cluster_name="{cluster_name}"' --project {project_id} --limit 50 --quiet
# Container errors: resource.type="k8s_container" AND resource.labels.container_name="{container_name}" AND severity>=ERROR
# OOMKilled events: resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"
# FailedScheduling: resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"
# Cloud Audit Logs: resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"
# Dataplane V2 network policy / packet drop logs: logName:"policy_action" AND jsonPayload.disposition="DENY"Distributed Tracing, Profiling & Recent Additions
- Cloud Trace & OpenTelemetry: Deploy the OpenTelemetry Collector
(
googlecloudexporter with Workload Identity) or enable Managed OpenTelemetry for GKE (Preview) on GKE 1.34.1-gke.2178000+ (gcloud beta container clusters update {cluster_name} --region {region} --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS). - Cloud Profiler: Continuous low-overhead production CPU/memory profiling.
- PSI (Pressure Stall Information) metrics: cAdvisor
container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_totalvia GMPClusterNodeMonitoring(GKE 1.35+).