Index of every page on this site: llms.txt
All skills
google avatar

/gke-observability

@5fb62c2
by googlegoogle/skills24k stars
2,026
Run it yourself

Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection, to diagnose GKE control plane, API server, admission webhook, etcd storage, node registration or label, and pod eviction, preemption, or scheduling issues using GKE metrics, Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets, PodMonitoring misconfiguration, rule/alert evaluation failures, and monitoring permission errors. Don't use to configure local application logging frameworks or external APMs outside GKE.

Use this Skill: https://skilld.dev/gh/google/skills/gke-observability

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈184 tokens for metadata: the name and description. ≈3.2k when used: this file. ≈11k more on demand in 4 files.

Description uses 9.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈738 characters, or 9.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

GKE Observability

This reference covers monitoring, logging, and metrics configuration and troubleshooting for Google Kubernetes Engine (GKE). The golden path enables comprehensive observability including control-plane metrics and Managed Service for Prometheus (GMP).

MCP Tools: get_cluster, list_k8s_events, get_k8s_logs, get_k8s_cluster_info, describe_k8s_resource. CLI-only: gcloud container clusters update --monitoring=..., gcloud logging read

Usage Attribution

Do not run live kubectl, gcloud, or curl commands unless the user names a target cluster/project or asks to inspect a live cluster (output commands and queries with placeholders, and skip Usage Attribution). Before running any live command, read Usage Attribution, prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)", and append agent-skills/1.0 (skill:gke-observability) to User-Agent headers (never in generated scripts, Terraform, or user-facing code).

Sub-Skill Reference Routing (Progressive Disclosure)

Complete Reference Index (no scripts/ directory or unlisted reference files exist — do not run ls on the skill folder; read only the matching reference below when answering deep diagnostic, metric, log, or GMP questions):

  1. Control-Plane Metrics, Storage, Webhooks & Capacity: Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not publicly exposed) and customer-visible alternatives; cluster etcd / etcd-compatible storage left, quota & per-resource scalability limits; admission webhook latency, system namespace/Lease/upgrade exclusions, APIService discovery bloat, client discovery caching bursts (/apis/custom.metrics.k8s.io, /apis/metrics.k8s.io) & proactive GKE Recommender insights; watch storms; controller-manager workqueue & scheduler lag; or control-plane headroom, VM autoscaling & node-count pre-scaling, network egress, connections (65k, no per-replica metric) & Protobuf.
  2. Cloud Logging, Audit Attribution, APF & API Server Traces: Read for kube-apiserver slow-request Trace[ logs (trace.go & KEP-647); httplog.go client load (user_agent, verb, apf_iseats); Cloud Audit Log noisy-neighbor, heavy LIST & unindexed CRD fieldSelector scan tail-latency attribution; API Priority and Fairness (APF) 429/504/404 throttling; 409 Conflict hot objects (ConfigMap -> Lease heartbeats) & RenewDeadline / 2 leader-election timeouts; pods/exec saturation, Konnectivity (No agent available), disabled Cloud KMS keys, or in-cluster GCP IAM OAuth token webhook authorizer saturation.
  3. Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP): Read for control-plane availability (kubernetes.io/cluster/kcp_healthchecks rate ratio & ruling out KCP outages); maintenance windows (in_maintenance_window), zonal resize downtime & cluster access errors (EOF / masterAuthorizedNetworksConfig); node readiness, registration (AUTO_REPAIR_NODES, Private Google Access) & persistent node-pool labels; pod startup timelines, crashing pod logs, tolerationSeconds: 0 mass eviction & Autopilot preemption; or multi-burn-rate SLO alerting (GlobalRules) & GMP / Grafana / PodMonitoring manifests & troubleshooting.

Golden Path Observability Defaults

Setting Golden Path Value Notes
loggingConfig components SYSTEM_COMPONENTS, WORKLOADS Full workload logging
monitoringConfig components SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER Full suite including control-plane
managedPrometheusConfig.enabled true Google-managed Prometheus
advancedDatapathObservabilityConfig.enableMetrics true Dataplane V2 flow metrics
loggingService logging.googleapis.com/kubernetes Cloud Logging
monitoringService monitoring.googleapis.com/kubernetes Cloud Monitoring

Enabling Full Monitoring & Logging

Include all four caveats whenever providing a --monitoring or --logging command so users avoid accidental metric loss, surprise billing, or silent IAM failures:

  1. Control-plane metrics and logs are NOT enabled by default. State this outright: API_SERVER, SCHEDULER, and CONTROLLER_MANAGER are off on every new cluster (as are DCGM, CADVISOR, KUBELET, and kube-state POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, STORAGE, JOBSET). Only SYSTEM is enabled by default. Link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics and https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs.
  2. The flag replaces, it does not append. Passing --monitoring or --logging overrides the existing component list; pass the full desired list including SYSTEM.
  3. Control-plane and kube-state metrics bill per sample ingested via Managed Service for Prometheus (GMP). Mention the cost impact on large clusters; use --monitoring=SYSTEM in non-production to reduce cost.
  4. Verify APIs and IAM permissions (PermissionDenied). Check gcloud container clusters describe {cluster_name} --region {region} --format="yaml(loggingConfig,monitoringConfig)", ensure logging.googleapis.com and monitoring.googleapis.com are enabled, and verify the node service account / GKE service agent has roles/monitoring.metricWriter and roles/container.defaultNodeServiceAgent.

CLI flag vs. API enum spelling (mixing them fails the command):

Component gcloud --monitoring= monitoringConfig API enum
System SYSTEM SYSTEM_COMPONENTS
API server API_SERVER APISERVER
Controller mgr CONTROLLER_MANAGER CONTROLLER_MANAGER
# Enable golden path monitoring and control-plane logging suite
gcloud container clusters update {cluster_name} --region {region} \
  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
  --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \
  --quiet

# Enable Managed Prometheus and Dataplane V2 flow/policy observability
gcloud container clusters update {cluster_name} --region {region} \
  --enable-managed-prometheus \
  --enable-dataplane-v2-flow-observability \
  --quiet

Core GKE Metrics & Alerting Baseline

Explicitly name Google Cloud Monitoring when proposing dashboards and alerts, and cover cluster/node CPU and memory capacity, pod restarts (CrashLoopBackOff), volume space exhaustion, and API server latency (apiserver_request_duration_seconds). Direct control-plane VM CPU/RAM metrics are not publicly exposed — read Control-Plane Metrics, Storage, Webhooks & Capacity whenever asked about control-plane VM CPU/memory or deep control-plane metrics.

Metric Source Use / Alert Threshold
apiserver_request_duration_seconds API Server (GMP) Control-plane latency (Alert: P99 > 5s)
scheduler_scheduling_attempt_duration_seconds Scheduler (GMP) Current GKE scheduling latency (scheduler_schedule_attempts_total{result="error"} > 0)
container_cpu_usage_seconds_total cAdvisor (Node) Worker-node container CPU usage
container_memory_working_set_bytes cAdvisor (Node) Worker-node container memory usage
kubernetes.io/node/cpu/core_usage_time Cloud Monitoring Node CPU usage (kubernetes.io/node/memory/used_bytes for memory)
kube_pod_status_phase kube-state-metrics Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics)
kube_pod_container_status_restarts_total kube-state-metrics Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics)
kube_node_status_condition kube-state-metrics Node readiness (condition="Ready", status!="True"; or kubernetes.io/node/status_condition)
kubelet_volume_stats_used_bytes / capacity Kubelet PVC capacity utilization (Alert: > 85%)
DCGM_FI_DEV_GPU_UTIL DCGM (GMP) GPU utilization (Alert: > 95% sustained)

Live Resource Usage & Cloud Logging Queries

Use kubectl top for live usage (no MCP or gcloud equivalent):

kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
kubectl top pods --containers -n {namespace}

Use gcloud logging read (or Cloud Logging Log Explorer) with LQL filters:

# Control-plane component logs (requires API_SERVER / SCHEDULER / CONTROLLER_MANAGER enabled)
gcloud logging read 'resource.type="k8s_control_plane_component" AND resource.labels.cluster_name="{cluster_name}"' --project {project_id} --limit 50 --quiet

# Container errors: resource.type="k8s_container" AND resource.labels.container_name="{container_name}" AND severity>=ERROR
# OOMKilled events: resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"
# FailedScheduling: resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"
# Cloud Audit Logs: resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"
# Dataplane V2 network policy / packet drop logs: logName:"policy_action" AND jsonPayload.disposition="DENY"

Distributed Tracing, Profiling & Recent Additions

  • Cloud Trace & OpenTelemetry: Deploy the OpenTelemetry Collector (googlecloud exporter with Workload Identity) or enable Managed OpenTelemetry for GKE (Preview) on GKE 1.34.1-gke.2178000+ (gcloud beta container clusters update {cluster_name} --region {region} --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS).
  • Cloud Profiler: Continuous low-overhead production CPU/memory profiling.
  • PSI (Pressure Stall Information) metrics: cAdvisor container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total via GMP ClusterNodeMonitoring (GKE 1.35+).

Supporting Links

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts15d3 checks · Risk SAFE
  • Gen Agent Trust Hub17d

    This skill provides tools and guidance for Google Kubernetes Engine (GKE) observability. It contains a standard security consideration regarding the ingestion of external data from cluster logs and events, which is common for monitoring and troubleshooting tools. See detailed analysis for context.

  • Socket15d

    No alerts

  • Snyk17d

    Risk: LOW · No issues

Signed by skilld at 5fb62c2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 weeks ago
metadata
{
  "version": "1.2.1",
  "category": "CloudObservabilityAndMonitoring"
}

README badge

README badge for google/skills/gke-observability