Index of every page on this site: llms.txt
All skills
google avatar

/gke-ai-troubleshooting-handle-disruption-gpu-tpu

@5fb62c2
by googlegoogle/skills24k stars
2,026
Run it yourself

Diagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.

Use this Skill: https://skilld.dev/gh/google/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈153 tokens for metadata: the name and description. ≈1.6k when used: this file. ≈496 more on demand in 1 file.

Description uses 7.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈613 characters, or 7.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Handle Disruption on GPUs and TPUs Troubleshooting

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu) to User-Agent headers.

🔍 Diagnostic Workflow

Step 0: Context Acquisition

  • Mandatory: When a user asks to debug or investigate an actual workload disruption, node crash, or unexpected restart without providing complete cluster details, you MUST immediately halt and request all missing mandatory parameters (project_id, location, cluster_name, timestamp) BEFORE delivering theories or general diagnostic commands. Only skip context acquisition if the user explicitly requests a generic reusable runbook or provides a complete static telemetry/log dump for offline analysis.
  • Optional: node_name, workload_name, workload_namespace, nodepool_name.

Step 1: [Low Risk] Check for Upcoming Scheduled Maintenance

  • Action: Propose running kubectl to check if nodes have the scheduled maintenance label indicating an upcoming disruption.

  • Example Command:

    kubectl get nodes -l cloud.google.com/scheduled-maintenance-time -L cloud.google.com/scheduled-maintenance-time
  • Interpretation: The SCHEDULED-MAINTENANCE-TIME column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur.

Step 2: [Low Risk] Investigation via Cloud Monitoring (PromQL)

  • Action: Call any available monitoring tool or provide PromQL for manual verification.

  • Mandatory Monitoring Rule: Whenever recommending follow-up monitoring or interruption tracking over time, you MUST explicitly present a PromQL query using the metric kubernetes_io:node_interruption_count filtered by interruption_reason="HW/SW Maintenance". Do not suggest general Cloud Monitoring dashboards or Metrics Explorer without providing this specific PromQL metric expression.

  • Example Query:

    # Fetch host maintenance events for nodes
    sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[${__interval}]))
    # See the interruption count aggregated by node pool
    sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{nodepool_name}" }[${__interval}]))
  • Interpretation: If kubernetes_io:node_interruption_count shows values > 0 for interruption_reason="HW/SW Maintenance", it indicates the underlying Compute Engine VM was interrupted due to scheduled host maintenance.

Step 3: [Low Risk] Investigation via Cloud Logging & Node Taints

  • Action: Call query_logs or instruct the user to filter their GKE logs for active host maintenance events, and check node taints.
  • Guidance: Look for occurrences in Cloud Logging where cloud.google.com/active-node-maintenance is set to ONGOING. To check if GKE has cordoned the terminating node to prevent new workloads from being scheduled, verify whether the cloud.google.com/impending-node-termination:NoSchedule taint is present (either in GKE event logs or directly via kubectl describe node).
  • Interpretation:
    • cloud.google.com/active-node-maintenance set to ONGOING means workloads are actively being stopped by GKE due to host maintenance.
    • cloud.google.com/impending-node-termination:NoSchedule taint means GKE has cordoned the node to prevent new Pods from being scheduled on the terminating node. DO NOT recommend tolerating this taint.

Step 4: Conclusion and Resolution

  • Action: Provide a summary of findings to the user and suggest appropriate mitigation strategies if host maintenance events were confirmed or scheduled.
  • Reporting Rule: Signal Only. Report high-signal information indicating that the disruption was caused by Compute Engine host maintenance, specifically affecting the underlying GPU/TPU nodes. DO NOT dump raw logs.
  • Negative Findings Rule-Out: If node scheduled-maintenance labels, PromQL interruption counts, and active maintenance logs all return negative/empty results, definitively conclude that Compute Engine host maintenance did NOT cause the disruption. Direct the user to investigate application-level causes (such as OOMKill events, CUDA runtime errors, or resource limits) and do not propose host maintenance mitigations as the primary resolution.
  • Mandatory Workload Protection Triad: Whenever host maintenance is identified or anticipated on GPU/TPU nodes, consistently recommend all three complementary mitigations together:
    1. Configure Graceful Termination: For workloads that need time to save state (e.g., ML frameworks checkpointing via Orbax), follow the guide to Enable disruption handling and set spec.terminationGracePeriodSeconds (up to 60 minutes) to handle the SIGTERM signal before node shutdown.
    2. Enable Opportunistic Maintenance: To automatically trigger maintenance when GKE detects that GPU/TPU nodes are idle, configure Opportunistic Maintenance.
    3. Configure PodDisruptionBudgets (PDBs): Ensure your workload uses a PodDisruptionBudget to maintain minAvailable replicas during evictions and disruptions.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts1mo3 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    This skill is designed for diagnosing and mitigating node disruptions in GKE GPU/TPU environments. It provides structured troubleshooting steps using standard Google Cloud observability tools and best practices. No security issues were identified.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

Signed by skilld at 5fb62c2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last month
metadata
{
  "version": "1.0.1",
  "category": "CloudObservabilityAndMonitoring"
}

README badge

README badge for google/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu