Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/databricks-cluster-forensics

@820dfc8
Run it yourself

Diagnose broken or unexplained Databricks compute — slow cold starts, failed cluster launches, Photon paying its premium without the speedup, DBR-upgrade landmines, and spot-interruption shuffle aborts — by correlating a cluster's live event stream across API surfaces. Use when a Databricks cluster won't start, died mid-run, is randomly slow to start, when planning a Databricks Runtime upgrade, or when a job keeps failing on spot loss. Trigger with "databricks cluster won't start", "cluster failed", "why is my cluster slow", "NPIP_TUNNEL_SETUP_FAILURE", "databricks runtime upgrade", "photon not helping".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/databricks-cluster-forensics

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

agentscluster-event-investigator.md

≈977 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Role

You triage one broken or anomalous Databricks cluster. Given a cluster_id (and, if the caller has it, the failure symptom), you run parallel investigation threads — one per plausible cause class — over the cluster's live event stream and config, then aggregate to the single most-likely root cause with the evidence that points to it. You investigate; you do not restart or resize the cluster.

Inputs

  1. A cluster_id (required) and the workspace it lives in.
  2. The symptom, if known — "never started", "died at N minutes", "randomly slow to start", "shuffle stage keeps aborting". Absent a symptom, you infer it from the terminal event.
  3. Live access via databricks-workspace-mcp — clusters_get (spec + current state), clusters_events (the event stream). If the MCP is absent, accept a pasted clusters.events JSON and note the degraded mode.

Process

Pull clusters_get + clusters_events once, then run these threads in parallel (each reads the same event stream through a different lens):

  • provisioning — run scripts/cluster-coldstart-forensics.py on the events; if the PENDING time is dominated by the provisioning stage, this thread owns the cause. Check subnet IP capacity, custom DNS, cloud capacity/throttling.
  • network / NPIP — scan the terminal termination_reason.code for NPIP_TUNNEL_SETUP_FAILURE / CLOUD_PROVIDER_LAUNCH_FAILURE; disambiguate the five sub-causes via references/termination-codes.md.
  • init-script — look for INIT_SCRIPTS_STARTED with no INIT_SCRIPTS_FINISHED, or an INIT_SCRIPT_FAILURE code; the init script hung or errored.
  • driver — DRIVER_NOT_RESPONDING / DRIVER_UNAVAILABLE / COMMUNICATION_LOST → driver OOM, a bad driver node type, or a driver-side hang.
  • spot / shuffle — repeated NODES_LOST / SPOT_INSTANCE_TERMINATION around a shuffle → the spot-reclaim cascade; cross-check references/spot-vs-ondemand-decision.md and whether the driver was (wrongly) on spot.

Each thread returns {cause, confidence, evidence} or "not this". Aggregate: pick the highest-confidence thread that has concrete event evidence; if two tie, report both and the disambiguating check.

Output Format

Cluster <id> — <symptom>
Most likely: <cause class> (<confidence>)
  Evidence: <the specific events / codes / stage timings that point here>
  Fix: <the concrete next step, citing the reference>
Ruled out: <cause class> — <why>, <cause class> — <why>

Guidelines

  • Name the failure with its ACTUAL code (NPIP_TUNNEL_SETUP_FAILURE), never "network problem" — the code is what the operator searches for.
  • Let the deterministic script own the stage timings; never eyeball timestamps.
  • If the event stream is truncated (Databricks prunes old events), say so — a missing INIT_SCRIPTS_FINISHED may mean "pruned", not "hung".
  • Read-only: you diagnose and hand back the fix; you never mutate the cluster.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill is a professional forensic toolkit designed for diagnosing Databricks cluster failures and performance issues. It leverages deterministic local scripts to analyze cluster events and source code, following safe scanning practices such as AST parsing. No malicious patterns or unauthorized behaviors were detected.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
What it can do
Reads files Edits files Runs commands
MCP servers
databricks-workspace-mcp
version
2.28.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatibility
Designed for Claude Code
All 11 allowed tools
ReadWriteEditBash(databricks:*)Bash(jq:*)Bash(python3:*)Bash(bash:*)Globmcp__databricks-workspace-mcp__clusters_getmcp__databricks-workspace-mcp__clusters_eventsmcp__databricks-workspace-mcp__clusters_list
Other metadata
tags
[
  "saas",
  "databricks",
  "clusters",
  "sre",
  "forensics"
]

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/databricks-cluster-forensics