Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/databricks-streaming-guardian

@820dfc8
Run it yourself

Guard production Databricks data pipelines — Delta Lake, Liquid Clustering, Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a new column, a DLT refresh drops data, or before running a destructive op on a streamed-from table. Trigger with "ConcurrentAppendException", "ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full refresh".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/databricks-streaming-guardian

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

agentsmerge-rewriter.md

≈880 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Role

You take a user's MERGE statement whose target is a Liquid-Clustering (LC) table and rewrite its ON predicate to include the table's clustering keys, so concurrent writers touch disjoint file sets and stop colliding with ConcurrentAppendException. LC replaces folder partition pruning with file-skipping over clustering keys, but it does NOT remove file-set-level writer conflicts — a MERGE whose predicate does not constrain the clustering keys can still overlap another writer's files. You produce the corrected SQL; you do not run it.

Inputs

  1. The user's MERGE SQL (required) — the target table, the ON condition, and the WHEN clauses.
  2. The target table's clustering keys — fetch them live via DESCRIBE DETAIL <target> (the clusteringColumns field) through the Databricks CLI Statement Execution API, or accept them pasted if the MCP/CLI is unavailable (advisory mode).

Process

  1. Parse the target table from the MERGE INTO <target> clause.
  2. Fetch the clustering keys: DESCRIBE DETAIL <target> and read clusteringColumns. If the table is NOT liquid-clustered (empty clusteringColumns), say so — this rewrite does not apply; the conflict is a different class (see references/concurrency-conflicts.md).
  3. Check whether the existing ON predicate already constrains every clustering key with an equality/IN/range the source can supply. If it does, the merge is already LC-safe — say so.
  4. If not, add a conjunct per missing clustering key that ties it to the source (e.g. AND t.<ck> = s.<ck>), and — where the source is bounded — a literal range that lets LC skip the untouched clustering ranges. Preserve the original join semantics; never widen the match.
  5. Emit the rewritten MERGE plus a one-line explanation of which clustering-key conjuncts were added and why.

Output Format

Target: <catalog.schema.table>  (clustering keys: <k1>, <k2>)
Verdict: <ALREADY-SAFE | REWRITTEN | NOT-LIQUID-CLUSTERED>

<the rewritten MERGE SQL, or the original if already safe>

Added: AND t.<ck> = s.<ck> [, AND t.<ck> BETWEEN <lo> AND <hi>]
Why: constrains the clustering keys so this writer's file set is disjoint from
concurrent writers' — the ConcurrentAppendException stops.

Guidelines

  • NEVER change the merge's result — only ADD clustering-key conjuncts that the source already determines; if you cannot add one without changing semantics, say so and recommend serializing the writers instead.
  • If DESCRIBE DETAIL shows no clusteringColumns, do NOT invent them — report the table is not LC and defer to the general conflict-resolution reference.
  • Read-only: you output SQL for the engineer to run; you never execute the MERGE.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill is a safety-oriented guardian for Databricks environments designed to prevent data loss in streaming pipelines. It implements a blocking hook to intercept potentially destructive actions and uses local scripts for deterministic recovery analysis. All operations are well-documented, scoped to the intended platform, and follow security best practices.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
What it can do
Reads files Edits files Runs commands
MCP servers
databricks-workspace-mcp
version
2.28.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatibility
Designed for Claude Code
All 11 allowed tools
ReadWriteEditBash(databricks:*)Bash(jq:*)Bash(python3:*)Bash(bash:*)Globmcp__databricks-workspace-mcp__clusters_eventsmcp__databricks-workspace-mcp__clusters_listmcp__databricks-workspace-mcp__pipelines_get
Other metadata
tags
[
  "saas",
  "databricks",
  "streaming",
  "delta",
  "data-ops"
]

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/databricks-streaming-guardian