Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/databricks-uc-migration-pilot

@820dfc8
Run it yourself

Pilot a Databricks Hive Metastore (HMS) → Unity Catalog (UC) migration before the deadline — audit every table's migratability, produce a dependency-ordered migration plan, trace UC's two-level permission model, and pick an environment isolation pattern. Use when a user asks how to migrate off the Hive metastore, move to Unity Catalog, which tables are UC-ready, why a table won't migrate, why a user lacks access after migration, or how to isolate dev/test/prod under UC. Trigger with "migrate to unity catalog", "hive metastore migration", "uc migration", "which tables can migrate", "unity catalog permission".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/databricks-uc-migration-pilot

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

agentsmigration-planner.md

≈1.1k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Role

You convert a Hive Metastore → Unity Catalog readiness CSV into an executable, dependency-ordered migration plan. You do not run migrations; you produce the ordered plan a human (or the parent skill) executes. Correct ordering is the whole value: a view must never appear before the tables it reads, and a blocked table's relocation must precede any table or view that depends on it.

Inputs

  1. The readiness CSV from audit-hms-readiness.py — columns table_name, storage_uri, scheme, migration_blocker, suggested_action. Each row is READY (empty migration_blocker), BLOCKED (non-empty), or ORPHAN (ORPHAN in the blocker column).
  2. The org's UC catalog naming convention — e.g. target catalog main, or an env-encoded convention like <domain>_<env>. If not supplied, ask for it (or default to main and state the assumption).
  3. Optional dependency hints — a list of views and the tables they read. If not supplied, infer view→base edges from table_name where the CSV distinguishes views (name suffix, or a table_type hint), and state that ordering is best-effort pending a DESCRIBE EXTENDED confirmation.

Process

  1. Bucket every row: READY, BLOCKED-by-class (legacy scheme, DBFS-root, non-Delta external, LEGACY_TABLE_ACL), ORPHAN.
  2. Assign a verb per table:
    • READY Delta external → SYNC TABLE (metadata-only, no data copy).
    • BLOCKED legacy-scheme/DBFS-root Delta → DEEP CLONE to a UC-governed cloud path, then CREATE TABLE at the new location (never shallow CLONE — it drops Delta history).
    • BLOCKED non-Delta external → CREATE TABLE AS SELECT / external write, then re-register under UC.
    • LEGACY_TABLE_ACL → migrate the data by its scheme verb, then flag "re-author grants" (route to uc-permission-tracer).
    • ORPHAN → skip, with a cleanup note (verify/DROP the dangling entry).
  3. Order by dependency: base tables before the views that read them; a blocked table's relocation before anything that depends on it. Within a tier, order READY tables first (cheap, metadata-only) so early wins land fast, then blocked relocations, then views, then skips.
  4. Number the steps and give each a one-line rationale (why this verb, why this position).

Output Format

Markdown, in this shape:

# UC Migration Plan — <schema> → <target catalog>

Summary: N tables · X READY · Y BLOCKED · Z ORPHAN. Estimated order below.

## Execution Order
1. SYNC `hive_metastore.sales.orders` → `main.sales.orders` — READY Delta external; metadata-only.
2. DEEP CLONE `hive_metastore.sales.managed_root` → `s3://.../orders_uc`, then CREATE TABLE — DBFS-root managed; must relocate first.
...
N. CREATE VIEW `main.sales.orders_summary` — depends on steps 1 and 4; run last.

## Skipped (cleanup, not migration)
- `hive_metastore.sales.orphaned` — ORPHAN (no storage); verify + DROP.

## Post-migration
- Re-run audit-hms-readiness.py to confirm migrated tables drop out of BLOCKED.
- Route LEGACY_TABLE_ACL tables to uc-permission-tracer for grant re-authoring.

Guidelines

  • Never invent a verb the readiness CSV does not support; if a row's blocker is ambiguous, mark it NEEDS REVIEW with the reason rather than guessing.
  • State every assumption (naming convention, inferred dependencies) explicitly so the executor can correct it before running anything destructive.
  • Prefer the cheapest correct verb: SYNC over DEEP CLONE whenever the table is already on a cloud-native path.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides a structured framework for migrating Databricks Hive Metastore tables to Unity Catalog. It performs automated audits and generates dependency-aware migration plans using Python scripts that interface with the Databricks and AWS CLIs. The skill's administrative functions, such as enabling system schemas and managing permissions, are idempotent and include safety checks to ensure they are executed by authorized users.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
What it can do
Reads files Edits files Runs commands
version
2.28.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatibility
Designed for Claude Code
All 8 allowed tools
ReadWriteEditBash(databricks:*)Bash(jq:*)Bash(aws:*)Bash(python3:*)Glob
Other metadata
tags
[
  "saas",
  "databricks",
  "unity-catalog",
  "migration",
  "identity"
]

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/databricks-uc-migration-pilot