skilld
All skills
aws avatar

/exploring-data-catalog

@b33847d
by Amazon Web Servicesaws/agent-toolkit-for-aws2.8k stars
337

Full inventory and audit of AWS Glue Data Catalog assets across S3 Tables, Redshift-federated, and remote Iceberg catalogs. Triggers on: inventory the catalog, audit databases, list all tables, catalog overview, data landscape, enumerate catalogs, data inventory, search the catalog. Do NOT use for finding specific data (use finding-data-lake-assets), running queries (use querying-data-lake), or creating tables (use creating-data-lake-table).

Ask your Agent

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/exploring-data-catalog

This session only. Nothing lands on disk.

referencesdiscovery-checklist.md

≈733 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Discovery Checklist

Output Structure

Present findings in this order:

  1. Catalog Landscape: catalog count by type (Glue, S3 Tables, Redshift-federated, Remote Iceberg), connection status for federated catalogs
  2. Executive Summary: total databases, total tables, primary formats, estimated volume
  3. Database Inventory: organized by catalog and database with table counts
  4. Unregistered Assets: S3 Tables not in Glue (not queryable via Athena), with registration instructions
  5. Schema Analysis: data types, nullable fields, key patterns
  6. Storage Analysis: formats, partitioning strategies, S3 locations
  7. Recommendations: optimization opportunities, quality issues, missing metadata, unregistered tables to register

Column Classification

Categorize each column as one of:

  • Identifier: Unique keys, foreign keys, entity IDs
  • Dimension: Categorical attributes for grouping/filtering (status, type, region)
  • Metric: Quantitative values for measurement (revenue, count, duration)
  • Temporal: Dates and timestamps (created_at, updated_at, event_date)
  • Text: Free-form text fields (description, notes)
  • Boolean: True/false flags
  • Structural: JSON, arrays, nested structures (common in Glue tables from JSON sources)

Quality Scoring

Rate each column's completeness:

  • Complete (>99% non-null): reliable for analysis
  • Mostly complete (95-99%): investigate the nulls before using in calculations
  • Incomplete (80-95%): understand why, may need imputation or filtering
  • Sparse (<80%): likely not usable without significant cleanup

Column Profiling (when deep-diving a table)

For numeric columns: min, max, mean, median, p5, p95, zero count, negative count For string columns: min/max length, empty string count, distinct values, pattern consistency For date columns: min/max date, null dates, future dates (if unexpected), gap detection For boolean columns: true/false/null distribution

What to Flag

  • Tables with no partition keys on datasets > 1GB
  • CSV tables that should be Parquet (cost and performance)
  • Databases or tables with no descriptions
  • Tables with no recent data (stale/abandoned)
  • Inconsistent naming conventions across databases
  • Tables with high null percentages in key columns
  • Columns that appear to be foreign keys (potential join targets)
  • Hierarchical dimensions (country > state > city)
  • Columns with suspiciously low cardinality (possible default values)
  • S3 Tables not registered in Glue (exist but not queryable via Athena)
  • Federated catalogs with connection errors or stale metadata

Format Detection

Map SerDe libraries to human-readable format names:

  • org.apache.hadoop.hive.ql.io.parquet = Parquet
  • org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe = CSV/TSV
  • org.openx.data.jsonserde.JsonSerDe = JSON
  • org.apache.hadoop.hive.serde2.OpenCSVSerde = CSV
  • org.apache.hadoop.hive.ql.io.orc = ORC

Source: SKILL.md on GitHub

Third-party checks

1 warning16d3 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    This skill includes security considerations regarding the ingestion of data catalog metadata, which could serve as a potential surface for indirect prompt injection. These considerations are well-addressed by explicit safety instructions within the skill architecture.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: MEDIUM · 1 issue

Provenance

Signed by skilld at b33847d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago

Capability

Other metadata
metadata
{
  "version": "2",
  "argument-hint": "'[search-term|catalog-name|database-name|s3://bucket-path|table-name]'"
}

README badge

README badge for aws/agent-toolkit-for-aws/exploring-data-catalog