All skills
aws avatar

/migrating-to-amazon-redshift

@5c779cf

Guides an end-to-end data-warehouse migration to Amazon Redshift — discovery, schema/SQL/stored-procedure/macro/script conversion, data migration, validation, performance comparison, and reporting. Source-routed via `references/<source>/`; Teradata (Vantage) is the supported source; additional sources are added as their own `references/<source>/` sets. Text-only knowledge (no executable code) — the AI generates all execution at runtime. Applies when a user wants to migrate Teradata to Amazon Redshift, convert Teradata DDL/SQL/stored procedures/macros/BTEQ to Redshift/RSQL, or assess Teradata-to-Redshift migration complexity. Applies only to migrations targeting Amazon Redshift; migrations to other platforms (Snowflake, BigQuery, Databricks, etc.) are out of scope regardless of source. Does not cover general Redshift administration, performance tuning, or troubleshooting of existing Redshift clusters (no migration involved), or sources not listed under references/.

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/migrating-to-amazon-redshift

This session only. Nothing lands on disk.

referencesteradatacommon-errors.md

≈947 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Common Errors, Edge Cases, and Resolutions

Redshift-specific errors

  • "number of segments exceeds the max allowed (30)" — complex joins (e.g. MicroStrategy). Remove the full outer join VLDB property; simplify joins.
  • "correlated subquery pattern is not supported" — flatten nested correlated subqueries; rewrite as CTEs or JOINs.
  • Reserved-word collision — double-quote aliases (returns, interval, language, position, result, trim, user, type, default, comment, value, year, month, day, zone, …).
  • SUBSTR not recognized — #1 failure cause; replace every SUBSTR( with SUBSTRING(.
  • Window function missing frame — add explicit frame: SUM(x) OVER (ORDER BY d ROWS UNBOUNDED PRECEDING).
  • Derived column order — Redshift requires a derived column be defined before it's referenced in the same SELECT.
-- Teradata (forward reference allowed)
SELECT pricepaid,
  CASE WHEN discount > 10 THEN 'Yes' ELSE 'No' END discount_flag,
  pricepaid - retailprice AS discount
FROM sales;

-- Redshift (define before use)
SELECT pricepaid,
  pricepaid - retailprice AS discount,
  CASE WHEN discount > 10 THEN 'Yes' ELSE 'No' END discount_flag
FROM sales;

Conversion edge cases (flag for manual review)

  • QUALIFY — Redshift supports QUALIFY natively; keep as-is (when QUALIFY directly follows FROM, the FROM relation must have an alias — add one). Reshape to a CTE with ROW_NUMBER() and filter in WHERE only when mixed with GROUP BY (dedupe-then-aggregate) — see conversion-rules.md.
  • RESET WHEN (HIGH) — CTE with LAG/SUM grouping.
  • EXPAND ON (HIGH) — generate_series() or recursive CTE.
  • TD_NORMALIZE_OVERLAP (HIGH) — CTE with LAG/SUM grouping.
  • TD_UNPIVOT (HIGH) — CROSS JOIN with CASE expressions.
  • NONSEQUENCED temporal (HIGH) — full rewrite required.
  • ALTER TABLE MODIFY (HIGH) — not supported; DROP+ADD column or recreate table.
  • Cursors (HIGH) — prefer set-based rewrites.
  • Triggers (HIGH) — not supported; rewrite as Lambda/Step Functions/app logic.

QUALIFY example

-- Plain QUALIFY: keep as-is — native in Redshift. One restriction: when QUALIFY directly
-- follows FROM, the FROM relation must carry an alias (here: o)
SELECT * FROM orders o
QUALIFY ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date DESC) = 1;

-- QUALIFY mixed with GROUP BY (dedupe-then-aggregate): reshape to a CTE
WITH ranked AS (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date DESC) AS rn
  FROM orders
)
SELECT * FROM ranked WHERE rn = 1;

Comparison operators

-- Integer comparison: don't quote integers
WHERE integer_sk > -13            -- (TD allowed '-13')

-- LIKE ANY/ALL → split into OR clauses
WHERE (col LIKE 'val1') OR (col LIKE 'val2') OR (col LIKE 'val3')

-- Timezone conversion
CONVERT_TIMEZONE('US/Eastern', col)   -- source must be TIMESTAMP w/o TZ

Data migration challenges

  • Large volumes (100TB+) — TPT parallel unload; Direct Connect or Snowball by bandwidth; Vantage unloads directly to S3.
  • Storage footprint — Redshift uses 1MB blocks; small tables may grow (expected).
  • Character encoding — TD often ISO-8859-1, RS default UTF-8; size VARCHARs for multibyte; set COPY encoding (UTF8/UTF16/…/ISO88591).

SCT considerations

  • Handled: secondary indexes → sort keys; join indexes → materialized views; PK/FK as optimizer hints (not enforced).
  • Not handled: range partitioning (manual timeseries setup), SET-table uniqueness (needs dedup logic), TITLE keyword, FALLBACK/JOURNAL (removed).
  • Run the SCT assessment report before automated conversion to gauge complexity.

Source: SKILL.md on GitHub

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides a structured methodology for migrating data from Teradata to Amazon Redshift, utilizing AI to generate environment-specific scripts at runtime. It incorporates strong security considerations such as identity-based access (IAM), encryption at rest and in transit, and secret management using native cloud services. While the skill utilizes dynamic script generation and external package installation, these are implemented with specific safety instructions like input validation and the use of trusted libraries.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 5c779cf. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
version
1

README badge

README badge for aws/agent-toolkit-for-aws/migrating-to-amazon-redshift