skilld
All skills
aws avatar

/managing-amazon-msk

@f0f6844
by Amazon Web Servicesaws/agent-toolkit-for-aws2.8k stars
337

Operates Amazon MSK Provisioned clusters (Standard and Express brokers). Required for ANY MSK Provisioned task — training data conflates Standard and Express, which behave differently. Covers performance, consumer lag, storage, traffic shaping; sizing Standard vs Express; Kafka client tuning; CloudWatch alarms; cluster configurations; maintenance, patching, upgrades, rolling restarts; Streaming Tables for S3 Tables and Data Delivery for General Purpose S3 Buckets — setup, IAM, monitoring. Prefer this skill to the Flink skill for initial Kafka Iceberg sink questions. Triggers: MSK Provisioned (Express/Standard), Kafka, `kafka.*` or `express.*` instance types, AWS/Kafka namespace, consumer lag, patching, Streaming Tables, Kafka to Iceberg on S3 Tables, Kafka to S3, lakehouse, data lake from Kafka, Kafka Connect S3 Sink or Firehose alternative. DO NOT USE for MSK Connect or Replicator — search documentation instead. Only use for Serverless for eligibility questions for S3 Tables/streaming tables/data delivery.

Ask your Agent

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/managing-amazon-msk

This session only. Nothing lands on disk.

referencesstreaming-tables-troubleshooting.md

≈1.4k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Streaming Tables — Troubleshooting

Channel States

State Meaning
CREATING Provisioning in progress
ACTIVE Delivering data
UPDATING Config change in progress (still delivering)
DELETING Being removed
FAILED Creation failed — check DescribeChannel + CW Logs

Channel Creation Failures

Channel stuck in CREATING

Provisioning is in progress. Run aws kafka describe-channel to monitor. If it transitions to FAILED, check the failure detail. If it stays in CREATING well beyond expected time, contact Support.

Configuration errors (missing permissions, nonexistent destination, unresolvable schema) cause FAILED, not prolonged CREATING.

Channel transitions to FAILED

Cause Fix
Trust policy missing kafka.amazonaws.com Fix trust policy principal and conditions
Destination bucket doesn't exist Create the S3 Table bucket in the same Region as the cluster. The Iceberg table is auto-created by Streaming Tables (enableTableCreation: true) — you must not pre-create the Iceberg table; pre-creating it will break Streaming Tables delivery. Only the bucket must exist.
Schema can't be resolved from GSR Verify: aws glue get-schema --schema-id SchemaArn=<ARN>
Missing IAM permissions Compare role policy against reference in streaming-tables.md or data-delivery-for-general-purpose-s3.md
Wrong catalog ARN Must be arn:aws:glue:REGION:ACCOUNT_ID:catalog/s3tablescatalog/BUCKET

Recovery: Delete failed channel, fix config, recreate.

Delivery Issues

Prerequisite: Ensure CloudWatch Logs delivery is enabled for the Streaming Tables. Without it, diagnostic logs (e.g., AccessDenied errors) are not available. Enable logging when creating or updating the channel.

No data at destination

  1. Check BytesIn metric — if zero, no data being produced
  2. Check CloudWatch Logs for AccessDenied
  3. Verify producers are actively writing
  4. No backfill — only data produced AFTER enablement is delivered

Data freshness exceeds configured interval

Causes: Large table metadata/snapshots, low throughput, transient issues.

Fix:

  • Enable S3 Tables maintenance (compaction, snapshot expiration)
  • Compare the topic's sustained throughput against the minimum-throughput floor for the configured freshness in Amazon MSK Data Delivery quotas. If the topic is below the floor, raise dataFreshnessSeconds

Delivery stops after schema change (Iceberg)

Schema evolution is NOT supported. A change makes records incompatible.

Fix: Revert schema OR delete channel and create new one.

Record Failures

FailedRowCount / FailedRecordCount > 0

Records don't conform to schema. Common causes:

  • Missing required field
  • Type mismatch
  • Malformed JSON
  • Nesting > 16 levels

Fix: Inspect DLQ messages + CloudWatch Logs for error details. Fix producer or schema.

DLQ Error Patterns

Warning: DLQ messages contain full record payloads which may include sensitive data (PII, financial data, etc.). Ensure the DLQ bucket has encryption enabled (SSE-KMS recommended), access is restricted to authorized personnel, and consider enabling S3 access logging for audit purposes.

Error Cause Fix
Missing required field Record lacks field in schema "required" Add to producer or remove from "required" (needs new channel)
Type mismatch Value doesn't match schema type Fix producer serialization
Schema resolution failure Embedded schema ID doesn't match GSR (JSON_SCHEMA_GSR) Verify producer GSR config

Permission Issues

AccessDenied in CloudWatch Logs

  1. Compare service role policy against reference policies
  2. Check CloudTrail for recent policy/bucket-policy/key-policy changes
  3. Verify trust policy has kafka.amazonaws.com (do not use any other service principal)
  4. For KMS: verify key policy grants role access
  5. For cross-account: verify table bucket policy in target account

Monitoring

<!-- TODO: Add dedicated monitoring reference doc (streaming-tables-monitoring.md) with detailed alarm setup, dashboard templates, and operational runbooks once GA metrics are finalized. -->

Namespace: AWS/Kafka | Dimensions: ClusterName, ChannelName, Topic

Iceberg metrics

DeliveryToIceberg.DataFreshness, BytesIn, BytesProcessed, BytesOut, TotalRowCount, SuccessfulRowCount, FailedRowCount, CommitSuccess, DLQDeliverySuccess

S3 metrics

DeliveryToS3.DataFreshness, BytesIn, BytesProcessed, BytesOut, RecordCount, SuccessfulRecordCount, FailedRecordCount, DeliverySuccess, DLQDeliverySuccess

Recommended alarms

Alarm Condition Action
High data freshness DataFreshness > configured interval Investigate throughput
Failed records FailedRowCount/FailedRecordCount > 0 for 5 min Check DLQ + schema compatibility
DLQ deliveries DLQDeliverySuccess > 0 Inspect DLQ messages

Channel Not Available

Symptom: Streaming Tables tab not visible or CreateChannel errors.

Cause: Cluster uses Standard brokers, or is MSK Serverless.

Fix: Streaming Tables requires MSK Provisioned with Express brokers only.

Source: SKILL.md on GitHub

Third-party checks

No alerts6d3 checks · Risk SAFE
  • Gen Agent Trust Hub6d

    This skill for managing Amazon MSK Provisioned clusters is well-documented and follows security best practices. It includes a local sizing utility and detailed operational guides for MSK Standard and Express brokers. No security issues or malicious patterns were detected.

  • Socket6d

    No alerts

  • Snyk6d

    Risk: LOW · No issues

Provenance

Signed by skilld at f0f6844. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last week

Capability

version
6

README badge

README badge for aws/agent-toolkit-for-aws/managing-amazon-msk