Index of every page on this site: llms.txt
All skills
datadog-labs avatar

/dd-aws-storage-management

@618fae3 official
Run it yourself

Set up Datadog Storage Management for AWS S3 by configuring S3 Inventory on the buckets the user picks and provisioning the destination buckets, destination bucket policy, Datadog read policy, and Storage Management sync with the DataDog/storage-management-datadog/aws Terraform module, including multi-Region and SSE-KMS destinations and optional S3 access logs. Use when the user wants to set up Storage Management for their S3 buckets, monitor their S3 storage in Datadog, or enable S3 storage management. Does not connect a new AWS account to Datadog and does not deploy the Datadog Forwarder Lambda (that is `dd-aws-log-forwarding`). Commercial AWS partition only; there is no supported setup for AWS GovCloud or China. Requires an existing role-based Datadog AWS integration; run `dd-aws-integration` first if it is missing.

Use this Skill: https://skilld.dev/gh/datadog-labs/agent-skills/dd-aws-storage-management

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesterraform.md

≈4.2k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Storage Management for AWS Terraform pieces

The HCL that Phase 3 of SKILL.md assembles. Choose the pieces with the "Assemble per scenario" table and the "Multiple Regions" rules in SKILL.md, and substitute the values collected in Phases 1 and 2. Section names in quotes refer to sections of SKILL.md, except "SSE-KMS destination buckets", which is below.

Placeholder Source
<DD_SITE> Phase 0 (DD_SITE)
<AWS_ACCOUNT_ID> Phase 1 (aws_account_id)
<DATADOG_INTEGRATION_ROLE_NAME> Phase 1 (auth_config.role_name)
<DD_ORG_ID> Phase 1, "Fetch your Datadog org id"
<SOURCE_BUCKET_NAME>, <DESTINATION_BUCKET_REGION> Phase 2 §1-§2
<DESTINATION_BUCKET_NAME>, <KMS_KEY_ARN> Phase 2 §3
<DATADOG_FORWARDER_ARN>, <ACCESS_LOG_BUCKET_NAME> Phase 2 §5

Providers

Providers (datadog is required because the module registers the sync as a Terraform resource):

terraform {
  required_providers {
    aws     = { source = "hashicorp/aws", version = ">= 5.0" }
    datadog = { source = "DataDog/datadog", version = ">= 3.85" }
  }
}

locals {
  # The DD_SITE env var value (e.g. datadoghq.com, datadoghq.eu, us3.datadoghq.com, ap1.datadoghq.com, ddog-gov.com).
  datadog_site = "<DD_SITE>"
}

# Single Region: one aws provider. Multi-Region: one aliased aws provider per Region (see "Multiple Regions").
provider "aws" {
  region              = "<DESTINATION_BUCKET_REGION>"
  allowed_account_ids = ["<AWS_ACCOUNT_ID>"]
}

# api_key / app_key come from DD_API_KEY / DD_APP_KEY. api_url MUST be derived from DD_SITE: the Datadog provider
# reads the site from DD_HOST / api_url (NOT DD_SITE) and otherwise defaults to US1 (https://api.datadoghq.com),
# which targets the wrong site and fails credential validation for EU / US3 / US5 / AP / Gov tenants.
provider "datadog" {
  api_url = "https://api.${local.datadog_site}"
}

Destination bucket

Destination bucket - only when creating a new one. Set SSE-S3 explicitly so the bucket never silently inherits a KMS account-default (which would break inventory delivery - see "SSE-KMS destination buckets"):

resource "aws_s3_bucket" "inventory_destination" {
  bucket = "<DESTINATION_BUCKET_NAME>"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "inventory_destination" {
  bucket = aws_s3_bucket.inventory_destination.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

The module

The module (one instance per Region):

module "storage_management" {
  source  = "DataDog/storage-management-datadog/aws"
  version = "~> 1.0"

  name                                 = "dd-sm-<DD_ORG_ID>"   # org-scoped (Phase 1) so the S3 Inventory config id + module-managed resource names don't collide when one AWS account is connected to multiple Datadog orgs; also suffix the Region when multi-Region
  datadog_aws_integration_role_name    = "<DATADOG_INTEGRATION_ROLE_NAME>"
  source_bucket_names                  = ["<SOURCE_BUCKET_NAME>"]  # the source buckets in THIS Region
  destination_bucket_name              = "<DESTINATION_BUCKET_NAME>"
  destination_prefix                   = "datadog-inventories/"
  destination_bucket_policy_management = "create"       # "create" for a new/empty-policy bucket; "none" for an existing bucket that already has a policy (see "Existing destination bucket policy")

  # New-bucket case only: expire inventory reports after 2 days. Omit it for an existing bucket, because the module's lifecycle resource replaces the bucket's whole lifecycle configuration.
  manage_destination_lifecycle         = true

  # Access logs - include these lines only if the user opted in AND provided a Forwarder ARN in Phase 2 §5.
  # If they declined or have no Forwarder yet, leave these commented out entirely.
  # When enabled, also emit the dedicated access-log bucket + grant (aws_s3_bucket{,_policy}) shown below the module.
  # enable_access_logging  = true
  # access_log_prefix      = "s3-access-logs/"
  # datadog_forwarder_arn  = "<DATADOG_FORWARDER_ARN>"
  # access_log_bucket_name = "<ACCESS_LOG_BUCKET_NAME>"  # required when enable_access_logging = true (the module has no default). Must be a same-Region SSE-S3 (AES256) bucket with NO existing notifications - the module manages this bucket's notification config authoritatively (overwrites existing), and S3 won't deliver logs to an SSE-KMS bucket. A fresh dedicated bucket is safest (Phase 2 §5).

  # New-bucket case only - ensure the bucket + its encryption exist first.
  depends_on = [aws_s3_bucket.inventory_destination, aws_s3_bucket_server_side_encryption_configuration.inventory_destination]
}

<DATADOG_INTEGRATION_ROLE_NAME> is the role_name from the account selected in Phase 1 (typically DatadogIntegrationRole). The module configures the inventory (DatadogStorageManagement name, CSV, daily, included_object_versions = All), the destination bucket policy, the Datadog read IAM policy, and the sync - no separate API call. For an existing destination bucket, drop the aws_s3_bucket / aws_s3_bucket_server_side_encryption_configuration / depends_on / manage_destination_lifecycle, set destination_bucket_name to the literal name, and choose none vs create per Phase 2 §3.

Existing destination bucket policy

Only for an existing destination bucket that already has a policy, or whose policy a resource in the project already manages. Do not use the module's "merge" mode: it reads the live policy on every plan and appends its statement again, so after the first apply each plan adds a duplicate S3InventoryWrite-<name> statement, and grants for removed source buckets are never dropped. Set the module's destination_bucket_policy_management = "none" and manage the merged policy here instead.

First check whether Terraform already manages this bucket's policy (Phase 3 of SKILL.md, "Check for Existing Terraform", runs the search). The AWS provider allows only one policy owner per bucket: with two, each apply overwrites the other's policy, so the original owner could drop the inventory grant and the resource below could overwrite later changes with its saved snapshot. If an aws_s3_bucket_policy for <DESTINATION_BUCKET_NAME> already exists in the project (or an aws_s3_bucket for it still sets the inline policy argument), do not emit the file or resource below. Append the S3InventoryWrite-<MODULE_NAME> statement from the block below to that resource's policy instead (as a statement block if it is built from an aws_iam_policy_document), keeping its existing statements, and add that resource to the module's depends_on. This also applies when the live bucket has no policy yet: the module stays on "none" so it does not become a second owner.

Only when nothing in the project manages the policy, save the policy recorded in Phase 2 §3 as existing-destination-policy.json next to the generated .tf file, byte for byte. Terraform reads it with file() rather than having it pasted into the HCL, because a pasted policy is parsed as a template and breaks on IAM policy variables such as ${aws:username}. The try(...) wraps a single-object Statement in a list, and the for drops any earlier copy of this module's statement, so the result is the same on every plan. <MODULE_NAME> is the module's name (dd-sm-<DD_ORG_ID>, plus the Region suffix when multi-Region), and the prefix in Resource must match destination_prefix:

locals {
  existing_destination_policy = jsondecode(file("${path.module}/existing-destination-policy.json"))
  existing_destination_statements = [
    for statement in try([merge(local.existing_destination_policy.Statement)], local.existing_destination_policy.Statement) :
    statement if try(statement.Sid, "") != "S3InventoryWrite-<MODULE_NAME>"
  ]
}

resource "aws_s3_bucket_policy" "inventory_destination" {
  bucket = "<DESTINATION_BUCKET_NAME>"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = concat(local.existing_destination_statements, [
      {
        Sid       = "S3InventoryWrite-<MODULE_NAME>"
        Effect    = "Allow"
        Principal = { Service = "s3.amazonaws.com" }
        Action    = "s3:PutObject"
        Resource  = "arn:aws:s3:::<DESTINATION_BUCKET_NAME>/datadog-inventories/*"
        Condition = {
          StringEquals = { "aws:SourceAccount" = "<AWS_ACCOUNT_ID>" }
          ArnLike      = { "aws:SourceArn" = ["arn:aws:s3:::<SOURCE_BUCKET_NAME>"] }
        }
      }
    ])
  })
}

List every source bucket in this Region in aws:SourceArn (the same list as the module's source_bucket_names), and add aws_s3_bucket_policy.inventory_destination to the module's depends_on. Multi-Region: one file, locals block and policy resource per Region, with Region-suffixed names (for example existing-destination-policy-eu-west-1.json) and provider = aws.<alias> on the resource.

Access-log bucket

Access-log bucket - create it + grant log delivery (only when enable_access_logging = true above). By default the skill provisions a fresh, dedicated SSE-S3 bucket for access logs and attaches the delivery grant the module omits - the same shape dd-aws-ccm uses for the CUR bucket. S3's log-delivery service can't write until the bucket policy allows it, and a fresh bucket has ACLs disabled, so the bucket policy is the only mechanism - without it the apply succeeds but no log is ever delivered:

resource "aws_s3_bucket" "access_logs" {
  bucket = "<ACCESS_LOG_BUCKET_NAME>"   # dedicated; name it like the destination, e.g. dd-s3-access-logs-<AWS_ACCOUNT_ID>-<REGION>
}

resource "aws_s3_bucket_server_side_encryption_configuration" "access_logs" {
  bucket = aws_s3_bucket.access_logs.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"   # SSE-S3 - server access logs are never delivered to an SSE-KMS bucket
    }
  }
}

resource "aws_s3_bucket_policy" "access_log_delivery" {
  bucket = aws_s3_bucket.access_logs.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "S3ServerAccessLogsPolicy"
        Effect    = "Allow"
        Principal = { Service = "logging.s3.amazonaws.com" }
        Action    = ["s3:PutObject"]
        Resource  = "${aws_s3_bucket.access_logs.arn}/s3-access-logs/*"   # prefix must match access_log_prefix
        Condition = {
          # Every source bucket in THIS Region logs here - list them all (matches the module's source_bucket_names).
          # aws:SourceAccount is the source buckets' account (AWS_ACCOUNT_ID from Phase 1).
          ArnLike      = { "aws:SourceArn" = ["arn:aws:s3:::<SOURCE_BUCKET_NAME>"] }
          StringEquals = { "aws:SourceAccount" = "<AWS_ACCOUNT_ID>" }
        }
      }
    ]
  })
}

Set the module's access_log_bucket_name = "<ACCESS_LOG_BUCKET_NAME>" and add aws_s3_bucket.access_logs, aws_s3_bucket_server_side_encryption_configuration.access_logs and aws_s3_bucket_policy.access_log_delivery to the module's depends_on so the bucket, its encryption, and its delivery grant exist before the module wires up source-bucket logging (the module takes only the bucket name, so there's no implicit edge to the grant otherwise). Bring-your-own: if the user supplies an existing dedicated log bucket, drop the aws_s3_bucket / aws_s3_bucket_server_side_encryption_configuration, point the policy's bucket at its name, replace ${aws_s3_bucket.access_logs.arn} in the statement's Resource with arn:aws:s3:::<ACCESS_LOG_BUCKET_NAME>, keep only the policy resource that ends up holding the statement in the module's depends_on (drop aws_s3_bucket.access_logs and aws_s3_bucket_server_side_encryption_configuration.access_logs), and merge this S3ServerAccessLogsPolicy statement into its current policy rather than emitting a standalone resource (a second aws_s3_bucket_policy would silently clobber the existing one). Multi-Region: one access-log bucket + grant per Region, each scoped to that Region's source buckets and using its aliased provider.

SSE-KMS destination buckets

SSE-KMS destination buckets (only if Phase 2 §3 found the destination bucket is SSE-KMS-encrypted). The module does not handle KMS, so add the two grants it omits - otherwise S3 can't write the encrypted report and Datadog can't read it (silent failure). <KMS_KEY_ARN> is the KMSMasterKeyID from Phase 2; the AWS-managed aws/s3 key is not supported by S3 Inventory, so this only applies to a customer-managed key (CMK).

  1. Read side - Datadog's role needs kms:Decrypt (the module's read policy only grants s3:GetObject). Add an inline policy:

    resource "aws_iam_role_policy" "datadog_storage_management_kms" {
      role = "<DATADOG_INTEGRATION_ROLE_NAME>"
      name = "DatadogStorageManagementKMSDecrypt-<DD_ORG_ID>-<DESTINATION_BUCKET_REGION>"
    
      policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Sid      = "DDStorageKMSDecrypt"
            Effect   = "Allow"
            Action   = ["kms:Decrypt"]
            Resource = "<KMS_KEY_ARN>"
          }
        ]
      })
    }

    IAM identifies an inline policy by role and name, so with more than one SSE-KMS destination, emit one of these per destination with a Region-unique Terraform label and the Region in name; two resources with the same name would keep overwriting each other.

  2. Key-policy grants - the CMK's key policy must let the S3 service generate a data key (kms:GenerateDataKey) and, for locked-down keys, let the Datadog role decrypt (kms:Decrypt). The inline IAM grant in step 1 only takes effect if the key policy delegates to IAM (the default Enable IAM User Permissions statement); a hardened key that removed it makes the IAM grant inert, so the key policy must name the role directly. A key policy is a whole-policy resource the module doesn't own. Add both grants to the Terraform resource that owns the key policy if the project has one, otherwise merge them out-of-band (see "Merge the KMS key policy" and "Check for Existing Terraform" in SKILL.md).

Multiple Regions

The HCL for the "Multiple Regions" case in SKILL.md: one Region-pinned provider alias per Region:

provider "aws" {
  alias               = "us_east_1"
  region              = "us-east-1"
  allowed_account_ids = ["<AWS_ACCOUNT_ID>"]
}

provider "aws" {
  alias               = "eu_west_1"
  region              = "eu-west-1"
  allowed_account_ids = ["<AWS_ACCOUNT_ID>"]
}

Then, per Region, the destination bucket and module instance (this example is the eu-west-1 one):

resource "aws_s3_bucket" "inventory_destination_eu_west_1" {
  provider = aws.eu_west_1
  bucket   = "<EU_DESTINATION_BUCKET>"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "inventory_destination_eu_west_1" {
  provider = aws.eu_west_1
  bucket   = aws_s3_bucket.inventory_destination_eu_west_1.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

module "storage_management_eu_west_1" {
  source    = "DataDog/storage-management-datadog/aws"
  version   = "~> 1.0"
  providers = { aws = aws.eu_west_1 }

  name                                 = "dd-sm-<DD_ORG_ID>-eu-west-1"
  datadog_aws_integration_role_name    = "<DATADOG_INTEGRATION_ROLE_NAME>"
  source_bucket_names                  = ["<EU_SOURCE_BUCKET>"]
  destination_bucket_name              = "<EU_DESTINATION_BUCKET>"
  destination_prefix                   = "datadog-inventories/"
  destination_bucket_policy_management = "create"
  manage_destination_lifecycle         = true  # new-bucket case only (see the single-Region block)

  # Access logs - include only if opted in with a same-Region Forwarder ARN (Phase 2 §5).
  # When enabled, also emit a same-Region dedicated access-log bucket + grant (aws_s3_bucket{,_policy}) for this
  # Region, and add them to depends_on below alongside the destination bucket.
  # enable_access_logging  = true
  # access_log_prefix      = "s3-access-logs/"
  # datadog_forwarder_arn  = "<EU_DATADOG_FORWARDER_ARN>"  # the Forwarder Lambda must be in THIS Region - S3 invokes only same-Region Lambdas
  # access_log_bucket_name = "<EU_ACCESS_LOG_BUCKET>"  # same-Region SSE-S3 (AES256) bucket with NO existing notifications - the module manages this bucket's notification config authoritatively (overwrites existing). A fresh dedicated bucket is safest (Phase 2 §5).

  # New-bucket case only - gate the module on the per-Region destination bucket (mirrors the single-Region block).
  depends_on = [aws_s3_bucket.inventory_destination_eu_west_1, aws_s3_bucket_server_side_encryption_configuration.inventory_destination_eu_west_1]
}

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts8d3 checks · Risk SAFE
  • Gen Agent Trust Hub8d

    The skill facilitates the setup of Datadog Storage Management for AWS S3 using Terraform. It adheres to security best practices for credential handling, validates AWS environments, and ensures all infrastructure changes are reviewed and approved by the user. It also includes a secure mechanism for downloading official supplemental skills.

  • Socket8d

    No alerts

  • Snyk8d

    Risk: LOW · No issues

Signed by skilld at 618fae3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated last week
Other metadata
metadata
{
  "version": "1.0.3",
  "author": "datadog-labs",
  "repository": "https://github.com/datadog-labs/agent-skills",
  "tags": "datadog,aws,storage,storage-management,s3,s3-inventory,terraform",
  "alwaysApply": "false",
  "tools": "terraform"
}

README badge

README badge for datadog-labs/agent-skills/dd-aws-storage-management