AWS Secrets Manager: building software around the API, and what good hygiene looks like
Audience: an engineering agent (or engineer) designing, writing, reviewing, or operating an application that consumes or manages secrets in AWS Secrets Manager.
Companion: ../crud/aws-secrets-manager-crud-guide.md for exact API calls in ten languages. Cross-provider rules live in ../../SKILL.md.
Documentation verified: October 26, 2026
The CRUD guide answers "how do I call the API correctly." This document answers "how should the software around those calls be shaped so that secrets stay secret, rotate cleanly, and fail safely." Every rule here is phrased so an agent can check it in a diff.
1. Choose the consumption pattern first
Decide how the running workload receives a secret before writing code. Prefer the option that puts the least plaintext in the fewest places while still supporting rotation.
| Pattern | Use when | Rotation story | Hygiene cost |
|---|---|---|---|
| In-process SDK fetch + TTL cache (CRUD guide §9) | Long-running services you own | Re-read on TTL expiry and on auth failure | Lowest exposure; you own retry/caching code |
| AWS caching component (Java, Python, .NET, Go, Rust) | Same as above, less code | TTL-driven | Configure TTL < rotation overlap |
| Workload Credentials Provider / Lambda extension (localhost:2773) | Lambda, mixed-language fleets, legacy apps | TTL; refreshNow=true |
Localhost HTTP + SSRF token; cached plaintext in agent memory |
| EKS Secrets Store CSI Driver (ASCP) | Kubernetes workloads that must read files | Rotation reconciler polling | Files on tmpfs; avoid syncing to Kubernetes Secret unless required |
ECS task definition secrets |
Simple containers, static config | Requires task restart | Env vars leak via /proc, crash reporters, child processes |
| Build-time injection | Never | — | Secret baked into image/artifact; treat as a leak |
Default for new services: in-process fetch through an abstraction (section 3) with a TTL cache and refresh-on-authentication-failure. Use the CSI driver or task-definition injection only when the application cannot be changed.
2. Identity and access design
- Workload identity only. The application authenticates with a role: ECS task role, EKS Pod Identity/IRSA, Lambda execution role, EC2 instance profile, or OIDC federation for CI. No
AWS_ACCESS_KEY_IDin application configuration, images, or repositories. A long-term key is a finding. - Split reader and writer roles. Runtime roles get
secretsmanager:GetSecretValue(pluskms:Decryptfor a customer managed key) on named ARNs. Provisioning/rotation roles getCreateSecret,PutSecretValue,UpdateSecretVersionStage,DeleteSecret,RestoreSecret,TagResource. Nobody needssecretsmanager:*. - Scope by name prefix and environment. Use names like
<app>/<env>/<purpose>and grantarn:aws:secretsmanager:<region>:<acct>:secret:<app>/<env>/*. Prefer separate AWS accounts per environment over prefixes inside one account; a prefix is a convention, an account is a boundary. - ABAC with care. Tag-based conditions (
secretsmanager:ResourceTag/app) scale well, but anyone who canTagResourcecan change access. Restrict tagging to the provisioning role. - Resource policies only for cross-account access. Any identity allowed
PutResourcePolicymust have thesecretsmanager:BlockPublicPolicy = truecondition. [S01] - KMS.
aws/secretsmanageris fine for single-account use. Use a customer managed key for cross-account access or separation of duties, and constrain it withkms:ViaService = secretsmanager.<region>.amazonaws.comand the Secrets Manager encryption context. [S01] - Network. Use an interface VPC endpoint for Secrets Manager.
aws:SourceVpce/aws:SourceVpcconditions can break AWS services that call on your behalf (for example rotation); test before enforcing. Avoidaws:SourceIpfor the same reason. [S01]
Least-privilege runtime policy template (agent: fill in exact ARNs; do not widen to *):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadOwnSecrets",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": [
"arn:aws:secretsmanager:us-east-1:123456789012:secret:billing/prod/db-??????",
"arn:aws:secretsmanager:us-east-1:123456789012:secret:billing/prod/stripe-??????"
]
},
{
"Sid": "DecryptViaSecretsManagerOnly",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:us-east-1:123456789012:key/REPLACE_WITH_KEY_ID",
"Condition": { "StringEquals": { "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com" } }
}
]
}-?????? matches exactly the six-character suffix Secrets Manager appends to every ARN, so billing/prod/db-?????? does not also match billing/prod/db-replica-AbCdEf, which db-* would. Use full ARNs when the secrets already exist.
3. Code structure
Isolate Secrets Manager behind a small port so business code never imports the SDK and tests never need AWS.
// secrets/port.go — the only interface business code sees.
type Provider interface {
// Get returns the current value. Implementations cache and must be safe for concurrent use.
Get(ctx context.Context, ref string) ([]byte, error)
// Invalidate drops a cached value, e.g. after the downstream rejects a credential.
Invalidate(ref string)
}Rules for the adapter (an agent should verify each in review):
- Construct the SDK client once per process and reuse it; it is concurrency-safe and pools connections.
- Reference secrets by ARN in configuration (
DB_SECRET_ARN=arn:...), never by value. Configuration files may contain ARNs; they must never contain values. - Cache with a TTL (typically 5–15 minutes) shorter than the rotation overlap window, use single-flight so a cache miss under load produces one API call, and add jitter to expiry.
- Refresh on authentication failure: when a downstream returns "invalid credentials," call
Invalidate, re-fetch once, retry once, then fail. This is what makes rotation seamless. - Fail fast at startup if a required secret is missing or unreadable; do not start in a half-configured state. After startup, prefer serving with the last good cached value during a transient Secrets Manager outage, bounded by a maximum staleness.
- Timeouts and retries: set a context deadline on every call; use the SDK
standard(oradaptive) retry mode; never write an unbounded retry loop. - Typed parsing: if the secret is JSON, unmarshal into a struct with only the fields you need, and validate presence. Never pass the raw JSON string around.
- Narrow lifetime: return the value to the call site that uses it (DSN builder, HTTP client constructor). Do not store it in global config objects that get logged or serialized.
- No secrets in exceptions: wrap errors with the ARN and AWS error code; never with the value or the full request.
4. Lifecycle: provision, rotate, revoke, delete
Provisioning (infrastructure as code). Create the secret container (name, KMS key, policy, tags, rotation config) in IaC. Treat secret values separately:
- Terraform/OpenTofu: a value set through
secret_stringlands in plaintext in state. Prefer the write-onlysecret_string_wowithsecret_string_wo_version(Terraform 1.11+, recent AWS provider) so the value never persists in state or plan, or let rotation populate the first value. Increment the version argument to push a new write-only value. [S02] [S03] - CloudFormation: use
GenerateSecretStringfor generated values, or dynamic references ({{resolve:secretsmanager:...}}) to consume without copying. - Never put a literal value in IaC source, CI variables printed in logs, or pull-request descriptions.
Rotation.
- Prefer managed rotation (for supported services such as RDS) or a rotation Lambda. Rotation can run as often as every 4 hours. [S01]
- Choose alternating users (two credentials, swap which is current) for systems that cannot tolerate a brief window where only the new password works; choose single user where clients re-read promptly.
- Consumers must read
AWSCURRENTonly, never pin aVersionId, and refresh on auth failure. - Rotation that only writes Secrets Manager without changing the credential at the source is not rotation. The source change and the
AWSPENDING → AWSCURRENTmove must be one orchestrated flow.
Revocation and deletion.
- On suspected exposure: rotate first (new value at source +
PutSecretValue), confirm consumers picked it up, then revoke the old credential at the source. Deleting the secret does not revoke anything. - Delete with a recovery window (7–30 days). Alarm on any
GetSecretValuefor that ARN during the window to discover forgotten consumers. UseForceDeleteWithoutRecoveryonly for secrets that were never used or were created by a test.
5. Logging, telemetry, and error hygiene
- SDK debug logging off in real environments. botocore
DEBUG, Go v2aws.LogResponseWithBody, Java wire logging, and .NET response logging print response bodies containingSecretString. Guard with configuration that cannot be enabled by an environment variable in production. - Structured logs carry identifiers, not values: secret ARN (or a hash of it), version ID, AWS request ID, error code, latency.
- Tracing: do not attach request/response bodies to spans for the Secrets Manager client or for downstream calls that carry the credential (database connect strings,
Authorizationheaders). - Crash and core dumps: disable or scrub in production; heap dumps contain cached secrets. Restrict who can trigger profilers and debuggers in production.
- Redaction as defense in depth: a log redactor for known secret shapes is useful but is not the control. The control is never handing the value to a logger.
- CloudTrail is the audit trail. It records who called
GetSecretValueon what, but not the value. Keep it enabled organization-wide and route Secrets Manager data events to your SIEM.
6. Testing strategy
| Layer | Approach |
|---|---|
| Unit | Inject a fake Provider returning generated values (crypto/rand), never real secrets or realistic-looking keys that trip scanners. |
| Adapter | Stub the SDK (botocore Stubber, Go smithy-go middleware stubs, aws-sdk-client-mock for JS) to test error mapping: ResourceNotFoundException, DecryptionFailure, throttling, scheduled deletion. |
| Integration | A dedicated sandbox account with a unique prefix per test run, cleaned up with ForceDeleteWithoutRecovery (test-created secrets only). Emulators such as LocalStack are fine for request shapes, not for IAM/KMS semantics. |
| Rotation | Run rotation in staging and assert that consumers keep working across the AWSPENDING → AWSCURRENT transition. |
| Regression for leaks | A test that sets a sentinel secret, exercises error paths, and asserts the sentinel never appears in captured logs. |
7. CI/CD and supply chain
- CI authenticates to AWS with OIDC federation to a role scoped to the pipeline's needs; no stored AWS keys in CI.
- Pipelines that deploy secret containers do not need
GetSecretValue. - Pin SDK versions; update them through reviewed dependency PRs; enable dependency vulnerability scanning.
- Run secret scanning (for example gitleaks, trufflehog, GitHub secret scanning, Amazon CodeGuru Reviewer's secrets detector) on every push and on history at onboarding. [S01]
- Do not echo environment variables in CI steps; mask is a display feature, not a control.
8. Operations and incident response
- Inventory: every secret has
owner,app,env, androtationtags (non-sensitive), and a documented consumer list. - Alerts:
GetSecretValuefrom unexpected principals or networks;PutResourcePolicy,DeleteSecret,RestoreSecret, KMS key policy changes; failed decrypts. - AWS Config / Security Hub: enable the Secrets Manager controls (rotation enabled, rotation succeeded, unused secrets removed).
- Leak playbook: (1) rotate at source and in Secrets Manager; (2) verify consumers refreshed; (3) revoke the old credential; (4) find the leak path and remove the copy (logs, tickets, repos); (5) review CloudTrail for use of the leaked credential; (6) record the incident.
- Break-glass: a separate role with
GetSecretValueon critical secrets, protected by MFA and alerting on every use.
9. Review checklist for the engineering agent
- No secret values, access keys, or realistic test secrets in code, config, IaC, fixtures, or docs.
- Application uses a role; no
AWS_ACCESS_KEY_IDconfiguration path exists. - IAM grants only the actions used, on exact ARNs or a tight prefix; reader and writer roles are separate.
- Secrets referenced by ARN; create stores and reuses the returned ARN.
- Single SDK client; context deadlines; SDK retry mode set; no unbounded loops.
- TTL cache, single-flight, refresh-on-auth-failure, fail-fast startup.
- JSON secrets parsed into typed structures; only needed fields kept.
- No SDK body/wire logging; no secret values in errors, logs, spans, metrics, or panics.
- Writes use
ClientRequestTokencorrectly; create is never treated as upsert. - Deletion uses a recovery window; force delete only for test-created secrets.
- Terraform does not store values in state (
*_woor rotation-populated). - Rotation strategy documented and tested; consumers tolerate the transition.
- Secret scanning and dependency scanning enabled in CI.
10. Anti-patterns
| Don't | Do instead |
|---|---|
aws secretsmanager create-secret --secret-string "$PASSWORD" |
--secret-string file://... from a 0600 file, or generate in-service |
| Fetch the secret on every request | TTL cache with refresh-on-auth-failure |
| Copy the value into an env var for "convenience" | Pass it directly to the client constructor that needs it |
Pin VersionId in config |
Read AWSCURRENT |
One secretsmanager:* policy for all services |
Per-service roles on exact ARNs |
PutSecretValue in a tight loop (e.g. per deploy) |
Write only when the value changes; respect the 10-minute guidance |
| Delete the secret to "revoke" a leaked key | Rotate and revoke at the source; then delete with a window |
| Store dev and prod secrets in one account distinguished by name | Separate accounts per environment |
| Log the parsed config object at startup | Log which ARNs were loaded and their version IDs |