CloudWatch Monitoring for AOS
Note: Enable OpenSearch application logs (index slow logs, search slow logs, error logs, audit logs) and configure CloudTrail for API-level auditing. Store logs in encrypted CloudWatch Logs groups (specify
--kms-key-idat log group creation:aws logs create-log-group --log-group-name /aws/opensearch/my-domain --kms-key-id arn:aws:kms:<region>:<account>:key/<key-id>).
Key Metrics to Monitor
| Metric | Threshold | Action |
|---|---|---|
CPUUtilization |
> 80% sustained | Scale up instance type or add nodes |
JVMMemoryPressure |
> 80% | Increase instance size; check for large aggregations |
ClusterStatus.red |
= 1 | Immediate: check for unassigned shards |
ClusterStatus.yellow |
= 1 | Investigate: replica shards not allocated |
FreeStorageSpace |
< 20 GB (adjust based on provisioned storage) | Add EBS capacity or migrate old indices to UltraWarm |
SearchLatency |
> 500ms p99 | Optimize queries; consider adding data nodes |
IndexingLatency |
> 100ms p99 | Check bulk queue; scale indexing capacity |
ThreadpoolSearchRejected |
> 0 | Search queue full; scale or throttle clients |
Creating CloudWatch Alarms
Cluster Health (Red)
aws cloudwatch put-metric-alarm --alarm-name aos-cluster-red \
--namespace AWS/ES --metric-name ClusterStatus.red \
--dimensions Name=DomainName,Value=my-domain Name=ClientId,Value=<account-id> \
--statistic Maximum --period 60 --evaluation-periods 1 \
--threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
--alarm-actions arn:aws:sns:<region>:<account>:my-alertsREQUIRED: SNS topics receiving CloudWatch alarms MUST have KMS encryption enabled. CloudWatch alarm notifications may contain cluster status, metric values, and other sensitive operational data. Enable encryption when creating the topic:
aws sns create-topic --name my-alerts \ --attributes KmsMasterKeyId=alias/aws/snsFor existing topics:
aws sns set-topic-attributes --topic-arn <arn> --attribute-name KmsMasterKeyId --attribute-value alias/aws/snsVerify all SNS subscription recipients belong to authorized personnel before deploying alarms.
JVM Memory Pressure
aws cloudwatch put-metric-alarm --alarm-name aos-jvm-pressure \
--namespace AWS/ES --metric-name JVMMemoryPressure \
--dimensions Name=DomainName,Value=my-domain Name=ClientId,Value=<account-id> \
--statistic Maximum --period 300 --evaluation-periods 3 \
--threshold 80 --comparison-operator GreaterThanOrEqualToThreshold \
--alarm-actions arn:aws:sns:<region>:<account>:my-alertsFree Storage Space
aws cloudwatch put-metric-alarm --alarm-name aos-low-storage \
--namespace AWS/ES --metric-name FreeStorageSpace \
--dimensions Name=DomainName,Value=my-domain Name=ClientId,Value=<account-id> \
--statistic Minimum --period 300 --evaluation-periods 1 \
--threshold 20480 --comparison-operator LessThanOrEqualToThreshold \
--alarm-actions arn:aws:sns:<region>:<account>:my-alertsRecommended Alarm Set
For production domains, create alarms for:
- ClusterStatus.red (immediate)
- ClusterStatus.yellow (sustained 15 min)
- JVMMemoryPressure > 80% (sustained 15 min)
- CPUUtilization > 80% (sustained 15 min)
- FreeStorageSpace < 20 GB (immediate; adjust based on provisioned storage)
- ThreadpoolSearchRejected > 0 (sum over 5 min)
- AutomatedSnapshotFailure > 0 (immediate)
