/amazon-opensearch-service

Log-analytics capability — entry point and guide

The AWS MCP server is recommended for executing these commands but is not required — all steps use standard awscurl / AWS CLI syntax.

This file is the entry point for the log-analytics capability. It covers log search at scale, observability, PPL queries, anomaly detection, OpenSearch Dashboards, alerting, and SIEM patterns — including replatforming from Splunk, Datadog, or self-managed ELK.

When to use this capability

SKILL.md routes here when the user is doing log analytics or observability on AOS / AOSS. Concrete triggers:

• Phrases: "PPL query", "OpenSearch Dashboards", "ingest logs", "anomaly detection", "alerting rule", "Splunk replatform", "Datadog alternative", "OSI pipeline", "log search", "SIEM"
• Tasks: query logs, set up OSI ingestion, configure ISM tiering / UltraWarm, build dashboards, define alerts, replatform a Splunk/Datadog/ELK stack

All log-analytics files (capability index)

Cross-cutting refs you may also load: observability.md (ISM / UltraWarm / Cold tiering details), security.md, personas.md.

Cross-capability handoff

• For provisioning the OSI pipeline infra (CloudFormation, IAM, source connectors): see provisioning-reference.md.
• For migrating an existing Splunk / Datadog / ELK stack: see assessment-workflow.md (use the Splunk-replatform shape).
• For trace data on the same domain: see trace-analytics-trace-queries.md.

Overview

This guide instructs you on how to perform log analytics against an existing OpenSearch domain or collection. The approach is discovery-first: understand what indices exist, learn the schema, sample the data, then build queries. Do not assume any particular index pattern or field names — discover them.

For the full PPL command and function catalog, see observability-ppl-reference.md.

Query discipline (MUST follow)

• Unknown command → upstream grammar. If a PPL command or function is not in observability-ppl-reference.md, or an emitted query fails with a syntax error, you MUST fetch the raw upstream doc from opensearch-project/sql under docs/user/ppl/ before answering. You MUST NOT invent PPL syntax, because OpenSearch PPL has version-specific commands that do not exist in other query languages.
• Verify or disclose. When the domain/collection endpoint is reachable, you MUST validate every emitted query against _plugins/_ppl; on 0 rows, fall back to _plugins/_ppl/_explain to confirm the plan and surface the empty result; on error, fix and re-validate. When no endpoint is reachable, you MUST state that the query is unverified, because an untested query can silently reference a non-existent field.
• Backtick-quote dotted fields. Field names containing a dot MUST be backtick-quoted (`log.level`, `host.name`), because an unquoted dot is parsed as a path separator and silently resolves to a non-existent field.

Data Plane Access with awscurl

Use awscurl for SigV4-authenticated HTTP requests to AOS/AOSS endpoints.

Setup

pip install awscurl

Credentials: source SigV4 signing credentials from an IAM role (instance profile, ECS task role, or an SSO session) rather than static access keys, and keep endpoints and other config in AWS Secrets Manager or SSM Parameter Store rather than hardcoding them.

Environment Variables

Base Commands

AOS (managed domains):

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "<PPL_QUERY>"}'

AOSS (serverless collections):

awscurl --service aoss --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "<PPL_QUERY>"}'

Use --service es for AOS domains, --service aoss for AOSS collections.

Security Considerations

• Verify domain/collection encryption at rest is enabled before ingesting sensitive data
• Use fine-grained access control (FGAC) to restrict index and field-level access
• Do not ingest PII or credentials without field-level encryption or masking
• Apply data retention policies via ISM to comply with regulatory requirements
• Enable CloudTrail logging to audit control plane API calls, and enable OpenSearch audit logs to track data plane operations (queries, indexing) for compliance

Connecting to the Domain/Collection

Determine the domain or collection type and endpoint using the AWS CLI (or call_aws if the AWS MCP server is available):

• If the user names a domain: aws opensearch describe-domain --domain-name <name> → extract Endpoint and ARN (region from ARN). With AWS MCP: call_aws opensearch describe-domain.
• If the user names a collection: aws opensearchserverless batch-get-collection --names <name> → extract collectionEndpoint. With AWS MCP: call_aws opensearchserverless batch-get-collection.
• If unclear: list with aws opensearch list-domain-names or aws opensearchserverless list-collections

This is important because the connection method, authentication, and available features differ between AOS domains and AOSS collections.

Phase 1 — Discover Available Indices

AOSS Note: OpenSearch Serverless does not support _cat APIs. Use --service aoss instead of --service es for all AOSS requests. For index discovery on AOSS, use PPL: source = * | stats count() by index.

Before writing any query, find out what log indices exist on the domain or collection.

List All Indices

awscurl --service es --region $AWS_REGION \
  "$OPENSEARCH_ENDPOINT/_cat/indices?format=json&h=index,health,docs.count,store.size&s=docs.count:desc"

Look for indices that suggest logs: names containing log, logs, events, audit, access, syslog, otel, cwl (CloudWatch Logs), or date-based patterns like logs-2024.01.15.

List Index Patterns with Aliases

awscurl --service es --region $AWS_REGION \
  "$OPENSEARCH_ENDPOINT/_cat/aliases?format=json&h=alias,index&s=alias"

Check Data Streams

awscurl --service es --region $AWS_REGION \
  "$OPENSEARCH_ENDPOINT/_data_stream"

After discovering indices, ask the user which index or index pattern they want to analyze if it's not obvious. If there are multiple log indices, ask about the relationship between them (e.g., are they daily rollover indices for the same data? different applications? different log levels?).

Phase 2 — Understand the Schema

Once you know the target index pattern, inspect its mapping to learn the available fields.

Get Index Mapping

awscurl --service es --region $AWS_REGION \
  "$OPENSEARCH_ENDPOINT/<INDEX_PATTERN>/_mapping"

Via PPL:

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "describe <INDEX_NAME>"}'

Use a concrete index name (e.g., logs-2024.01.15) for describe, not a wildcard pattern.

Identify Key Fields

From the mapping, identify:

1. Timestamp field — usually @timestamp, timestamp, time, or event.created
2. Log level field — level, log.level, severity, severityText, loglevel
3. Message field — message, msg, body, log, event.original
4. Service/source field — service, service.name, host.name, source, kubernetes.pod.name, resource.attributes.service.name
5. Error fields — error.message, error.stack_trace, exception.type
6. Correlation fields — traceId, trace_id, spanId, request_id, correlation_id

If the mapping is large or unclear, ask the user: "I see fields like X, Y, Z — which field contains the log message? Which one is the log level?"

Sample Documents

Always look at a few real documents to understand the actual data shape — mappings alone can be misleading (e.g., dynamic fields, nested objects, multi-value fields):

Via PPL:

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=<INDEX_PATTERN> | head 5"}'

Review the sample documents to confirm:

• Which fields are actually populated (vs defined but empty)
• The format of timestamps, log levels, and messages
• Whether the message field is structured JSON or free-text
• Whether there are nested objects that need backtick-quoting in PPL

Phase 3 — Ask Clarifying Questions (If Needed)

If the schema is not self-explanatory, ask the user:

• "What does this index contain? Application logs, access logs, audit logs?"
• "I see multiple log indices (X, Y, Z) — are these from different services or different time periods?"
• "The message field appears to contain JSON — should I parse specific fields from it?"
• "I see a trace_id field — do you want to correlate logs with traces?"
• "What time range are you interested in?"

Do not skip this step if the data is ambiguous. Getting the schema right upfront saves failed queries later.

Phase 4 — Perform Analytics

With the schema understood, build PPL queries using the actual field names discovered above. All examples below use placeholder field names — substitute with the real ones.

Running PPL Queries

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "<PPL_QUERY>"}'

For AOSS, use --service aoss instead of --service es.

Log Volume Over Time

source=<INDEX_PATTERN> | stats count() as volume by span(<TIMESTAMP_FIELD>, 1h)

Error Count by Service

source=<INDEX_PATTERN> | where <LEVEL_FIELD> = 'ERROR' | stats count() as errors by <SERVICE_FIELD> | sort - errors

Error Rate Trend

source=<INDEX_PATTERN> | stats count() as total, sum(case(<LEVEL_FIELD> = 'ERROR', 1 else 0)) as errors by span(<TIMESTAMP_FIELD>, 1h)

Recent Errors

source=<INDEX_PATTERN> | where <LEVEL_FIELD> = 'ERROR' | fields <TIMESTAMP_FIELD>, <SERVICE_FIELD>, <MESSAGE_FIELD> | sort - <TIMESTAMP_FIELD> | head 20

Full-Text Search

source=<INDEX_PATTERN> | where match(<MESSAGE_FIELD>, 'connection timeout') | sort - <TIMESTAMP_FIELD> | head 20

Top Error Messages

source=<INDEX_PATTERN> | where <LEVEL_FIELD> = 'ERROR' | top 10 <MESSAGE_FIELD>

Rare Error Messages

source=<INDEX_PATTERN> | where <LEVEL_FIELD> = 'ERROR' | rare <MESSAGE_FIELD>

Log Pattern Discovery

Automatically cluster similar log messages:

source=<INDEX_PATTERN> | where <LEVEL_FIELD> = 'ERROR' | patterns <MESSAGE_FIELD> | fields <MESSAGE_FIELD>, patterns_field | head 30

Error Breakdown by Level and Service

source=<INDEX_PATTERN> | stats count() by <LEVEL_FIELD>, <SERVICE_FIELD>

Time-Filtered Queries

source=<INDEX_PATTERN> | where <TIMESTAMP_FIELD> > DATE_SUB(NOW(), INTERVAL 1 HOUR) | stats count() by <LEVEL_FIELD>

Unique Services/Hosts

source=<INDEX_PATTERN> | stats distinct_count(<SERVICE_FIELD>) as services, distinct_count(<HOST_FIELD>) as hosts

Latency from Structured Logs

If logs contain a duration/latency field:

source=<INDEX_PATTERN> | stats avg(<DURATION_FIELD>) as avg_ms, percentile(<DURATION_FIELD>, 95) as p95_ms, percentile(<DURATION_FIELD>, 99) as p99_ms by <SERVICE_FIELD>

Extract Fields from Unstructured Messages

If the message field contains unstructured text, use grok or parse to extract fields:

source=<INDEX_PATTERN> | grok <MESSAGE_FIELD> '%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:path} %{NUMBER:status}' | stats count() by status

Caveat: grok processes all matching rows in memory. Add | head N before grok on large indices to avoid resource errors.

Phase 5 — Advanced Analysis

Cross-Index Correlation

If logs span multiple indices (e.g., application logs + access logs), correlate using shared fields like request_id, trace_id, or timestamp proximity:

Step 1 — Find an event of interest in one index:

source=<APP_LOGS> | where <LEVEL_FIELD> = 'ERROR' | fields <CORRELATION_FIELD>, <TIMESTAMP_FIELD>, <MESSAGE_FIELD> | head 10

Step 2 — Look up the same correlation ID in the other index:

source=<ACCESS_LOGS> | where <CORRELATION_FIELD> = '<VALUE>' | fields <TIMESTAMP_FIELD>, <MESSAGE_FIELD>

Anomaly Detection

Use PPL's built-in anomaly detection on numeric fields (e.g., log volume, error count):

source=<INDEX_PATTERN> | stats count() as volume by span(<TIMESTAMP_FIELD>, 5m) | ad time_field=<TIMESTAMP_FIELD>

The ad command auto-detects input fields from the pipeline. It works best on time-series data with regular intervals.

Query DSL for Complex Aggregations

For queries that PPL doesn't support well (nested aggregations, scripted fields), fall back to Query DSL:

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/<INDEX_PATTERN>/_search" \
  -H 'Content-Type: application/json' \
  -d '{
  "size": 0,
  "query": {
    "bool": {
      "must": [{"match": {"<LEVEL_FIELD>": "ERROR"}}],
      "filter": [{"range": {"<TIMESTAMP_FIELD>": {"gte": "now-1h"}}}]
    }
  },
  "aggs": {
    "by_service": {
      "terms": {"field": "<SERVICE_FIELD>", "size": 20},
      "aggs": {
        "over_time": {
          "date_histogram": {"field": "<TIMESTAMP_FIELD>", "fixed_interval": "5m"}
        }
      }
    }
  }
}'

Common Log Schemas Reference

When you encounter these common schemas, use the field mappings below:

Elastic Common Schema (ECS)

Timestamp: @timestamp, Level: log.level, Message: message, Service: service.name, Host: host.name, Error: error.message

OTel Logs (logs-otel-v1-*)

Timestamp: @timestamp, Level: severityText, Message: body, Service: `resource.attributes.service.name` (backtick-quoted), Trace: traceId, Span: spanId

Simple JSON Logs

Timestamp: timestamp or @timestamp, Level: level, Message: message or msg, Service: service, Host: host

Syslog

Timestamp: @timestamp, Level: severity, Message: message, Host: host, Program: program, Facility: facility

Apache/Nginx Access Logs

Client: clientip, Request: request, Status: response, Bytes: bytes, Method: verb, Agent: agent

Key PPL Tips for Log Analytics

• Always backtick-quote dotted field names: `log.level`, `host.name`
• Use head N before memory-intensive commands (grok, streamstats, eventstats)
• Use span(<timestamp>, <interval>) for time bucketing — common intervals: 5m, 15m, 1h, 1d
• Use match() for full-text search, like for wildcard patterns, match_phrase() for exact phrases
• Use patterns for automatic log message clustering
• Use dedup to find unique error messages: dedup <MESSAGE_FIELD> | fields <MESSAGE_FIELD>

Index Management with awscurl

Create Log Index with Mappings

awscurl --service es --region $AWS_REGION \
  -X PUT "$OPENSEARCH_ENDPOINT/application-logs" \
  -H 'Content-Type: application/json' \
  -d '{
  "settings": {"number_of_shards": 1, "number_of_replicas": 1},
  "mappings": {
    "properties": {
      "@timestamp": {"type": "date"},
      "level": {"type": "keyword"},
      "message": {"type": "text"},
      "service": {"type": "keyword"},
      "trace_id": {"type": "keyword"}
    }
  }
}'

Bulk Index Log Documents

awscurl --service es --region $AWS_REGION \
  -X POST "$OPENSEARCH_ENDPOINT/_bulk" \
  -H 'Content-Type: application/x-ndjson' \
  -d '{"index": {"_index": "application-logs"}}
{"@timestamp": "2024-01-15T10:30:00Z", "level": "ERROR", "message": "Connection timeout to database", "service": "order-service"}
{"index": {"_index": "application-logs"}}
{"@timestamp": "2024-01-15T10:30:05Z", "level": "INFO", "message": "Retry succeeded", "service": "order-service"}
'

Create ISM Policy for Log Rotation

awscurl --service es --region $AWS_REGION \
  -X PUT "$OPENSEARCH_ENDPOINT/_plugins/_ism/policies/log-rotation-policy" \
  -H 'Content-Type: application/json' \
  -d '{
  "policy": {
    "description": "Hot-warm-delete lifecycle for logs",
    "default_state": "hot",
    "states": [
      {"name": "hot", "actions": [], "transitions": [{"state_name": "warm", "conditions": {"min_index_age": "7d"}}]},
      {"name": "warm", "actions": [{"read_only": {}}], "transitions": [{"state_name": "delete", "conditions": {"min_index_age": "30d"}}]},
      {"name": "delete", "actions": [{"delete": {}}], "transitions": []}
    ],
    "ism_template": [{"index_patterns": ["application-logs*"], "priority": 100}]
  }
}'

Create Data Stream for Time-Series Logs

# Create index template for data stream
awscurl --service es --region $AWS_REGION \
  -X PUT "$OPENSEARCH_ENDPOINT/_index_template/logs-template" \
  -H 'Content-Type: application/json' \
  -d '{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": {"number_of_shards": 1},
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "level": {"type": "keyword"}
      }
    }
  }
}'

# Create the data stream
awscurl --service es --region $AWS_REGION \
  -X PUT "$OPENSEARCH_ENDPOINT/_data_stream/logs-stream"