All skills
aws avatar

/amazon-opensearch-service

@04f39cf

Guides migration, provisioning, search, log-analytics, trace-analytics, and Agentic AI Assistant workflows for Amazon OpenSearch Service and Serverless across six capabilities — migration (Solr/ES/self-managed into AOS/AOSS, schema/query translation, sizing, cutover); provisioning (domain + AOSS lifecycle, upgrades, FGAC, monitoring); search (vector / semantic / hybrid / RAG with Bedrock); log-analytics (PPL, OSI, anomaly detection, Dashboards); trace-analytics (OTel spans, service maps, Data Prepper); ai-assistant (natural language data exploration, incident investigation, root cause analysis). Triggers on OpenSearch, AOS, AOSS, Elasticsearch, Solr, vector/k-NN/semantic/hybrid search, RAG, log analytics, PPL, trace analytics, ISM, FAISS, HNSW, Migration Assistant, UltraWarm, OR1, query my data, analyze logs, investigate errors, root cause analysis.

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/amazon-opensearch-service

This session only. Nothing lands on disk.

referencesassessment-workflow.md

≈12k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Migration capability — assessment workflow

This file is the entry point for the migration capability. It owns the workflow for producing a migration assessment from Apache Solr (6.x–9.x), Elasticsearch (1.x–8.x), or self-managed OpenSearch (in-place upgrades 1.3 → 2.19 → 3.x) to Amazon OpenSearch Service or Serverless. It also indexes the rest of the migration capability content.

When to use this capability

SKILL.md routes here when the user is migrating to AOS / AOSS. Concrete triggers:

  • Phrases: "migrate from X", "move off Solr", "ES → OpenSearch", "Migration Assistant for Amazon OpenSearch Service", "Historical Data Migration", "Live Traffic Migration", "Capture and Replay", "refactor my schema.xml", "should I migrate?", "what's the path?", "high-level overview"
  • Pasted artifacts: schema.xml, solrconfig.xml, _cat/indices, _cluster/health, _nodes/stats, version strings ("ES 7.10", "OS 1.3", "Solr 8.11"), vendor names ("Elastic Cloud", "Amazon OpenSearch")
  • Stakeholder intake: "what do you need from me", "before we go deeper", "starting to look at migrating"

All migration files (capability index)

After loading this entry, you can discover every migration-capability file from this list. There are NO other migration files outside references/assessment-*.md.

File Purpose
assessment-workflow.md (this file) Workflow + intake + compatibility scan + path selection + sizing handoff + readiness
assessment-gotchas.md Production gotcha catalog. Each entry carries a Category: tag (TRUE_BLOCKER / MIGRATION_SPECIFIC / OPERATIONAL_CONSIDERATION / COST_TCO / CLARIFICATION) that determines whether it surfaces under Migration specifics or Risks/blockers. Cite by number (#1–#N).
assessment-knowledge-retrieval.md Topic → tool → URL recipe for batched verification
assessment-shape-full-assessment.md Shape recipe: 9-section FULL_ASSESSMENT
assessment-shape-overview.md Shape recipe: OVERVIEW_REQUEST (3–4 phases + 1 URL + next step)
assessment-shape-focused-operational.md Shape recipe: FOCUSED_OPERATIONAL runbook
assessment-shape-translation.md Shape recipe: drop-in DSL translation
assessment-shape-schema-conversion.md Shape recipe: field-by-field mapping
assessment-shape-sizing-only.md Shape recipe: instance class + count + storage
assessment-shape-comparative-decision.md Shape recipe: pick + comparison table + decision driver
assessment-shape-anti-pattern-pushback.md Shape recipe: refusal + right-tool recommendation

Cross-cutting refs you may also load: sizing.md, vector-knn.md, observability.md, security.md, personas.md, assessment-gotchas.md.

Step 0a: detect the response shape

Once in this capability, classify the prompt into ONE of the 8 shapes. State the detected shape in your first sentence (e.g., "Detected shape: FULL_ASSESSMENT — Solr 8.11 with schema.xml paste.").

Shape Detect from Output expectations
FULL_ASSESSMENT Rich prompt with workload context, cluster sizing, asks for migration plan / "produce an assessment" / pasted schema.xml + _cat/indices + traffic numbers 9 sections (Executive Summary / Source / Target / Migration Path / Sizing / Readiness / Risks / Next Steps / Citations)
OVERVIEW_REQUEST "What's the path?" / "high-level overview" / "walk me through it" / business-stakeholder framing without artifacts 3–4 named phases + 1 inline URL + clear next step. NOT a 6-question intake.
FOCUSED_OPERATIONAL "Cheapest path", "<100 GB", "quickest way", "smallest reindex window", a specific operational ask Concrete runbook with reindex.remote.allowlist or equivalent; no full report scaffold
TRANSLATION_TASK "Translate this Solr query" / "convert this DSL" / "what's the OpenSearch equivalent of X" Drop-in JSON / code with caveats inline
SCHEMA_CONVERSION User pasted schema.xml, ES mapping, or asks "map these fields" Field-by-field mapping, gap register, brief migration path callout
SIZING_ONLY "What instance class?" / "size this cluster" / a workload spec but no migration Instance + count + storage formula derivation
COMPARATIVE_DECISION "Managed vs Serverless?" / "should we A or B?" / "FAISS or Lucene?" / "how do you reconcile these constraints?" / prompt names ≥3 simultaneous hard constraints (e.g., zero-downtime + zero-data-loss + no-third-party-tooling + EU residency) Pick-one + comparison table + decision driver. Constraint-trilemma sub-shape when ≥3 constraints are named — see § 2.5 of the recipe.
ANTI_PATTERN_PUSHBACK Wrong-fit migration (e.g. Postgres + transactional + small dataset; ID-only lookups; sub-GB exact-match workload framed as a search migration) REFUSE to size; recommend right tool; list future-fit triggers that would make OpenSearch correct later

After choosing a shape, load references/assessment-shape-<shape>.md for the recipe.

Always-true migration facts

These facts are stable-core for the AWS OpenSearch / Migration Assistant for Amazon OpenSearch Service ecosystem and do not need per-claim verification.

ES → OpenSearch fork rules:

  • ES ≤ 7.10.2 (pre-fork): Snapshot/Restore directly into Amazon OpenSearch is supported.
  • ES ≥ 7.11 (post-fork ELv2/SSPL — includes 7.11–7.17 and all 8.x): Snapshot/Restore is NOT supported into Amazon OpenSearch. Use Migration Assistant for Amazon OpenSearch Service Historical Data Migration, or _reindex from remote for small datasets.
  • ES 1.x / 2.x / 5.x / 6.x: Migration Assistant for Amazon OpenSearch Service Historical Data Migration is the primary path (multi-major hop required). Historical Data Migration supports source ES versions all the way back to 1.0.

Solr → OpenSearch is a refactor, not a lift-and-shift:

  • Schema, queries, configs all need translation. Document-level migration only — there is NO segment/snapshot path between Solr and OpenSearch.
  • Migration Assistant for Amazon OpenSearch Service does support Solr backfill (and Live Traffic Migration). Do NOT tell a customer the service is Elasticsearch-only. For target restrictions and source/target eligibility, see § "Source / target rules" below.
  • Solr <uniqueKey> → bind to _id on _bulk/index AND map as keyword.
  • Solr <copyField source="A" dest="B"/> → "copy_to": "B" in OpenSearch mapping.
  • Solr mm syntax passes UNCHANGED as minimum_should_match.
  • eDisMax qf field^boost → multi_match type: best_fields with the same boosts.
  • Solr q.op=AND → set default_operator: AND on query_string (OpenSearch defaults to OR — top cause of result divergence).

OpenSearch in-place upgrade rules:

  • The mechanism is called blue/green upgrade (aws opensearch start-domain-upgrade --target-version OpenSearch_<x.y>). Name it explicitly when recommending an in-place upgrade — do not hand-wave with "upgrade in place" or describe it as "a long minor-version chain." AOS spins up a green cluster at the target version, syncs, and cuts over.
  • AOS supports multi-version jumps within 2.x and within 3.x via blue/green — you do NOT need to step every minor version (e.g., 2.5 → 2.7 → 2.9 → 2.11 → 2.19 is wrong; 2.5 → 2.19 in one blue/green is correct). The only mandatory waypoint is 2.19 when crossing into 3.x. Source < 1.3 needs a 1.x → 1.3 hop first because only 1.3 can upgrade to 2.x.
  • The 1.3 → 2.19 → 3.x mandatory waypoints are about the engine version, not Lucene segments. Pre-2.x indexes carry Lucene 8 segments; OS 3.x runs Lucene 10. Lucene's segment format is forward-only — Lucene 10 cannot read Lucene 8, so any pre-2.0 index destined for 3.x MUST be reindexed (typically on a 2.x intermediate) before the 3.x hop.
  • In-place blue/green upgrades are free for managed customers.

Source / target rules:

  • The Solr-target restriction is architectural: Migration Assistant for Amazon OpenSearch Service Solr migrations (both Historical Data Migration backfill and Live Traffic Migration live cutover) target OpenSearch 3.x or Amazon OpenSearch Serverless ONLY — never OS 1.x/2.x. The legacy "Solr is RFS-only / not supported by Capture & Replay" wording is OUTDATED.
  • Migration Assistant for Amazon OpenSearch Service 3.0 deploys to Amazon EKS (Kubernetes). Earlier versions used ECS — plan EKS prereqs.

Source / target version support is canonical at the AWS docs page — do NOT replicate version cells in this skill. Cite https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/source-and-target-versions.html when version-range questions come up. ES 8.x is supported by both Historical Data Migration and Live Traffic Migration (confirmed); the documented page is the source of truth for the current floor and ceiling on each mode.

Components of a migration

Every migration to Amazon OpenSearch Service decomposes into up to three independent components. Pick which apply for this customer; not all migrations need all three.

Component What it covers When you need it
1. Historical Data Migration Move the existing data corpus (documents, indexes, mappings) from source to target. Almost always — unless the customer is starting greenfield with no historical data.
2. Live Traffic Migration Replicate live writes during cutover so the target stays in sync until you flip readers/writers. Only when the maintenance window the customer can grant is shorter than the time Historical Data Migration takes for this dataset. Skip when the window comfortably covers HDM duration, or for batch / read-heavy workloads.
3. Application Code Rewrite Update the application's client code, query DSL, schema, configs, and language-specific bindings to match OpenSearch idioms. Required for Solr → OpenSearch (different APIs entirely) and for major-version rewrites (Lucene segment wall, X-Pack feature port, etc.). Skipped on like-for-like ES → AOS where the wire-protocol overlap is sufficient.

Strategy selection happens per component — see the three sections below.

1. Historical Data Migration — strategies

Source/target version eligibility for each tool: https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/source-and-target-versions.html (canonical; do not replicate version cells in this skill).

Tool What it does Notes
Migration Assistant for Amazon OpenSearch Service Historical Data Migration Managed backfill / historical-data migration into AOS or AOSS. Solr → OS 1.x/2.x is NOT supported (target must be OS 3.x or Serverless).
Snapshot/Restore (direct) One-shot snapshot from a self-managed source restored on AOS. BLOCKED for ES ≥ 7.11 (post-fork license).
_reindex from remote Native OpenSearch API; reindexes from a remote cluster. PRIMARY for <100 GB ES ≥ 7.11 with ≥30 min cutover window.
OSI (OpenSearch Ingestion) Managed Data Prepper pipelines (good when paired with Application Code Rewrite that emits to OSI). NOT for Solr sources.
In-place blue/green upgrade AWS-managed engine version step (use for OS-self-managed → AOS-managed at the same engine version). Free for managed customers.

Primary-tool selection rules:

  • ES ≥ 7.11 sources <100 GB with a ≥30-minute maintenance window → _reindex from remote.
  • ES ≥ 7.11 sources >500 GB, multi-index complex, or unreachable from target → Migration Assistant for Amazon OpenSearch Service Historical Data Migration.
  • ES ≤ 7.10.2 (pre-fork) with a maintenance window → Snapshot/Restore.
  • Solr (any version) → Migration Assistant for Amazon OpenSearch Service Historical Data Migration (Solr is document-level only; no segment path; target must be OS 3.x or Serverless).
  • OS 1.3+ → OS 2.19/3.x at the same self-managed → AOS-managed boundary → in-place blue/green.

2. Live Traffic Migration — strategies

Tool What it does Notes
Migration Assistant for Amazon OpenSearch Service Live Traffic Migration Captures source writes (Capture Proxy in front of source) and replays them onto the target until clocks sync. Pair with Historical Data Migration for full historical + live. Same Solr-target restriction (OS 3.x / Serverless only).
Application-layer dual-write Customer's application code writes to both source and target during cutover. NOT a third-party tool — it's customer code under customer change control. Useful when the customer rejects "third-party tooling" but still needs zero downtime.
Read-only window Pause writes for the duration of Historical Data Migration; cut over once HDM completes. The read-only window IS however long HDM takes for this dataset (gated by source size, network bandwidth, ingest worker count). Cheapest. Default whenever the maintenance window comfortably covers the estimated HDM duration.

Skip Live Traffic Migration entirely when: the customer's maintenance window covers the time Historical Data Migration takes for this dataset, OR the workload is batch / read-heavy with no live-write SLA. Estimate HDM duration up-front (cluster size, bandwidth, parallelism) and validate it fits the budget before committing to skip Live Traffic Migration.

3. Application Code Rewrite — strategies

Code rewrites cover schema (schema.xml → OpenSearch mappings), query DSL (eDisMax → multi_match/bool, ES X-Pack → OpenSearch native plugins), language-binding swaps (solrj → opensearch-java, elasticsearch-py → opensearch-py), and ingest-pipeline conversion.

Strategy What it does When to recommend
Agentic tools (e.g. Amazon Q Developer Agent for code transformation, Claude / Cursor with appropriate prompts) Iterative LLM-driven rewrite of the customer's application source. Low ceremony; works well for small-to-medium codebases and language-binding swaps. Default for one-off / small-team rewrites where the customer can review diffs case-by-case.
AWS Transform Custom AWS-managed bulk code transformation pipeline. Migration Assistant for Amazon OpenSearch Service ships with an example solrj → opensearch-java transformation that customers can use as the starting template, then extend for their own bindings. Best fit for large codebases, regulated rewrites where the transformation pipeline must be auditable, or when the customer already has an AWS Transform deployment for other languages.
Manual rewrite Engineer-driven port. The customer's own team writes the new code. Only when the codebase is small AND the team needs the cycles to internalize OpenSearch's mental model — pedagogical, not efficient.

Trigger: Any time the source is Solr OR uses ES X-Pack-only features (ELSER, Watcher, Canvas, ES SQL with non-portable functions) OR has client code in a language whose opensearch-* client has API differences (notably solrj ↔ opensearch-java, ES Painless scripts ↔ OpenSearch Painless), Application Code Rewrite is required and must appear as its own line in the migration plan.

Skip Application Code Rewrite when: ES → AOS at the same major engine version with no X-Pack dependencies, the existing elasticsearch-* client is wire-compatible with the target OpenSearch version (test the version-compatibility matrix before assuming), and the customer keeps their existing schema.

Sizing-related universal rules (apply when this capability sizes a target)

  • Current-generation instances. Default to Graviton (r7g/r8g for memory-optimized; m7g/m8g for cluster managers). r6g/r6gd only with explicit justification (existing RIs, specific compatibility need).
  • Input honesty. When sizing on UNKNOWN inputs, lead with [BLOCKER — need input] OR present 2–3 tiered bands (small/medium/large workload assumption). Never present a single point estimate built on invented numbers.

Cross-capability handoff

If the user prompt spans capabilities — for example "migrate from Solr AND set up RAG on the new domain" — produce the migration response and close with a one-line handoff:

Workflow at a glance

0. ANTI_PATTERN_GUARD → halt + pushback if wrong-fit migration
1. IDENTIFY → first sentence restates source/version/region/persona
2. FINGERPRINT → JSON shape from artifacts; mark UNKNOWN for missing
3. COMPATIBILITY SCAN → gap register with severity (BLOCKING / HIGH / MEDIUM / LOW)
4. TARGET SHAPE → Managed (default) vs Serverless NextGen vs Classic
5. MIGRATION PATH → for each component the customer needs (Historical Data Migration, Live Traffic Migration, Application Code Rewrite — not all are required), pick a primary strategy from the per-component tables in § "Components of a migration". Skip components that don't apply to this workload.
6. SIZING → instance class + node count + storage + shards (mandatory ONLY when Step 0 anti-pattern guard does not trigger)
7. READINESS → 7-dimension 0-100 score; tier GREEN/YELLOW/RED
8. RENDER → templates in assets/; required sections in order
9. VERIFY → batched pass for [verify] markers; only resolve in ONE pass

Speed contract. Steps 3–7 draft directly from this file's tables (stable-core). Tag every version-volatile value with [verify]. Resolve all [verify] markers in ONE batched pass at Step 9 — never do per-claim retrieval.


Step 0: ANTI_PATTERN_GUARD

Before doing anything else, check whether this is a wrong-fit migration:

  • Workload is exact-match + small (<10K records) + transactional + relational integrity (foreign keys, hierarchy, audit logs)
  • Common anti-patterns: Postgres HR DB, simple key-value cache, transactional payment ledger, audit log with regulatory immutability

If TRUE: HALT this workflow. Dispatch to references/assessment-shape-anti-pattern-pushback.md.

The recipe says: REFUSE to provide OpenSearch sizing. Verbatim refusal template: "I'm not going to spec instance types or shard counts because recommending a topology for a migration that shouldn't happen lends false confidence to the wrong path."

FORBIDDEN HEDGES (never use): "Option B", "if you insist", "search-only sidecar", "if you do go this path", "for completeness".

Recommend the right tool (e.g. Postgres pg_trgm + tsvector + GIN) with concrete DDL recipe. Name future-fit triggers that WOULD change the answer.


Step 1 — Identify

Restate in first sentence: source engine + version + target region + persona. Examples:

  • "You're on Apache Solr 8.11 SolrCloud, target Amazon OpenSearch Service us-east-1, DevOps / Platform Engineer — here's the assessment."
  • "ES 7.17 on Elastic Cloud → Amazon OpenSearch Service us-west-2, Search Relevance Engineer persona — here's the path."
  • "OS 1.3 with NMSLIB k-NN → OpenSearch 3.x — here's the upgrade plan."

Persona detection:

Cue Persona
"I'm a product manager" / "I'm a director" / "I'm a TPM" / "I'm in product" Business Stakeholder — six business questions
Explicit "what do you need from me" + no technical artifact + no migration question Business Stakeholder
"What's the path?" / "high-level overview" / "what's involved?" Overview request — produce 2–4 phase substantive overview, NOT business intake
Pastes schema.xml, _cat/*, query DSL, sizing spec Search Relevance Engineer OR DevOps / Platform Engineer
Mentions latency, sizing, instance types, JVM, sharding DevOps / Platform Engineer
Mentions BM25, query relevance, custom analyzers, ELSER, eDisMax Search Relevance Engineer
Mixed signals Pick most technical voice; add 1-page exec header

Step 2 — Fingerprint

For technical personas, capture this JSON from whatever artifacts the customer pasted. Mark missing fields UNKNOWN. Don't run a multi-prompt interview.

{
  "source_engine": "elasticsearch | opensearch | solr",
  "version": "7.10.2",
  "summary": {
    "node_count": 6,
    "index_count": 120,
    "total_docs": 3200000000,
    "total_gb": 8000,
    "plugin_count": 7,
    "health_status": "green",
    "ilm_used": false,
    "watcher_used": false,
    "runtime_fields_used": false,
    "source_disabled": false,
    "post_fork": false,
    "dih_used": false,
    "velocity_response_writer": false,
    "xslt_response_writer": false,
    "custom_lib_count": 0
  },
  "indices": [
    {"name": "logs-2024-11", "docs": 50000000, "store_size": "120gb", "primary": 6, "replica": 1}
  ],
  "plugins": [
    {"node": "ip-10-0-1-12", "component": "analysis-icu", "version": "7.10.2"}
  ],
  "files_provided": ["_cat/indices.json", "_cluster/health.json", "_nodes/stats.json"]
}

For Solr, build from schema.xml, solrconfig.xml, and intake answers.

For Business Stakeholder persona, run the six-question intake first (see § Business Stakeholder intake).

Business Stakeholder intake

Six business questions only — frame in business terms, no technical artifacts:

  1. Use case — what is the search system powering today? E-commerce? Internal documents? Support knowledge base? Log analytics? Security/SIEM?
  2. Users — internal employees vs external customers? Approximate user count (DAU and total)?
  3. Criticality / SLA — Tier-1 customer-facing, important-but-not-critical, best-effort? Any explicit availability SLA (99.9%, 99.95%)? RPO/RTO?
  4. Traffic — peak QPS and sustained QPS? If unknown, give user count + usage pattern; we'll estimate.
  5. Index updates — how many docs added/updated per day? Streaming (continuous) or bulk (nightly batch)? 12–24 month growth projection?
  6. Document size — average size in KB, or one-line description of what a typical document looks like?

You MUST NOT ask a Business Stakeholder for: schema.xml, solrconfig.xml, _cat/indices JSON, shard/replica counts, plugin lists, instance types, JVM heap sizes, query DSL, custom analyzers, eDisMax syntax, version preferences, budget figures, auth-backend specifics. Asking any of those FAILS the Business Stakeholder branch.

After the six are answered, translate them into a technical fingerprint internally and proceed to the compatibility scan.


Step 3 — Compatibility scan / gap register

Emit one gap-register entry per finding:

{
  "id": "ES_RUNTIME_FIELDS",
  "feature": "Elasticsearch runtime fields",
  "severity": "BLOCKING|HIGH|MEDIUM|LOW",
  "lane": "migration-specific|risk-blocker",
  "category": "schema|query|auth|ops|dashboards|plugin|version|sizing",
  "description": "...",
  "workaround": "...",
  "citation_url": "..."
}

Severity + Lane rubric

Every gap-register entry MUST carry both a Severity and a Lane. Severity is the magnitude of the behavioral impact; Lane is the framing for the customer (does the migration plan already handle it, or does the customer need to act?). Canonical vocabulary lives in compatibility-rubric.md; the abbreviated copy is below.

Severity Meaning
BLOCKING No workaround in OpenSearch; customer must rearchitect, accept feature loss, or stop
HIGH Major behavioral difference or required rewrite — affects code or queries
MEDIUM Configuration / mapping difference handled at migration time
LOW Cosmetic / negligible (terminology rename, metric name change)
Lane When to use
migration-specific The migration plan already includes a documented remediation (transformer flag, sanitizer, default override) that the path applies on the customer's behalf. Frame as "this is how the migration handles X".
risk-blocker The item genuinely constrains the migration: no known fix, capacity-plan implications, irreversible target choices, or customer action required to land.

The Severity × Lane combination determines whether the row deducts from the Compatibility readiness weight (see readiness-rubric.md — only risk-blocker-lane rows deduct).

Always-flag list (apply on every assessment)

Elasticsearch sources (canonical X-Pack → OpenSearch plugin/feature map; do not duplicate elsewhere — link to this section):

Feature Severity Lane OpenSearch equivalent
ES Runtime fields HIGH risk-blocker Partial: derived fields (OS 2.15+) — limited functionality
X-Pack ILM MEDIUM risk-blocker ISM — JSON does NOT import; rebuild policy
X-Pack Watcher HIGH risk-blocker Alerting plugin — rewrite all monitors
X-Pack ML jobs / anomaly detection HIGH risk-blocker Anomaly Detection plugin — different API; rewrite
ELSER (Elastic Learned Sparse Encoder) HIGH risk-blocker Use neural_sparse query with SageMaker-hosted model
ES SQL HIGH migration-specific OpenSearch SQL plugin — most queries work; verify edge cases
Cross-Cluster Replication (CCR) MEDIUM risk-blocker CCR plugin available on Managed (not Serverless)
Cross-Cluster Search (CCS) HIGH risk-blocker Not supported on Serverless; partial on Managed
Painless inline scripts MEDIUM risk-blocker Supported on Managed (not Serverless)
_type (multi-type) HIGH (ES 5.x/6.x) migration-specific Removed in OS 1.0; Migration Assistant metadata transformer flattens before reindex
ES _parent (5.x) HIGH risk-blocker Replaced by join field type — schema redesign
fielddata: true on text (ES 1.x/2.x) BLOCKING migration-specific OOM risk if untouched, but Migration Assistant metadata transformer strips it and adds .keyword subfield automatically
Field-level encryption LOW migration-specific Field masking via FGAC
Authentication: native realm / file realm MEDIUM migration-specific Internal user database via FGAC
Authentication: LDAP / AD MEDIUM migration-specific Supported via FGAC backend
Authentication: SAML MEDIUM migration-specific Supported via Cognito or direct SAML
Snapshot from ES ≥ 7.11 BLOCKING risk-blocker ELv2/SSPL license lockout — no snapshot path; use Migration Assistant Historical Data Migration or _reindex

Solr sources:

Feature Severity Lane OpenSearch equivalent
<uniqueKey> field MEDIUM migration-specific Map as keyword AND bind to _id on every _bulk/index
<copyField source="A" dest="B"/> LOW migration-specific "copy_to": "B" on field A in mapping
_version_ field LOW migration-specific OMIT — OpenSearch has its own _version
Deprecated/removed Solr field types (Trie*, etc.) HIGH migration-specific For the full Solr 7/8/9 deprecation list, see assessment-shape-schema-conversion.md §Section D — Gap register.
solr.CurrencyField HIGH migration-specific Denormalize: price_amount (scaled_float) + price_currency (keyword) + price_base numeric
solr.EnumField / EnumFieldType MEDIUM migration-specific Denormalize: <name> (keyword) + <name>_rank (integer)
solr.ICUCollationField LOW migration-specific icu_collation_keyword — analysis-icu plugin pre-installed on AOS
Solr ≤ 5.x TF-IDF default similarity HIGH risk-blocker OpenSearch defaults BM25 — relevance tuning required
eDisMax qf field^boost LOW migration-specific multi_match type: best_fields with same boosts
eDisMax pf (phrase boost) MEDIUM risk-blocker should + multi_match type:phrase — behavioral approximation; A/B against Solr
eDisMax tie LOW migration-specific tie_breaker on multi_match type: best_fields
Solr mm (e.g. 2<-25%) LOW migration-specific minimum_should_match — same syntax, passes UNCHANGED
Solr q.op=AND HIGH migration-specific default_operator: AND on query_string (when source solrconfig.xml overrides Solr's OR default to AND; OpenSearch defaults to OR — top divergence cause)
Removed Solr handlers/writers (DIH, Velocity, XSLT, etc.) HIGH risk-blocker For the full Solr 7/8/9 deprecation list, see assessment-shape-schema-conversion.md §Section D — Gap register.
Custom analyzers (Java JARs) HIGH risk-blocker Audit Migration Assistant for Amazon OpenSearch Service's auto-translation; rare cases need transformer override
<dynamicField> regex patterns MEDIUM migration-specific Migration Assistant for Amazon OpenSearch Service usually auto-translates; audit edge cases
<requestHandler class="solr.SearchHandler"> LOW migration-specific Translate to _search endpoint with default_field and default_operator from solrconfig.xml

OpenSearch in-place upgrade:

  • The upgrade chain is OS 1.0–1.2 → 1.3 → 2.19 → 3.x. The 1.3-and-2.19 mandatory hops are policy (won't change); each minor inside that chain is a moving target. For the current per-version hop matrix, see version-migration.html.

Lucene segment-format wall (root cause for pre-2.x reindex)

OS 1.3 indexes ship Lucene 8 segments. OS 3.x ships Lucene 10. Lucene's segment format is forward-only — Lucene 10 cannot read Lucene 8. Any pre-OS-2.0 index destined for OS 3.x MUST be reindexed before reaching 3.x.

Parallel cause: NMSLIB k-NN engine was REMOVED in OS 3.0 (deprecated in 2.19). Pre-existing NMSLIB indexes must reindex into FAISS before reaching 3.x.

OS 3.x breaking changes

  • JDK 21 minimum (was JDK 17 in 2.x)
  • Security Manager → Java agent
  • Removed k-NN settings: knn.algo_param.ef_construction (legacy), several others
  • Lucene 10 baseline (segment format wall — see above)
  • NMSLIB engine removed
  • Default search.allow_expensive_queries = false (more strict)

Stable-core ES → OS facts (drafted directly)

  • ES 7.10.2 is the engine fork point. ES 1.0 GA was Feb 2014; OS 1.0 GA was July 2021.
  • ES 7.0 removed _type placeholder; OS 1.0 removed types entirely (placeholder _doc blows up _reindex).
  • ES 7.11+ relicensed to ELv2/SSPL (Jan 2021) — Snapshot/Restore from those versions is NOT supported into Amazon OpenSearch Service.
  • ES 5.x/6.x cannot one-hop snapshot/restore into modern OpenSearch (Lucene segment versions and snapshot format are incompatible).

Stable-core Solr → OS facts

  • Solr → OpenSearch is document-level, NOT segment-level. There is NO snapshot path between the two engines.
  • Solr stored="false" fields can ONLY be recovered via Migration Assistant for Amazon OpenSearch Service Historical Data Migration (reads source Lucene segments).
  • Lucene 10 (OS 3.x) cannot read Lucene 8 (pre-OS 2.0) — segment format is forward-only.

Step 4 — Target shape

Default to Managed Domain when ambiguous. Re-evaluate Serverless after stable traffic.

Managed-only requirements

If ANY of these is needed, the answer is Managed:

  • SIEM / Security Analytics plugin
  • Custom plugins (Java JARs, custom analyzers, custom processors)
  • Lucene k-NN engine, FAISS IVF, FAISS PQ
  • Cross-Cluster Replication (CCR) or Cross-Cluster Search (CCS)
  • UltraWarm / Cold tiering
  • Manual snapshots
  • Inline scripts (Painless)
  • T-class burstable instances (only available on Managed)
  • User-tunable sharding
  • Predictable steady-state (RI savings opportunity)
  • Very small clusters (≤ 2 OCU steady-state — Managed is cheaper)

Serverless eligibility

Serverless is a fit when ALL of these hold:

  • Workload is full-text search, time-series logs (Classic only), or vector (NextGen or Classic)
  • Bursty traffic (10×+ swings) or zero-ops preference
  • No custom plugins, no CCR/CCS, no manual snapshots, no inline scripts
  • Vector workload uses simplified API (NextGen) OR FAISS HNSW only (Classic)

NextGen vs Classic Serverless (CRITICAL distinction)

NextGen collections:

  • Support Search and Vector Search types only (no TIME_SERIES on NextGen)
  • Vector Search uses simplified API — system auto-picks engine and configuration
  • Custom document IDs supported
  • 32x compression by default
  • GPU index build acceleration available
  • 10s refresh interval

Classic collections:

  • Support Search, Vector Search, AND TIME_SERIES
  • Vector Search requires explicit engine: faiss (Lucene/IVF/PQ NOT supported on Classic)
  • TIME_SERIES and VECTORSEARCH reject custom _id PUT/upsert (Classic only — NextGen vector accepts custom IDs)
  • 60s refresh interval for vector Classic; 10s for search Classic

OCU model:

  • 1 OCU = 6 GiB RAM + matching vCPU + ~120 GiB ephemeral storage
  • Redundancy ON: minimum 1 indexing OCU (0.5 × 2) + 1 search OCU (0.5 × 2) — billed even idle
  • Redundancy OFF: minimum 0.5 OCU × 2 for first collection
  • Default max: 10 OCUs each indexing/search; up to 1700 each on request

Tiebreaker rules

  • Vector + simplified API + custom IDs needed → Serverless NextGen Vector
  • Vector + IVF/PQ/Lucene needed → Managed
  • Logs > 2.5 TiB hot → time-series Classic Serverless OR Managed UltraWarm (compare cost)
  • Mixed keyword + vector (Classic) → ⚠️ Vector Search collections cannot share OCUs with Search/TimeSeries collections — doubles idle floor
  • Otherwise → Managed default; re-evaluate after stable traffic

Step 5 — Migration path (per-component selection)

For each of the three components, decide whether it applies and which strategy fits. Skip components that don't apply. See § "Components of a migration" above for the strategy menu under each.

Component Required when… Skip when…
Historical Data Migration The customer has existing data they want preserved on the target (almost always). Greenfield; no historical data; or full re-emit from the system of record is faster than migrating.
Live Traffic Migration The maintenance window the customer can grant is shorter than the time Historical Data Migration will take, AND the workload has live writes during cutover. The maintenance window comfortably covers the duration of Historical Data Migration (the read-only window IS however long HDM takes for this dataset) — pause writes, run HDM, cut over. Also skip when workload is read-heavy / batch with no live-write SLA.
Application Code Rewrite Source is Solr (different APIs); ES uses X-Pack-only features (ELSER, Watcher, ES SQL with non-portable functions); language-binding swap required (solrj → opensearch-java); major Lucene segment-format wall (OS 3.x). ES → AOS at the same major engine version, no X-Pack, schema preserved, existing elasticsearch-* client wire-compatible.

Once you've decided which components apply, pick the primary strategy under each from the per-component tables in § "Components of a migration". Common combinations:

Customer profile Components in plan
Solr → AOS (any size) Historical Data Migration + (optional) Live Traffic Migration + Application Code Rewrite
Pre-fork ES → AOS, maintenance window OK Historical Data Migration only (Snapshot/Restore strategy)
Post-fork ES (≥ 7.11) <100 GB, 30-min window Historical Data Migration only (_reindex from remote strategy)
Post-fork ES, large or multi-index, zero-downtime Historical Data Migration + Live Traffic Migration (both via Migration Assistant for Amazon OpenSearch Service)
ES with X-Pack-only features Historical Data Migration + Application Code Rewrite (replace X-Pack code-paths)
OS self-managed → AOS, same engine Historical Data Migration only (in-place blue/green strategy)
Greenfield (new app, no source) None of the above — go to the provisioning capability instead.

Historical Data Migration — quick strategy lookup

This is a fast lookup over the strategies in § "Components of a migration → Historical Data Migration". Pick by source profile.

Source / Scenario Strategy Notes
Solr (any volume) Migration Assistant for Amazon OpenSearch Service Historical Data Migration → OS 3.x or Serverless Required for stored="false" fields; target restriction is architectural (no OS 1.x / 2.x for Solr)
Solr, all stored="true", small dataset, easy re-emit Solr /export + _bulk (manual) Cheap; auditing required to confirm no stored="false"
Multi-major ES backfill (pre-7.x) Migration Assistant for Amazon OpenSearch Service Historical Data Migration Multi-major hop only practical here
Pre-fork ES (≤ 7.10.2) Snapshot/Restore Pre-fork — simplest path while license boundary allows it
Post-fork ES (≥ 7.11), small dataset, ≥ 30 min window _reindex from remote (PRIMARY) Snapshot/Restore BLOCKED post-fork; HDM overkill at small scale
Post-fork ES (≥ 7.11), large or multi-index complex Migration Assistant for Amazon OpenSearch Service Historical Data Migration Snapshot/Restore BLOCKED post-fork
OS in-place upgrade Blue/green upgrade Free; mandatory 2.19 hop for 1.3 → 3.x
OS self-managed → AOS Migration Assistant for Amazon OpenSearch Service preferred Same engine; Migration Assistant for Amazon OpenSearch Service streamlines
Cross-cloud / cross-account Migration Assistant for Amazon OpenSearch Service OR OSI with SigV4 auth
GovCloud Migration Assistant for Amazon OpenSearch Service Historical Data Migration Verify current shard-size cap against live docs ([verify])

Live Traffic Migration — quick strategy lookup

Workload profile Strategy
Pre-fork ES (≤ 7.10.2), zero-downtime Migration Assistant for Amazon OpenSearch Service Live Traffic Migration paired with Historical Data Migration (Snapshot/Restore for the bulk)
Post-fork ES (≥ 7.11), zero-downtime Migration Assistant for Amazon OpenSearch Service Live Traffic Migration paired with Historical Data Migration
Solr, zero-downtime Migration Assistant for Amazon OpenSearch Service Live Traffic Migration paired with Historical Data Migration
Continuous replication post-cutover Live Traffic Migration or CCR (CCR if both ends are AOS)
High-throughput live writes OSI fan-out OR staged migration; Live Traffic Migration is fine for typical sustained throughput
Customer rejects "third-party tooling" Application-layer dual-write (customer code; not third-party)
Maintenance window long enough to cover Historical Data Migration Skip Live Traffic Migration entirely — pause writes, run Historical Data Migration, cut over. The read-only window IS however long Historical Data Migration takes for this dataset. Estimate that duration up-front (gated by source size, network bandwidth, ingest worker count) and validate it fits the customer's maintenance budget.

Source/target version eligibility for each tool: see Migration Assistant for Amazon OpenSearch Service source-and-target versions. The ES 7.11 fork point is architectural (license boundary — Snapshot/Restore is blocked into AOS post-fork) and stays inline.

Always-true rules (across components)

These hold regardless of which component you're picking a strategy for:

  • Solr sources: Historical Data Migration via Migration Assistant for Amazon OpenSearch Service is the PRIMARY HDM strategy regardless of volume — required for stored="false" fields. Recommend non-Migration Assistant alternatives only when (a) every needed field is stored="true", (b) easy re-emit from system of record, AND (c) dataset is small. Flag the trade-off.
  • _source: false indexes: Migration Assistant for Amazon OpenSearch Service Historical Data Migration is the ONLY supported HDM path — verify _source status before recommending anything else.
  • Post-fork ES (≥ 7.11): do NOT recommend Snapshot/Restore for HDM — the license fork is architectural; use Migration Assistant for Amazon OpenSearch Service Historical Data Migration or _reindex from remote.
  • Multi-major ES backfill (pre-7.x → modern OS): Migration Assistant for Amazon OpenSearch Service Historical Data Migration is the only practical multi-major path; pair with Live Traffic Migration when zero-downtime is required.
  • Target = Serverless: Migration Assistant for Amazon OpenSearch Service Live Traffic Migration is supported but document IDs are preserved only on SEARCH collection types (TIMESERIES and VECTORSEARCH Classic use server-generated IDs unless using NextGen vector with custom-ID support).
  • Post-fork ES (≥ 7.11) at small scale with usable maintenance window: _reindex from remote is the PRIMARY HDM strategy — Migration Assistant for Amazon OpenSearch Service Historical Data Migration becomes primary only when the dataset is large, multi-index/complex, OR source→target network reachability is impossible.

Step 6 — Sizing

Sizing is mandatory ONLY when Step 0 anti-pattern guard does not trigger. When Step 0 halts the workflow, do NOT produce a sizing recommendation — providing topology for a migration that shouldn't happen lends false confidence to the wrong path.

See references/sizing.md for the full math. Quick rules for migration-assessment sizing:

  • Match-source rule (when source sizing is provided): stay close to source RAM (8–16 GB per data node) unless workload signals (peak QPS, retention, page-cache headroom) justify uplift. Recommending 4× source RAM without explicit signal-based justification is a sizing miss.
  • Default starting point (no source sizing provided): 3 × r7g.large.search (8 GB heap each) + 3 × m7g.large.search cluster managers + gp3 200 GiB. Multi-AZ.
  • Storage formula: source_data × (1 + replicas) × 1.45.
  • Shard size: 10–30 GiB for search, 30–50 GiB for write-heavy/logs.
  • Primary shards: (source + growth) × 1.1 / desired_shard_size, rounded up to multiple of data-node count.
  • Cluster manager and per-node shard caps: see sizing.md §Topology defaults. Source of truth: bp.html#bp-sharding.

Output mandatory for technical persona (when Step 0 does not trigger): instance class + node count + storage + shard count. Pointing at the Pricing Calculator without naming the instance is incomplete.

Sizing under UNKNOWNs

When source metrics are UNKNOWN (data volume, peak QPS, doc count), use ONE of two patterns — never single point-estimate on assumed value:

Pattern A — [BLOCKER — need input]:

[BLOCKER — need source data volume to size]
Cannot recommend instance class until source size is known.
SHARE: total docs, total GB, peak QPS, retention.

Pattern B — Tiered bands keyed to unknown variable:

| Source size | Recommended | Reason |
|---|---|---|
| <100 GB | 3 × m7g.large.search | Match-source for tiny |
| 100–500 GB | 6 × r7g.large.search | Standard mid-tier |
| >500 GB | 9 × r7g.2xlarge.search | Headroom for growth |

Step 7 — Readiness score

Canonical scoring rules and worked example live in readiness-rubric.md. The abbreviated form:

Score across 7 dimensions (0–100 total). Tier:

  • GREEN ≥ 80 — proceed; surface top items to flag in §7 (split across Migration specifics and Risks/blockers)
  • YELLOW 60–79 — PoC + spike on weakest dimension before committing
  • RED < 60 — do not commit; revisit weakest dimension first

Tier override: any BLOCKING risk-blocker-lane row caps the readiness tier at YELLOW until the customer commits to the remediation path.

Dimension Weight What it captures
Compatibility 25 Number/severity of risk-blocker-lane gap-register entries. migration-specific-lane entries do NOT deduct because the migration plan already handles them.
Operational readiness 15 Team familiarity with OpenSearch, on-call coverage
Sizing fitness 15 Confidence in instance class + count for projected workload
Data-movement complexity 15 Volume, transformations, cutover style
Cutover complexity 10 Downtime tolerance, dual-write feasibility, rollback plan
Sizing-input completeness 10 How much sizing input the customer provided
Stakeholder alignment 10 Sign-off from product/security/infra

You MUST cross-reference at least 1 gotcha from assessment-gotchas.md by number — many gotchas are not in any AWS doc and missing them is the most common readiness gap. Whether the gotcha contributes to the Compatibility deduction depends on its Category: tag (only TRUE_BLOCKER and customer-action MIGRATION_SPECIFIC items deduct).


Step 8 — Render report

Templates in assets/:

  • report-template.md → MIGRATION_ASSESSMENT.md (full assessment, source-agnostic)
  • executive-summary-template.md → EXECUTIVE_SUMMARY.md (Business Stakeholder)
  • tech-deepdive-template.md → TECHNICAL_DEEP_DIVE.md (Search Relevance Engineer / DevOps)
  • Solr-specific: solr-report-template.md, solr-index-template-skeleton.md, solr-gap-register.md
  • ES-specific: elasticsearch-report-template.md, elasticsearch-index-template-skeleton.md, elasticsearch-gap-register.md

Required sections (in this order):

  1. Executive Summary
  2. Source
  3. Target
  4. Migration Path
  5. Sizing
  6. Readiness
  7. Risks
  8. Next Steps
  9. Citations

Step 9 — Verify (batched)

Collect every [verify] marker into one list. Resolve in ONE batch:

  1. Gather all [verify] markers (feature-parity rows, plugin-support, current instance families + regional availability, NextGen/Migration Assistant for Amazon OpenSearch Service capability rows, per-version k-NN default engine, exact per-version limits)
  2. Retrieve in as few calls as possible: one AWS-docs sweep, one OpenSearch-project sweep, one regional-availability call. Run independent retrievals concurrently.
  3. Resolve each tag: replace with confirmed value + add source URL + retrieval timestamp to Citations.

Pre-delivery checklist (reproduce in response, tick each):

- [ ] All 9 required sections emitted, in order
- [ ] Every [verify] marker resolved
- [ ] Citations section: ≥ 3 unique URLs with retrieval timestamp
- [ ] https://calculator.aws surfaced for cost handoff
- [ ] ≥ 1 gotcha from assessment-gotchas.md cross-referenced
- [ ] Target shape default = MANAGED unless workload justifies Serverless
- [ ] Each required component (Historical Data Migration / Live Traffic Migration / Application Code Rewrite) has a primary strategy named
- [ ] Persona-correct depth
- [ ] No embedded credentials/endpoints/master usernames
- [ ] Security section cites references/security.md and confirms each control
- [ ] Step 0 anti-pattern guard evaluated; if triggered, NO sizing emitted

If any box can't be ticked, fix the gap before responding.


Always-true rule reminders (already in SKILL.md — repeated here for context)

  • ES 7.10.2 is the engine fork point. ES ≥ 7.11 (post-fork) snapshot is NOT supported into AOS.
  • Solr → OpenSearch is document-level, NOT segment-level — refactor, not lift-and-shift.
  • OS 1.3 → 2.19 → 3.x. (1.0–1.2 need 1.3 hop first.)
  • Lucene 8 → 10 wall: pre-2.x indexes must reindex before reaching OS 3.x.
  • q.op=AND divergence — when the source solrconfig.xml sets q.op=AND (a common production override; Solr's own default is OR), OpenSearch defaults to OR. Set default_operator: AND on query_string.
  • Solr mm syntax — passes UNCHANGED as minimum_should_match.
  • NMSLIB engine REMOVED in OS 3.0+ (was deprecated in 2.19). FAISS default since 2.18.
  • Migration Assistant for Amazon OpenSearch Service Solr backfill targets only OS 3.x or Serverless.
  • Migration Assistant for Amazon OpenSearch Service 3.0 deploys to Amazon EKS.
  • NextGen Vector simplified API (no engine/mode); supports custom doc IDs.
  • Classic Serverless Vector requires engine: faiss; rejects custom _id on TIMESERIES/VECTORSEARCH.
  • Vector Search collections cannot share OCUs with Search/TimeSeries — doubles idle floor.
  • Default to current Graviton families: r7g/r8g for memory-optimized; m7g/m8g for cluster managers.
  • T-class for prod data nodes is forbidden (CPU credits exhaust).

Source: SKILL.md on GitHub

No alerts28d3 checks · Risk SAFE
  • Gen Agent Trust Hub28d

    This skill is a highly structured and security-conscious guide for managing Amazon OpenSearch Service and Serverless. It provides comprehensive instructions for migrations, provisioning, and analytics while strictly adhering to AWS security best practices, such as using SigV4 signing, IAM least-privilege, and AWS Secrets Manager for credential handling.

  • Socket28d

    No alerts

  • Snyk28d

    Risk: LOW · No issues

Signed by skilld at 04f39cf. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
metadata
{
  "version": "2"
}

README badge

README badge for aws/agent-toolkit-for-aws/amazon-opensearch-service