Gotchas — production failure modes
The traps experienced practitioners hit on Amazon OpenSearch. Each one is a real failure mode that silently breaks plans. Cite by number when the profile matches.
Each entry carries a **Category:** tag that determines which lane it surfaces under in the FULL_ASSESSMENT §7 split (and which assets it deducts from in readiness-rubric.md):
| Category | Meaning | Lane in §7 |
|---|---|---|
TRUE_BLOCKER |
No clean fix; constrains target choice or forces rearchitecture. Deducts from Compatibility weight. | Risks/blockers |
MIGRATION_SPECIFIC |
The migration plan already includes a documented remediation (transformer, sanitizer, config override). Does not deduct unless customer action is required. | Migration specifics |
OPERATIONAL_CONSIDERATION |
Default-behavior thing to know about; affects sizing or operations rather than correctness. | Risks/blockers (when actionable) or Migration specifics (when path-handled). Use judgment. |
COST_TCO |
Pricing/billing trap that affects TCO model accuracy but doesn't block the migration. | Migration specifics — reframe the TCO model. |
CLARIFICATION |
The gotcha is "the customer's claim is wrong / ambiguous"; resolution is pre-work, not a remediation. | Surface as a question, not in either §7 lane. |
1. Solr → OpenSearch is document-level, NOT segment-level
Category: TRUE_BLOCKER
There is NO snapshot path between Solr and OpenSearch — different codecs, schema layouts. Schema, queries, configs all need translation.
Detect: "lift and shift Solr to OpenSearch", "snapshot Solr" Fix: State explicitly that this is a refactor migration. Use Migration Assistant for Amazon OpenSearch Service Solr backfill (Historical Data Migration) or document-level export+bulk for small datasets.
2. ES ≥ 7.11 snapshot/restore is NOT supported on AOS
Category: TRUE_BLOCKER
ES 7.11+ relicensed to ELv2/SSPL (Jan 2021). Snapshot/Restore from those versions to Amazon OpenSearch Service is NOT supported.
Detect: ES version ≥ 7.11 in source fingerprint; customer plans snapshot path
Fix: Use Migration Assistant for Amazon OpenSearch Service Historical Data Migration, or _reindex from remote for small datasets (<100 GB).
3. Lucene 8 → 10 segment-format wall at OS 3.0
Category: TRUE_BLOCKER
OS 3.x ships Lucene 10. Pre-2.x indexes carry Lucene 8 segments. Lucene's segment format is forward-only — Lucene-10 cannot read Lucene-8.
Detect: OS 1.x source upgrading to OS 3.x; ES 7.10 indexes; any pre-OS 2.0 indexes Fix: Reindex affected indexes before upgrading to OS 3.x. Applies to hot, UltraWarm, and cold storage.
4. Per-node shard cap
Category: OPERATIONAL_CONSIDERATION
Detect: shard count > 800/node trending up.
Fix: see sizing.md §Topology defaults for current cluster-manager + shard-cap values; source of truth is bp.html#bp-sharding. Architectural rule: Multi-AZ-with-Standby clusters cap at 1000/node regardless of OS version.
5. Cold storage is NOT directly queryable
Category: OPERATIONAL_CONSIDERATION
Cold storage holds detached indexes — must reattach to UltraWarm before querying. Migration is one index at a time, queue depth 100. Watch WarmToColdMigrationQueueSize.
Detect: "occasional queries on archived data" Fix: Accept warm-up latency (minutes-to-hours), keep data in UltraWarm permanently, or use S3+Athena for true on-demand archives.
6. Serverless redundancy adds an OCU floor
Category: COST_TCO
Architectural rule: Redundancy ON adds an idle OCU floor (separate indexing + search minimums billed continuously).
Detect: Bursty/low-volume customer thinking "I'll only pay for what I use"
Fix: For current OCU minimums, see sizing.md §OCU model and serverless-scaling.html. For tiny non-prod workloads, consider small Managed t3.medium.search. NEVER t2.* or t3.small.search in prod.
7. Vector Search collections cannot share OCUs with Search/TimeSeries
Category: COST_TCO
Architectural rule: a vector search collection can't share OCUs with search and time series collections, even with same KMS key. Adding one vector collection adds a separate idle floor.
Detect: Mixed keyword + vector workload; user assumes one bill
Fix: For current OCU minimums, see sizing.md §OCU model and serverless-scaling.html. If vector is exploratory, run k-NN on existing Managed cluster instead.
8. Serverless ignores most user-supplied index settings
Category: MIGRATION_SPECIFIC
Number of shards, intervals, refresh interval are NOT modifiable on Serverless. index.translog.* and index.routing.allocation.* are dropped. Cannot restore a snapshot to Serverless directly.
Detect: Plan involves restoring an existing snapshot to Serverless
Fix: Use Migration Assistant for Amazon OpenSearch Service's metadata-migration Serverless sanitizer, or hand-strip settings before bulk. Re-validate post-load with GET <idx>/_settings.
9. NextGen TIME_SERIES does NOT exist
Category: TRUE_BLOCKER
NextGen Serverless supports only Search and Vector Search types. TIME_SERIES is Classic-only.
Detect: Customer wants time-series collection AND mentions "NextGen" Fix: Use Classic for TIME_SERIES; or use Managed Domain with ISM-managed time-series indexes (often a better fit at scale).
10. NMSLIB removed in OS 3.0
Category: TRUE_BLOCKER
Detect: source uses NMSLIB engine, target is OS 3.x.
Fix: reindex into FAISS HNSW or FAISS IVF before the 3.x upgrade. Engine matrix and reindex recipe live in vector-knn.md; source of truth for current engines is knn.html.
11. q.op=AND divergence (Solr → OpenSearch)
Category: MIGRATION_SPECIFIC
Solr defaults q.op=OR; if user sets AND, OpenSearch defaults must explicitly match. OpenSearch's default operator on query_string is OR.
Detect: Solr source with <q.op>AND</q.op> or eDisMax with q.op=AND in solrconfig.xml
Fix: Set default_operator: AND on query_string, OR operator: AND on match. Most common cause of result divergence.
12. fielddata: true on text fields will OOM data nodes
Category: MIGRATION_SPECIFIC
Pre-ES 2.0, text fields used in-memory fielddata for sort/agg. ES 1.x mappings still carry "fielddata": true and will OOM AOS data nodes on first aggregation.
Detect: Source = ES 1.x or 2.x; mapping JSON contains fielddata
Fix: Strip fielddata. Add a .keyword subfield: "title": {"type":"text", "fields": {"keyword": {"type":"keyword"}}}. Migration Assistant for Amazon OpenSearch Service transformer does this automatically; hand-rolled _reindex MUST do it explicitly.
13. ES 7 → OS 1 _type removal
Category: MIGRATION_SPECIFIC
ES 7 still allows the placeholder type _doc; OS 1.0 removed types entirely. Templates with "_doc": {...} blow up _reindex/_bulk with [mapper_parsing_exception] unsupported parameters: [_doc].
Detect: ES 7 source with index templates
Fix: Migration Assistant for Amazon OpenSearch Service metadata transformer, OR pre-flatten with jq 'del(.mappings._doc) | .mappings = .mappings._doc' template.json.
14. NAT Gateway charges silently inflate VPC OpenSearch bills
Category: COST_TCO
A private cluster fetching plugins, Bedrock embeddings, IDP metadata, or external knowledge sources accumulates NAT-Gateway charges. NAT Gateway charges per VPC pricing.
Detect: Private VPC cluster with external integrations Fix: Use VPC endpoints for S3, Bedrock, STS, OpenSearch Service. Project residual NAT egress per VPC pricing.
15. Manual snapshots bill against YOUR S3 bucket
Category: COST_TCO
AOS automated snapshots: kept 14 days (hourly, up to 336), no additional charge, in AOS-preconfigured bucket. Manual snapshots: stored in YOUR S3 bucket at standard S3 rates plus PUT charges.
Detect: Compliance retention > 14 days; cross-region snapshot requirements
Fix: Add S3 line to sizing model: data_size × retention_days / 30 × $/GB-mo plus PUT cost.
16. UltraWarm uw.medium cannot host k-NN indexes
Category: TRUE_BLOCKER
The instance lacks RAM headroom to hold k-NN graphs.
Detect: k-NN indexes scheduled for UltraWarm migration on uw.medium
Fix: Use ultrawarm1.large.search instead. For current UltraWarm RAM-per-instance figures and circuit-breaker sizing, see ultrawarm.html.
17. OR1 trades RAM-bound aggregations for indexing throughput
Category: OPERATIONAL_CONSIDERATION
OR1 stores segments in S3 with local NVMe cache. ~2× r6g indexing throughput, replica=1 sufficient (S3 durable). Loses to r-family on cache-miss aggregations and k-NN graphs (RAM-bound).
Detect: k-NN, large-cardinality aggs, or cache-miss-sensitive workloads on OR1
Fix: Use OR1 only when peak_indexing × avg_doc_size > 50 GB/day/node. Use one replica unless durability model demands more. Migration to OR1 is irreversible.
18. Cluster goes read-only at flood-stage watermark (95%)
Category: OPERATIONAL_CONSIDERATION
When any node hits 95% disk, AOS applies index.blocks.read_only_allow_delete: true to all indexes with shards on that node. Releases automatically when below high (90%).
Detect: Cluster size near 90%; observability indexes growing fast
Fix: Alert on FreeStorageSpace < 25 GB or storage > 80%. Add storage / shrink shards / move data to UltraWarm BEFORE this hits.
19. Multi-AZ ≠ Multi-AZ with Standby
Category: CLARIFICATION
Multi-AZ: 99.9% SLA. Multi-AZ with Standby: 99.99% SLA. Standby pre-positions one zone as inactive, sub-minute failover. Standby requirements: 3 AZs, 3 dedicated cluster managers, 3 (or multiple of 3) data nodes, ≥2 replicas, Auto-Tune ON, GP3 storage.
Detect: Customer expects "no downtime ever" without Standby Fix: Recommend Multi-AZ-with-Standby for tier-1 production. Standby is "available at no extra cost" but applies caps on per-shard size and total cluster shard count. For current Standby caps, see managedomains-multiaz.html.
20. Logstash default distro license check rejects OpenSearch
Category: MIGRATION_SPECIFIC
Default Logstash distro has Elastic license check that rejects OpenSearch. Two workarounds:
Detect: Customer using Logstash with new Amazon OpenSearch destination
Fix: Use OSS distro of Logstash (Apache 2.0) OR logstash-output-opensearch plugin. Better: switch to OpenSearch Ingestion (managed Data Prepper) or Fluent Bit.
21. Cross-AZ data transfer is FREE within AOS clusters
Category: COST_TCO
Self-managed Elasticsearch on EC2 across AZs pays cross-AZ data-transfer at the standard regional rate for primary→replica replication. Amazon OpenSearch Service does NOT bill for intra-cluster cross-AZ replication.
Detect: Customer's current TCO model includes a cross-AZ line item for self-managed ES replication Fix: Call this out as a savings the migration unlocks. Cross-AZ data transfer between customer-owned resources (e.g., app tier ↔ AOS endpoint, or NAT Gateway egress) is still billed normally.
22. AOS-managed gp3 storage is priced separately from raw EBS gp3
Category: COST_TCO
The exact AOS-managed gp3 list price (volume + baseline IOPS + service overhead) is published on the AOS pricing page, NOT the raw EBS rate. TCO calculators reusing raw EBS underestimate.
Detect: Customer-built TCO calculator uses raw EBS rates
Fix: Plug AOS-managed gp3 rate from https://calculator.aws into customer's TCO model. RI / Savings Plan / EDP discounts apply only there.
23. Cluster manager sizing scales with cluster size
Category: OPERATIONAL_CONSIDERATION
Architectural rule: 3 dedicated cluster managers (formerly "master node"), odd quorum. NEVER 1, 2, 4, or 5.
Detect: Cluster scaling beyond 30 nodes; shard count growth
Fix: For current cluster-manager sizing (heap-to-nodes / shard tier), see sizing.md §Topology defaults.
24. Migration from Managed → Serverless requires reindex
Category: MIGRATION_SPECIFIC
There is NO automatic migration from Managed Domain to Serverless. Must reindex.
Detect: Customer wants "easy switch" from Managed to Serverless
Fix: Plan a reindex migration. Use Migration Assistant for Amazon OpenSearch Service or _reindex from remote. Validate sizing on Serverless before cutover.
25. Authentication complexity is the #1 setup blocker
Category: OPERATIONAL_CONSIDERATION
Forum data: 60%+ of new-user issues are auth-related. FGAC + IAM + Cognito + SAML + master-user combinations have many failure modes.
Detect: Any auth question; first-time AOS user
Fix: See security.md for the FGAC + IAM + Cognito + SAML decision tree. Common pattern:
- Internal users only → IAM SigV4 from app
- External / human users → Cognito user pool + FGAC mapped to Cognito groups
- Enterprise SSO → SAML to FGAC backend role mapping
26. ELSER is proprietary to Elastic — not on Amazon OpenSearch
Category: TRUE_BLOCKER
Don't promise ELSER on AOS. Use neural sparse search with SageMaker-hosted SPLADE/equivalent, or dense vectors via Bedrock Titan / Cohere.
Detect: Customer asks for ELSER on AOS
Fix: Recommend neural_sparse query with SageMaker-hosted sparse encoder, OR hybrid (BM25 + dense vectors). Most ELSER use cases work fine with hybrid.
27. Painless scripts not supported on Serverless
Category: TRUE_BLOCKER
Inline scripts work on Managed but not Serverless. If customer relies on script_score, script_fields, or update-by-script, they need Managed.
Detect: Customer mentions Painless / script_score / scripted fields with Serverless target
Fix: Move to Managed, OR rewrite scripted logic into ingest pipeline / search pipeline / function_score.
28. ES Runtime fields have only partial parity in OpenSearch
Category: TRUE_BLOCKER
OpenSearch added "derived fields" in 2.15 — limited functionality compared to ES Runtime fields. Not full parity.
Detect: ES source heavily uses Runtime fields; OS target Fix: For each Runtime field, decide: (a) pre-compute at ingest, (b) use derived fields if simple, or (c) move logic to query-time scripted fields (Managed only).
29. ILM JSON does NOT import as ISM
Category: MIGRATION_SPECIFIC
Elasticsearch ILM and OpenSearch ISM are conceptually similar but JSON formats differ. Must rebuild policies.
Detect: Customer has many ILM policies and assumes they "just work" on OS
Fix: Translate each ILM policy to ISM. Common patterns: rollover, force_merge, warm/cold migration, delete. AWS-specific ISM operations: warm_migration, cold_migration, cold_delete.
30. AOS automated snapshots are NOT a backup strategy
Category: OPERATIONAL_CONSIDERATION
See #15 (canonical) — automated snapshots are kept only 14 days and are not a DR strategy.
Detect: Customer plans to "use automated snapshots for DR" Fix: See #15. Set up manual snapshots to your own S3 bucket with appropriate retention; build a cross-region snapshot strategy if DR is in scope.
31. FAISS HNSW IS supported on Serverless Vector Search
Category: CLARIFICATION
Architectural rule: FAISS HNSW is the underlying engine on BOTH Serverless Vector Search collection types (NextGen and Classic). The difference is configurability, not support. Saying 'FAISS HNSW is unavailable on Serverless' is WRONG.
For the per-config breakdown of NextGen vs Classic Vector Search (which engines/parameters each surfaces, what pins a workload to Managed Domain), see vector-knn.md.
Detect: Customer claims FAISS HNSW is unavailable on Serverless; vector workload routing decision
Fix: Affirm FAISS HNSW availability on both Serverless Vector Search variants. Use vector-knn.md to decide whether the workload pins to Managed Domain.
32. OS 1.x version line
Category: CLARIFICATION
There is NO OS 1.7 GA release. OS 1.x had GA releases up through 1.3. For the current canonical version list, see version-migration.html.
If a customer says 'OS 1.7' they likely mean:
- Elasticsearch 1.7 (different product, pre-fork era), OR
- Misremembered OS 1.3 (the latest 1.x), OR
- Confusion with a 2.x or 3.x version
Clarify before proceeding with upgrade plan.
Detect: Customer cites "OS 1.7" or any OS 1.x version above 1.3 Fix: Confirm the actual source version (ES 1.7 vs OS 1.3 vs OS 2.x/3.x) before scoping the upgrade. The Lucene-segment-format wall (#3) and other version-specific gotchas hinge on knowing the true source.
