Index of every page on this site: llms.txt
All skills
proyecto26 avatar

/distributed-search

@228d57d
Run it yourself

This skill should be used when the user designs a "search system", needs "full-text search", asks about an "inverted index", "Elasticsearch / OpenSearch", "relevance ranking" (TF-IDF/BM25), "search autocomplete / typeahead", an "indexing pipeline", or "faceted search". It gives the crawl/index/search architecture, index sharding and replication, ranking, and near-real-time indexing. Use it whenever users must query text by relevance rather than fetch rows by key, even if they don't say "search engine".

Use this Skill: https://skilld.dev/gh/proyecto26/system-design-skills/distributed-search

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovidersgeneric.md

≈614 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Distributed search — generic / self-hosted

The vendor-neutral default. When no cloud is named, this is the answer.

What to run

  • Elasticsearch / OpenSearch — distributed search engines over Lucene; REST API, JSON docs, sharding + replication, near-real-time refresh, aggregations (facets), BM25 by default. The default full-text choice. OpenSearch is the Apache-2.0 fork; pick on licensing/feature needs.
  • Apache Lucene — the underlying library (inverted index, segments, BM25). Embed it directly only when you want a single-node, in-process index and no cluster operations.
  • Apache Solr — also Lucene-based; mature faceting and config-driven schemas. A reasonable alternative to Elasticsearch for classic enterprise search.

Map the SKILL's options: build mode = refresh interval + bulk vs streaming indexing; ranking = default BM25, add function-score/boosts for hybrid; autocomplete = completion suggester (FST) or edge-n-gram field; distribution = shard count at index creation + replica count per shard.

Topology

  • Single node for small corpora / dev.
  • Multi-node cluster with dedicated roles for scale: master-eligible (cluster state), data nodes (shards), and coordinating nodes (fan-out/merge). Isolate indexing-heavy and query-heavy load onto separate node pools when they contend.
  • Shards per index fixed at creation; replicas adjustable. Size shards from corpus bytes up front (reindex to change shard count).

Limits / things that bite (verify against current docs)

  • Shard count is effectively immutable post-creation — under/over-sharding both hurt; plan from index bytes ÷ target shard size.
  • JVM heap pressure and large GC pauses on a data node spike query tail latency across every fan-out query.
  • Deep from/size pagination is expensive cluster-wide; use search_after / scroll / point-in-time cursors.
  • Refresh interval trades freshness for write cost; merges contend with queries.

Pitfalls

  • Treating it as a primary store — no transactions; it's a derived, rebuildable index. Keep the source of truth elsewhere and own a full-reindex path.
  • Mismatched index-time vs query-time analyzers → silent zero-result queries.
  • Running unthrottled bulk reindex against a live cluster (merge storm).
  • No durable buffer in front of the indexer, so a write surge stalls or drops.
  • You operate it: capacity, upgrades, snapshots, security (TLS/auth) are on you.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts24d3 checks · Risk SAFE
  • Gen Agent Trust Hub24d

    This skill is a technical architecture guide for designing distributed search systems and contains no security risks. It consists entirely of documentation and design patterns.

  • Socket24d

    No alerts

  • Snyk24d

    Risk: LOW · No issues

Signed by skilld at 228d57d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 weeks ago.

Activeupdated 4 weeks ago

README badge

README badge for proyecto26/system-design-skills/distributed-search