Index of every page on this site: llms.txt
All skills
wshobson avatar

/similarity-search-patterns

@4220308
by Seth Hobsonwshobson/agents43k stars
4,518
Run it yourself

Implement efficient similarity search with vector databases. Use when building semantic search, implementing nearest neighbor queries, or optimizing retrieval performance.

Use this Skill: https://skilld.dev/gh/wshobson/agents/similarity-search-patterns

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈50 tokens for metadata: the name and description. ≈637 when used: this file. ≈3.9k more on demand in 1 file.

Description uses 2.5% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈200 characters, or 2.5%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Similarity Search Patterns

Patterns for implementing efficient similarity search in production systems.

When to Use This Skill

  • Building semantic search systems
  • Implementing RAG retrieval
  • Creating recommendation engines
  • Optimizing search latency
  • Scaling to millions of vectors
  • Combining semantic and keyword search

Core Concepts

1. Distance Metrics

| Metric | Formula | Best For | | ------------------ | ------------------ | --------------------- | --- | -------------- | | Cosine | 1 - (A·B)/(‖A‖‖B‖) | Normalized embeddings | | Euclidean (L2) | √Σ(a-b)² | Raw embeddings | | Dot Product | A·B | Magnitude matters | | Manhattan (L1) | Σ | a-b | | Sparse vectors |

2. Index Types

┌─────────────────────────────────────────────────┐
│                 Index Types                      │
├─────────────┬───────────────┬───────────────────┤
│    Flat     │     HNSW      │    IVF+PQ         │
│ (Exact)     │ (Graph-based) │ (Quantized)       │
├─────────────┼───────────────┼───────────────────┤
│ O(n) search │ O(log n)      │ O(√n)             │
│ 100% recall │ ~95-99%       │ ~90-95%           │
│ Small data  │ Medium-Large  │ Very Large        │
└─────────────┴───────────────┴───────────────────┘

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Best Practices

Do's

  • Use appropriate index - HNSW for most cases
  • Tune parameters - ef_search, nprobe for recall/speed
  • Implement hybrid search - Combine with keyword search
  • Monitor recall - Measure search quality
  • Pre-filter when possible - Reduce search space

Don'ts

  • Don't skip evaluation - Measure before optimizing
  • Don't over-index - Start with flat, scale up
  • Don't ignore latency - P99 matters for UX
  • Don't forget costs - Vector storage adds up

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo5 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides standard implementation templates and documentation for popular vector databases, including Pinecone, Qdrant, pgvector, and Weaviate. The code follows standard security practices for database interactions and uses well-known, legitimate libraries for search and machine learning tasks.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

  • Runlayer8mo

    1/1 file flagged

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 4220308. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 6 months ago
  • similarity-search
  • vector-databases
  • semantic-search
  • rag
  • nearest-neighbor
  • hnsw
  • embeddings
  • retrieval
  • recommendation-engines

README badge

README badge for wshobson/agents/similarity-search-patterns

Teaches patterns for efficient similarity search across vector databases, covering distance metrics, index types (flat, HNSW, IVF+PQ), and hybrid search strategies. Useful for building semantic search systems, RAG retrieval, and recommendation engines at scale.

Generated from the current SKILL.md.

What vector databases does this skill cover?
The skill teaches patterns applicable across vector databases (Faiss, Pinecone, Weaviate, etc.) rather than being tied to a specific one. It focuses on distance metrics, index types, and retrieval strategies that work across platforms.
Does this skill include code templates?
Yes. Detailed worked examples and concrete templates are in `references/details.md`, which you should read when implementing a specific pattern.
What distance metric should I use?
Cosine distance is the default for normalized embeddings. Euclidean (L2) works for raw embeddings, dot product when magnitude matters, and Manhattan (L1) for sparse vectors.
When should I use HNSW vs flat vs IVF+PQ indexes?
Use flat for small datasets (exact O(n) search), HNSW for medium to large (O(log n) with 95-99% recall), and IVF+PQ for very large scale (O(√n) with 90-95% recall and lower memory).

Generated from the current SKILL.md. These answers refresh after source changes.