similarity-search-patterns by wshobson

Similarity Search Patterns

Patterns for implementing efficient similarity search in production systems.

When to Use This Skill

Building semantic search systems
Implementing RAG retrieval
Creating recommendation engines
Optimizing search latency
Scaling to millions of vectors
Combining semantic and keyword search

Core Concepts

1. Distance Metrics

| Metric | Formula | Best For | | ------------------ | ------------------ | --------------------- | --- | -------------- | | Cosine | 1 - (A·B)/(‖A‖‖B‖) | Normalized embeddings | | Euclidean (L2) | √Σ(a-b)² | Raw embeddings | | Dot Product | A·B | Magnitude matters | | Manhattan (L1) | Σ | a-b | | Sparse vectors |

2. Index Types

┌─────────────────────────────────────────────────┐
│                 Index Types                      │
├─────────────┬───────────────┬───────────────────┤
│    Flat     │     HNSW      │    IVF+PQ         │
│ (Exact)     │ (Graph-based) │ (Quantized)       │
├─────────────┼───────────────┼───────────────────┤
│ O(n) search │ O(log n)      │ O(√n)             │
│ 100% recall │ ~95-99%       │ ~90-95%           │
│ Small data  │ Medium-Large  │ Very Large        │
└─────────────┴───────────────┴───────────────────┘

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Best Practices

Do's

Use appropriate index - HNSW for most cases
Tune parameters - ef_search, nprobe for recall/speed
Implement hybrid search - Combine with keyword search
Monitor recall - Measure search quality
Pre-filter when possible - Reduce search space

Don'ts

Don't skip evaluation - Measure before optimizing
Don't over-index - Start with flat, scale up
Don't ignore latency - P99 matters for UX
Don't forget costs - Vector storage adds up

Third-party checks

No alerts16d5 checks · Risk SAFE

Gen Agent Trust Hub16d
The skill provides standard implementation templates and documentation for popular vector databases, including Pinecone, Qdrant, pgvector, and Weaviate. The code follows standard security practices for database interactions and uses well-known, legitimate libraries for search and machine learning tasks.
Socket16d
No alerts
Snyk16d
Risk: LOW · No issues
Runlayer6mo
1/1 file flagged
ZeroLeaks5mo
Score: 93/100 · 2 sections analyzed

Frequently asked

What vector databases does this skill cover?

The skill teaches patterns applicable across vector databases (Faiss, Pinecone, Weaviate, etc.) rather than being tied to a specific one. It focuses on distance metrics, index types, and retrieval strategies that work across platforms.

Does this skill include code templates?

Yes. Detailed worked examples and concrete templates are in `references/details.md`, which you should read when implementing a specific pattern.

What distance metric should I use?

Cosine distance is the default for normalized embeddings. Euclidean (L2) works for raw embeddings, dot product when magnitude matters, and Manhattan (L1) for sparse vectors.

When should I use HNSW vs flat vs IVF+PQ indexes?

Use flat for small datasets (exact O(n) search), HNSW for medium to large (O(log n) with 95-99% recall), and IVF+PQ for very large scale (O(√n) with 90-95% recall and lower memory).

Generated from the current SKILL.md. These answers refresh after source changes.

/similarity-search-patterns

SKILL.md