vector-database-ops by bagelhole | skilld

[Skip to main content](#main-content)

[skilld](https://skilld.dev/)

[Skills](https://skilld.dev/skills) [Collections](https://skilld.dev/collections) [People](https://skilld.dev/people)

[GitHub repository (opens in new tab)](https://github.com/harlan-zw/skilld)

[All skills](https://skilld.dev/skills)

[![bagelhole avatar](https://github.com/bagelhole.png?size=96)bagelhole profile](https://skilld.dev/gh/bagelhole)

# vector-database-ops

[bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills)

Agent-ready DevOps, security, infrastructure, and compliance knowledge base with 80+ skills across Kubernetes, Terraform, AWS/Azure/GCP, AI platform operations, container hardening, SOC2/ISO27001, and incident response—plus ready-to-run scripts, templates, and playbooks for SRE, platform, and security teams.

Community skill from bagelhole, source updated last month.

2 19 3 Updated last month First seen 3 months agosteady·No curators yetSign in to curate

## Install

skilld

skills.sh

`npx -y skilld add gh:bagelhole/devops-security-agent-skills -s vector-database-ops`

Works with Claude Code · Codex · Cursor · Copilot · Gemini CLI

[GitHub](https://github.com/bagelhole/devops-security-agent-skills) [skills.sh](https://skills.sh/bagelhole/vector-database-ops) [Raw](https://skilld.dev/api/skills-raw/bagelhole/devops-security-agent-skills/vector-database-ops)

## Skill content

Copy as markdown

Preview

Markdown

# Vector Database Operations

Run production vector databases for AI-powered search, RAG, and recommendation systems.

## When to Use This Skill

Use this skill when:

- Setting up a vector database for a RAG or semantic search application
- Choosing between Qdrant, Weaviate, pgvector, or Pinecone
- Managing collections, indexes, and data migrations
- Optimizing query performance and indexing for production loads
- Implementing multi-tenant vector search with namespace isolation

## Vector Database Comparison

| Database | Best For | Hosting | Filtering | Scale |
| --- | --- | --- | --- | --- |
| **Qdrant** | High-performance, rich filtering, self-hosted | Self / Cloud | Excellent | Very High |
| **Weaviate** | Schema-first, hybrid search, multi-modal | Self / Cloud | Good | High |
| **pgvector** | Already on Postgres, simple use cases | Self | Good | Medium |
| **Pinecone** | Zero-ops managed, serverless | Managed only | Good | Very High |
| **Chroma** | Local dev, prototyping | Self only | Basic | Low-Medium |

## Qdrant — Production Deployment

```bash
# Docker (single node)
docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -p 6334:6334 \
  -v $(pwd)/qdrant-data:/qdrant/storage \
  qdrant/qdrant:latest

# With custom config
docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -v $(pwd)/qdrant-data:/qdrant/storage \
  -v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml \
  qdrant/qdrant:latest
```

```yaml
# qdrant-config.yaml
storage:
  storage_path: /qdrant/storage
  on_disk_payload: true          # store payload on disk (saves RAM)

service:
  max_request_size_mb: 32

hnsw_index:
  m: 16                          # graph connections per node
  ef_construct: 100              # accuracy vs build time trade-off
  full_scan_threshold: 10000     # switch to brute force below this

quantization:
  scalar:
    type: int8
    quantile: 0.99
    always_ram: true             # keep quantized index in RAM

telemetry_disabled: true
```

## Qdrant Collection Management

```python
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
)

client = QdrantClient("http://localhost:6333")

# Create optimized collection
client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(
        size=1536,                         # OpenAI ada-002 / text-embedding-3-small
        distance=Distance.COSINE,
        on_disk=True,                      # save RAM — vectors stored on disk
    ),
    hnsw_config=HnswConfigDiff(
        m=32,                              # higher = better recall, more RAM
        ef_construct=200,
        on_disk=False,                     # keep HNSW graph in RAM for speed
    ),
    quantization_config=QuantizationConfig(
        scalar=ScalarQuantizationConfig(
            type=ScalarType.INT8,
            quantile=0.99,
            always_ram=True,
        )
    ),
)

# Create payload index for fast filtering
client.create_payload_index(
    collection_name="documents",
    field_name="tenant_id",
    field_schema="keyword",
)
client.create_payload_index(
    collection_name="documents",
    field_name="created_at",
    field_schema="datetime",
)

# Collection info
info = client.get_collection("documents")
print(f"Vectors: {info.vectors_count}, Status: {info.status}")
```

## Qdrant Filtered Search

```python
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range

# Tenant-isolated search (multi-tenant RAG)
results = client.query_points(
    collection_name="documents",
    query=query_embedding,
    query_filter=Filter(
        must=[
            FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")),
            FieldCondition(key="doc_type", match=MatchValue(value="contract")),
        ],
        should=[
            FieldCondition(key="created_at", range=Range(gte="2024-01-01")),
        ],
    ),
    limit=10,
    with_payload=True,
)
```

## pgvector — PostgreSQL Extension

```sql
-- Enable extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- Create HNSW index (faster queries, more memory)
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- Create IVFFlat index (less memory, slower build)
-- CREATE INDEX ON documents
-- USING ivfflat (embedding vector_cosine_ops)
-- WITH (lists = 100);

-- Semantic search with metadata filtering
SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
ORDER BY embedding <=> $1::vector
LIMIT 10;
```

```bash
# Deploy pgvector via Docker
docker run -d \
  --name pgvector \
  -e POSTGRES_PASSWORD=secret \
  -e POSTGRES_DB=vectordb \
  -p 5432:5432 \
  -v pgvector-data:/var/lib/postgresql/data \
  pgvector/pgvector:pg16
```

## Weaviate Deployment

```yaml
# docker-compose for Weaviate
services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false"
      AUTHENTICATION_APIKEY_ENABLED: "true"
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}"
      AUTHENTICATION_APIKEY_USERS: "admin"
      PERSISTENCE_DATA_PATH: /var/lib/weaviate
      ENABLE_MODULES: text2vec-openai,generative-openai
      OPENAI_APIKEY: "${OPENAI_API_KEY}"
      CLUSTER_HOSTNAME: node1
    volumes:
      - weaviate-data:/var/lib/weaviate
    restart: unless-stopped

volumes:
  weaviate-data:
```

## Backup and Restore

```bash
# Qdrant — snapshot backup
curl -X POST "http://localhost:6333/collections/documents/snapshots"
# Download snapshot
curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"
# Restore
curl -X POST "http://localhost:6333/collections/documents/snapshots/recover" \
  -H "Content-Type: application/json" \
  -d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'

# pgvector — standard pg_dump
pg_dump -h localhost -U postgres -d vectordb \
  --table=documents --format=custom > documents-backup.dump

# Restore
pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
```

## Performance Tuning

```python
# Qdrant — optimize collection after bulk load
client.update_collection(
    collection_name="documents",
    optimizer_config={"indexing_threshold": 0},  # force indexing now
)

# Wait for optimization to complete
import time
while True:
    info = client.get_collection("documents")
    if info.status.value == "green":
        break
    time.sleep(5)
    print(f"Optimizing... segments: {info.segments_count}")
```

## Common Issues

| Issue | Cause | Fix |
| --- | --- | --- |
| Slow queries | No HNSW index built yet | Wait for indexing; check `status == green` |
| High RAM usage | Vectors in memory | Enable `on_disk=True` for vectors |
| Poor recall | Low `ef` search param | Increase `ef` in search request (at query time) |
| pgvector slow | Using IVFFlat without vacuum | Run `VACUUM ANALYZE documents` |
| Weaviate OOM | Too many objects | Enable async indexing; increase heap |

## Best Practices

- Use cosine distance for normalized embeddings; dot product for unnormalized.
- Always create payload indexes on filter fields ( `tenant_id`, `doc_type`).
- For datasets >10M vectors, use `on_disk` vectors + `always_ram` quantization.
- Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
- Snapshot before any bulk delete or migration operation.

## Related Skills

- [rag-infrastructure](https://skilld.dev/../../local-ai/rag-infrastructure/) - Full RAG pipeline
- [databases](https://skilld.dev/../databases/) - General database management
- [postgresql](https://skilld.dev/../postgresql/) - pgvector host database ops

Source: [SKILL.md on GitHub](https://github.com/bagelhole/devops-security-agent-skills/blob/dd77232b16c4f6445c51347eb516c3e98ac9c8d4/infrastructure/databases/vector-database-ops/SKILL.md)

## Why curators picked this

No curator note yet. [Be the first to add yours](https://skilld.dev/collections/new?skill=vector-database-ops&skillsOwner=bagelhole&skillsRepo=devops-security-agent-skills) — one line on why you reach for this skill.

## Install

skilld

skills.sh

`npx -y skilld add gh:bagelhole/devops-security-agent-skills -s vector-database-ops`

Works with Claude Code · Codex · Cursor · Copilot · Gemini CLI

[GitHub](https://github.com/bagelhole/devops-security-agent-skills) [skills.sh](https://skills.sh/bagelhole/vector-database-ops) [Raw](https://skilld.dev/api/skills-raw/bagelhole/devops-security-agent-skills/vector-database-ops)

## Metadata

<dl>

<dt>Description</dt>
<dd>309 chars · repository</dd>

<dt>Frontmatter</dt>
<dd>4 keys</dd></dl>

## Capability

<details>

<summary>Other metadata</summary>



<dl>

<dt>license</dt>
<dd>MIT</dd>

</dl></details>



## Receipts

Indexed from [github.com/bagelhole/devops-security-agent-skills](https://github.com/bagelhole/devops-security-agent-skills) on branch `main`.

<dl>

<dt>Commit</dt>
<dd>[dd77232](https://github.com/bagelhole/devops-security-agent-skills/commit/dd77232b16c4f6445c51347eb516c3e98ac9c8d4 "dd77232b16c4f6445c51347eb516c3e98ac9c8d4")</dd>

<dt>SKILL.md</dt>
<dd>[infrastructure/databases/vector-database-ops/SKILL.md](https://github.com/bagelhole/devops-security-agent-skills/blob/dd77232b16c4f6445c51347eb516c3e98ac9c8d4/infrastructure/databases/vector-database-ops/SKILL.md)</dd>

<dt>Last modified</dt>
<dd>2 months ago</dd>

<dt>History</dt>
<dd>[View commits](https://github.com/bagelhole/devops-security-agent-skills/commits/main/infrastructure/databases/vector-database-ops/SKILL.md)</dd>

</dl>

Verified 2 days ago (stale)

## Related skills

From bagelhole/devops-security-agent-skills

[![bagelhole avatar](https://github.com/bagelhole.png?size=48) linux-administration bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/linux-administration) [![bagelhole avatar](https://github.com/bagelhole.png?size=48) sops-encryption bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/sops-encryption) [![bagelhole avatar](https://github.com/bagelhole.png?size=48) vpn-setup bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/vpn-setup) [![bagelhole avatar](https://github.com/bagelhole.png?size=48) reverse-proxy bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/reverse-proxy) [![bagelhole avatar](https://github.com/bagelhole.png?size=48) hashicorp-vault bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/hashicorp-vault) [![bagelhole avatar](https://github.com/bagelhole.png?size=48) kubernetes-ops bagelhole/devops-security-agent-skills](https://skilld.dev/gh/bagelhole/devops-security-agent-skills/kubernetes-ops)

[Stats](https://skilld.dev/skills/stats) [Accessibility](https://skilld.dev/accessibility)

[GitHub repository (opens in new tab)](https://github.com/harlan-zw/skilld)

Built by [Harlan Wilton](https://harlanzw.com)