Index of every page on this site: llms.txt
All skills
github avatar

/scaling-data-volume

@f10fdb8 official
by githubgithub/awesome-copilot41k stars
5,399
Run it yourself

Guides Qdrant data volume scaling decisions. Use when someone asks 'data doesn't fit on one node', 'too much data', 'need more storage', 'vertical or horizontal scaling', 'tenant scaling', 'time window rotation', or 'data growth exceeds capacity'.

  • 5 files
  • 16.1 KB
  • Updated 7 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/github/awesome-copilot/scaling-data-volume

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈67 tokens for metadata: the name and description. ≈399 when used: this file. ≈3.6k more on demand in 4 files.

Description uses 3.5% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈276 characters, or 3.5%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Scaling Data Volume

This document covers data volume scaling scenarios, where the total size of the dataset exceeds the capacity of a single node.

Tenant Scaling

If the use case is multi-tenant, meaning that each user only has access to a subset of the data, and we never need to query across all the data, then we can use multi-tenancy patterns to scale.

The recommended way is to use multi-tenant workloads with payload partitioning, per-tenant indexes, and tiered multitenancy.

Learn more Tenant Scaling

Sliding Time Window

Some use-cases are based on a sliding time window, where only the most recent data is relevant. For example an index for social media posts, where only the last 6 months of data require fast search.

Learn more Sliding Time Window

Global Search

Most general use-cases require global search across all data. In these situations, we might need to fall back to vertical scaling, and then horizontal scaling when we reach the limits of vertical scaling.

Vertical Scaling

When data doesn't fit in a single node, the first approach is to scale the node itself — more RAM, better disk, quantization, mmap. Exhaust vertical options before going horizontal, as horizontal scaling adds permanent operational complexity.

Learn more Vertical Scaling

Horizontal Scaling

When a single node can't hold the data even with quantization and mmap, distribute data across multiple nodes via sharding.

Learn more Horizontal Scaling

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at f10fdb8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 7 months ago
What it can do
Reads files
All 3 allowed tools
ReadGrepGlob
  • qdrant
  • scaling
  • vector-database
  • sharding
  • multi-tenancy
  • data-volume
  • vertical-scaling
  • horizontal-scaling

README badge

README badge for github/awesome-copilot/scaling-data-volume

Guides decisions for scaling Qdrant vector database when data exceeds single-node capacity. Covers tenant partitioning, time-window rotation, vertical scaling (RAM, quantization, mmap), and horizontal scaling via sharding.

Generated from the current SKILL.md.

When should I use tenant scaling versus horizontal scaling?
Use tenant scaling if each user only accesses a subset of data and you never query across all tenants. Use horizontal scaling for general use-cases that require global search across all data.
Should I scale vertically or horizontally first?
Exhaust vertical scaling options (more RAM, better disk, quantization, mmap) before going horizontal, since horizontal scaling adds permanent operational complexity.
What approach works for time-series or sliding window data?
If only recent data needs fast search (e.g. social media posts from the last 6 months), use sliding time window rotation to manage data growth without scaling the full dataset.

Generated from the current SKILL.md. These answers refresh after source changes.