Index of every page on this site: llms.txt
All skills
davila7 avatar

/inference-serving-llama-cpp

@053feb2
Run it yourself

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

Use this Skill: https://skilld.dev/gh/davila7/claude-code-templates/inference-serving-llama-cpp

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesquantization.md

≈1.2k tokens on demand. Your agent reads this file only when SKILL.md points to it.

GGUF Quantization Guide

Complete guide to GGUF quantization formats and model conversion.

Quantization Overview

GGUF (GPT-Generated Unified Format) - Standard format for llama.cpp models.

Format Comparison

Format Perplexity Size (7B) Tokens/sec Notes
FP16 5.9565 (baseline) 13.0 GB 15 tok/s Original quality
Q8_0 5.9584 (+0.03%) 7.0 GB 25 tok/s Nearly lossless
Q6_K 5.9642 (+0.13%) 5.5 GB 30 tok/s Best quality/size
Q5_K_M 5.9796 (+0.39%) 4.8 GB 35 tok/s Balanced
Q4_K_M 6.0565 (+1.68%) 4.1 GB 40 tok/s Recommended
Q4_K_S 6.1125 (+2.62%) 3.9 GB 42 tok/s Faster, lower quality
Q3_K_M 6.3184 (+6.07%) 3.3 GB 45 tok/s Small models only
Q2_K 6.8673 (+15.3%) 2.7 GB 50 tok/s Not recommended

Recommendation: Use Q4_K_M for best balance of quality and speed.

Converting Models

HuggingFace to GGUF

# 1. Download HuggingFace model
huggingface-cli download meta-llama/Llama-2-7b-chat-hf \
    --local-dir models/llama-2-7b-chat/

# 2. Convert to FP16 GGUF
python convert_hf_to_gguf.py \
    models/llama-2-7b-chat/ \
    --outtype f16 \
    --outfile models/llama-2-7b-chat-f16.gguf

# 3. Quantize to Q4_K_M
./llama-quantize \
    models/llama-2-7b-chat-f16.gguf \
    models/llama-2-7b-chat-Q4_K_M.gguf \
    Q4_K_M

Batch quantization

# Quantize to multiple formats
for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
    ./llama-quantize \
        model-f16.gguf \
        model-${quant}.gguf \
        $quant
done

K-Quantization Methods

K-quants use mixed precision for better quality:

  • Attention weights: Higher precision
  • Feed-forward weights: Lower precision

Variants:

  • _S (Small): Faster, lower quality
  • _M (Medium): Balanced (recommended)
  • _L (Large): Better quality, larger size

Example: Q4_K_M

  • Q4: 4-bit quantization
  • K: Mixed precision method
  • M: Medium quality

Quality Testing

# Calculate perplexity (quality metric)
./llama-perplexity \
    -m model.gguf \
    -f wikitext-2-raw/wiki.test.raw \
    -c 512

# Lower perplexity = better quality
# Baseline (FP16): ~5.96
# Q4_K_M: ~6.06 (+1.7%)
# Q2_K: ~6.87 (+15.3% - too much degradation)

Use Case Guide

General purpose (chatbots, assistants)

Q4_K_M - Best balance
Q5_K_M - If you have extra RAM

Code generation

Q5_K_M or Q6_K - Higher precision helps with code

Creative writing

Q4_K_M - Sufficient quality
Q3_K_M - Acceptable for draft generation

Technical/medical

Q6_K or Q8_0 - Maximum accuracy

Edge devices (Raspberry Pi)

Q2_K or Q3_K_S - Fit in limited RAM

Model Size Scaling

7B parameter models

Format Size RAM needed
Q2_K 2.7 GB 5 GB
Q3_K_M 3.3 GB 6 GB
Q4_K_M 4.1 GB 7 GB
Q5_K_M 4.8 GB 8 GB
Q6_K 5.5 GB 9 GB
Q8_0 7.0 GB 11 GB

13B parameter models

Format Size RAM needed
Q2_K 5.1 GB 8 GB
Q3_K_M 6.2 GB 10 GB
Q4_K_M 7.9 GB 12 GB
Q5_K_M 9.2 GB 14 GB
Q6_K 10.7 GB 16 GB

70B parameter models

Format Size RAM needed
Q2_K 26 GB 32 GB
Q3_K_M 32 GB 40 GB
Q4_K_M 41 GB 48 GB
Q4_K_S 39 GB 46 GB
Q5_K_M 48 GB 56 GB

Recommendation for 70B: Use Q3_K_M or Q4_K_S to fit in consumer hardware.

Finding Pre-Quantized Models

TheBloke on HuggingFace:

Example:

# Download pre-quantized Llama 2-7B
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

Importance Matrices (imatrix)

What: Calibration data to improve quantization quality.

Benefits:

  • 10-20% perplexity improvement with Q4
  • Essential for Q3 and below

Usage:

# 1. Generate importance matrix
./llama-imatrix \
    -m model-f16.gguf \
    -f calibration-data.txt \
    -o model.imatrix

# 2. Quantize with imatrix
./llama-quantize \
    --imatrix model.imatrix \
    model-f16.gguf \
    model-Q4_K_M.gguf \
    Q4_K_M

Calibration data:

  • Use domain-specific text (e.g., code for code models)
  • ~100MB of representative text
  • Higher quality data = better quantization

Troubleshooting

Model outputs gibberish:

  • Quantization too aggressive (Q2_K)
  • Try Q4_K_M or Q5_K_M
  • Verify model converted correctly

Out of memory:

  • Use lower quantization (Q4_K_S instead of Q5_K_M)
  • Offload fewer layers to GPU (-ngl)
  • Use smaller context (-c 2048)

Slow inference:

  • Higher quantization uses more compute
  • Q8_0 much slower than Q4_K_M
  • Consider speed vs quality trade-off

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning18d3 checks · Risk SAFE
  • Gen Agent Trust Hub18d

    This skill serves as a comprehensive technical guide for using the llama.cpp project to run LLM inference on local hardware. It includes instructions for installation from trusted sources, model quantization using GGUF formats, and deploying an OpenAI-compatible server. The skill follows industry standard practices and contains no malicious code or suspicious patterns.

  • Socket18d

    No alerts

  • Snyk18d

    Risk: MEDIUM · 1 issue

Signed by skilld at 053feb2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
Other metadata
metadata
{
  "version": "1.0.0",
  "author": "Orchestra Research",
  "tags": [
    "Inference Serving",
    "Llama.cpp",
    "CPU Inference",
    "Apple Silicon",
    "Edge Deployment",
    "GGUF",
    "Quantization",
    "Non-NVIDIA",
    "AMD GPUs",
    "Intel GPUs",
    "Embedded"
  ],
  "dependencies": [
    "llama-cpp-python"
  ]
}

README badge

README badge for davila7/claude-code-templates/inference-serving-llama-cpp