Index of every page on this site: llms.txt
All skills
davila7 avatar

/inference-serving-llama-cpp

@053feb2
Run it yourself

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

Use this Skill: https://skilld.dev/gh/davila7/claude-code-templates/inference-serving-llama-cpp

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesoptimization.md

≈415 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Performance Optimization Guide

Maximize llama.cpp inference speed and efficiency.

CPU Optimization

Thread tuning

# Set threads (default: physical cores)
./llama-cli -m model.gguf -t 8

# For AMD Ryzen 9 7950X (16 cores, 32 threads)
-t 16  # Best: physical cores

# Avoid hyperthreading (slower for matrix ops)

BLAS acceleration

# OpenBLAS (faster matrix ops)
make LLAMA_OPENBLAS=1

# BLAS gives 2-3× speedup

GPU Offloading

Layer offloading

# Offload 35 layers to GPU (hybrid mode)
./llama-cli -m model.gguf -ngl 35

# Offload all layers
./llama-cli -m model.gguf -ngl 999

# Find optimal value:
# Start with -ngl 999
# If OOM, reduce by 5 until fits

Memory usage

# Check VRAM usage
nvidia-smi dmon

# Reduce context if needed
./llama-cli -m model.gguf -c 2048  # 2K context instead of 4K

Batch Processing

# Increase batch size for throughput
./llama-cli -m model.gguf -b 512  # Default: 512

# Physical batch (GPU)
--ubatch 128  # Process 128 tokens at once

Context Management

# Default context (512 tokens)
-c 512

# Longer context (slower, more memory)
-c 4096

# Very long context (if model supports)
-c 32768

Benchmarks

CPU Performance (Llama 2-7B Q4_K_M)

Setup Speed Notes
Apple M3 Max 50 tok/s Metal acceleration
AMD 7950X (16c) 35 tok/s OpenBLAS
Intel i9-13900K 30 tok/s AVX2

GPU Offloading (RTX 4090)

Layers GPU Speed VRAM
0 (CPU only) 30 tok/s 0 GB
20 (hybrid) 80 tok/s 8 GB
35 (all) 120 tok/s 12 GB

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning18d3 checks · Risk SAFE
  • Gen Agent Trust Hub18d

    This skill serves as a comprehensive technical guide for using the llama.cpp project to run LLM inference on local hardware. It includes instructions for installation from trusted sources, model quantization using GGUF formats, and deploying an OpenAI-compatible server. The skill follows industry standard practices and contains no malicious code or suspicious patterns.

  • Socket18d

    No alerts

  • Snyk18d

    Risk: MEDIUM · 1 issue

Signed by skilld at 053feb2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
Other metadata
metadata
{
  "version": "1.0.0",
  "author": "Orchestra Research",
  "tags": [
    "Inference Serving",
    "Llama.cpp",
    "CPU Inference",
    "Apple Silicon",
    "Edge Deployment",
    "GGUF",
    "Quantization",
    "Non-NVIDIA",
    "AMD GPUs",
    "Intel GPUs",
    "Embedded"
  ],
  "dependencies": [
    "llama-cpp-python"
  ]
}

README badge

README badge for davila7/claude-code-templates/inference-serving-llama-cpp