Index of every page on this site: llms.txt
All skills
davila7 avatar

/inference-serving-llama-cpp

@053feb2
Run it yourself

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

Use this Skill: https://skilld.dev/gh/davila7/claude-code-templates/inference-serving-llama-cpp

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesserver.md

≈565 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Server Deployment Guide

Production deployment of llama.cpp server with OpenAI-compatible API.

Server Modes

llama-server

# Basic server
./llama-server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -c 4096  # Context size

# With GPU acceleration
./llama-server \
    -m models/llama-2-70b.Q4_K_M.gguf \
    -ngl 40  # Offload 40 layers to GPU

OpenAI-Compatible API

Chat completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-2",
    "messages": [
      {"role": "system", "content": "You are helpful"},
      {"role": "user", "content": "Hello"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

Streaming

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-2",
    "messages": [{"role": "user", "content": "Count to 10"}],
    "stream": true
  }'

Docker Deployment

Dockerfile:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y git build-essential
RUN git clone https://github.com/ggerganov/llama.cpp
WORKDIR /llama.cpp
RUN make LLAMA_CUDA=1
COPY models/ /models/
EXPOSE 8080
CMD ["./llama-server", "-m", "/models/model.gguf", "--host", "0.0.0.0", "--port", "8080"]

Run:

docker run --gpus all -p 8080:8080 llama-cpp:latest

Monitoring

# Server metrics endpoint
curl http://localhost:8080/metrics

# Health check
curl http://localhost:8080/health

Metrics:

  • requests_total
  • tokens_generated
  • prompt_tokens
  • completion_tokens
  • kv_cache_tokens

Load Balancing

NGINX:

upstream llama_cpp {
    server llama1:8080;
    server llama2:8080;
}

server {
    location / {
        proxy_pass http://llama_cpp;
        proxy_read_timeout 300s;
    }
}

Performance Tuning

Parallel requests:

./llama-server \
    -m model.gguf \
    -np 4  # 4 parallel slots

Continuous batching:

./llama-server \
    -m model.gguf \
    --cont-batching  # Enable continuous batching

Context caching:

./llama-server \
    -m model.gguf \
    --cache-prompt  # Cache processed prompts

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning18d3 checks · Risk SAFE
  • Gen Agent Trust Hub18d

    This skill serves as a comprehensive technical guide for using the llama.cpp project to run LLM inference on local hardware. It includes instructions for installation from trusted sources, model quantization using GGUF formats, and deploying an OpenAI-compatible server. The skill follows industry standard practices and contains no malicious code or suspicious patterns.

  • Socket18d

    No alerts

  • Snyk18d

    Risk: MEDIUM · 1 issue

Signed by skilld at 053feb2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 days ago
Other metadata
metadata
{
  "version": "1.0.0",
  "author": "Orchestra Research",
  "tags": [
    "Inference Serving",
    "Llama.cpp",
    "CPU Inference",
    "Apple Silicon",
    "Edge Deployment",
    "GGUF",
    "Quantization",
    "Non-NVIDIA",
    "AMD GPUs",
    "Intel GPUs",
    "Embedded"
  ],
  "dependencies": [
    "llama-cpp-python"
  ]
}

README badge

README badge for davila7/claude-code-templates/inference-serving-llama-cpp