Index of every page on this site: llms.txt
All skills
huggingface avatar

/huggingface-local-models

@5291c2b official
by Hugging Facehuggingface/skills11k stars
797
Run it yourself

Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.

Use this Skill: https://skilld.dev/gh/huggingface/skills/huggingface-local-models

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈62 tokens for metadata: the name and description. ≈876 when used: this file. ≈3k more on demand in 3 files.

Description uses 3.1% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈249 characters, or 3.1%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Hugging Face Local Models

Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.

Default Workflow

  1. Search the Hub with apps=llama.cpp.
  2. Open https://huggingface.co/<repo>?local-app=llama.cpp.
  3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.
  4. Confirm exact .gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.
  5. Launch with llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.
  6. Fall back to --hf-repo plus --hf-file when the repo uses custom file naming.
  7. Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start

Install llama.cpp

brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

Authenticate for gated repos

hf auth login

Search the Hub

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

Run directly from the Hub

llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run an exact GGUF file

llama-server \
    --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
    --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -c 4096

Convert only when no GGUF is available

hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
    --outfile model-f16.gguf \
    --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Smoke test a local server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about exception handling"}
    ]
  }'

Quant Choice

  • Prefer the exact quant that HF marks as compatible on the ?local-app=llama.cpp page.
  • Keep repo-native labels such as UD-Q4_K_M instead of normalizing them.
  • Default to Q4_K_M unless the repo page or hardware profile suggests otherwise.
  • Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows.
  • Consider Q3_K_M, Q4_K_S, or repo-specific IQ / UD-* variants for tighter RAM or VRAM budgets.
  • Treat mmproj-*.gguf files as projector weights, not the main checkpoint.

Load References

  • Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.
  • Read quantization.md for format tables, model scaling, quality tradeoffs, and imatrix.
  • Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

Resources

  • llama.cpp: https://github.com/ggml-org/llama.cpp
  • Hugging Face GGUF + llama.cpp docs: https://huggingface.co/docs/hub/gguf-llamacpp
  • Hugging Face Local Apps docs: https://huggingface.co/docs/hub/main/local-apps
  • Hugging Face Local Agents docs: https://huggingface.co/docs/hub/agents-local
  • GGUF converter Space: https://huggingface.co/spaces/ggml-org/gguf-my-repo

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides comprehensive instructions for selecting, downloading, and running AI models locally using llama.cpp and Hugging Face. It utilizes standard community tools and trusted platforms for model distribution. While the workflow involves external downloads and command execution, these are essential components of the skill's intended functionality.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 5291c2b. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 7 months ago
  • llama.cpp
  • gguf
  • huggingface
  • quantization
  • local-inference
  • cpu
  • cuda
  • metal
  • rocm
  • model-serving

README badge

README badge for huggingface/skills/huggingface-local-models

Finds GGUF-quantized models on Hugging Face Hub compatible with llama.cpp, selects appropriate quantization levels, and launches them locally via llama-cli or llama-server with CPU, Metal, CUDA, or ROCm acceleration. Covers model discovery, quant selection, exact file lookup, format conversion, and OpenAI-compatible local serving.

Generated from the current SKILL.md.

Does this skill work with GPU acceleration?
Yes. The skill covers llama.cpp builds for Metal (Mac), CUDA (Nvidia), and ROCm (AMD), plus CPU inference. Hardware-specific setup is detailed in the bundled hardware.md reference.
What if a model repo doesn't have GGUF files pre-quantized?
The skill includes a conversion workflow: download Transformers weights with `hf download`, convert to GGUF using `convert_hf_to_gguf.py`, then quantize with `llama-quantize`.
How do I choose the right quantization level?
Start with the quant the Hugging Face local-app page recommends, default to Q4_K_M otherwise, and prefer Q5_K_M or Q6_K for code workloads if memory allows. The bundled quantization.md reference has format tables and tradeoff details.
Can I run the model as an OpenAI-compatible server?
Yes. Use `llama-server` instead of `llama-cli` to launch an OpenAI-compatible API on localhost:8080 that accepts `/v1/chat/completions` requests.
Does this work with gated models?
Yes, if you authenticate first with `hf auth login`. The skill assumes you have Hugging Face Hub access for the repo you want to run.

Generated from the current SKILL.md. These answers refresh after source changes.