Pull a llamacpp Model
When using a custom ROCm image such as kyuz0/amd-strix-halo-toolboxes:rocm-7.2.4 for llamacpp inference on AMD Strix Halo / gfx1151, Harbor's pull mechanism starts an ephemeral container with --n-gpu-layers 0; the custom image fails in that context without ROCm device access. Use the standard CPU image just for pulling, then restore.
Steps
1. Save the current image and switch to the standard CPU image
original_rocm_image="$(harbor config get llamacpp.image.rocm)"
harbor config set llamacpp.image.rocm ghcr.io/ggml-org/llama.cpp:server2. Pull the model
harbor pull <hf-owner/model-repo:quantization>
# Examples:
harbor pull bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
harbor pull unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF:UD-Q4_K_XLHarbor detects the HuggingFace model spec and routes it through llamacpp automatically.
3. Restore the custom image
harbor config set llamacpp.image.rocm "$original_rocm_image"Notes
- Run these steps in the same shell so
original_rocm_imagesurvives. Restore it even if the pull fails. - The restore step is mandatory — without it, llamacpp will not use the GPU on the next
harbor up llamacpp - Extra args (e.g.
llama-server -fa 1 -dio --no-mmap -ngl 99 --no-mmproj) are stored separately in config and are not affected - To verify the model after pulling:
harbor llamacpp model - To verify the image was restored:
harbor config get llamacpp.image.rocm