hf-mem
uvx hf-mem <model> -o json [options]<model> is a Hub ID (org/name, org/name@revision), a Hub URL, a local path, or org/name-GGUF:Q4_K_M for one GGUF quantization.
Options:
--gpu-memory 24(GiB per GPU) or8x80(8 GPUs): addsfit.--max-model-len N: context length, defaults to the model's maximum, which can dominate the total.--batch-size N: concurrent sequences, defaults to 1.--kv-cache-dtype fp8(vLLM names for Safetensors, llama.cpp cache types such asq8_0for GGUF).--file <file or variant>and--format safetensors|gguf|onnxto pick the weights.--max-num-batched-tokens 2048on A100s.- Gated or private models need
HF_TOKENor--hf-token.
JSON (sizes in bytes, divide by 2^30 for GiB):
totalis the GPU memory for one copy of the model:weights+kv_cache+overhead(vLLM for Safetensors, llama.cpp for GGUF, none for ONNX).offloadedare weights the engine keeps in CPU memory: excluded fromweightsandtotal, but needed as free system RAM.fit(with--gpu-memory):fits,neededGPUs,max_model_len(the longest context that fits), and therequiredvsusablememory across the given GPUs. Userequired/ GPUs for the memory per GPU, nottotal/ GPUs, as each GPU also holds its own overhead and replicated KV cache.- GGUF and ONNX repositories with several files return a
fileslist to compare quantizations; pick one with:TAGor--fileto getfit. - Report
warningsto the user, and errors explain how to fix them.