Methodology

Weights

weights_GB = params × bits_per_weight ÷ 8 ÷ 1e9

Bits per weight are the approximate effective sizes of llama.cpp GGUF formats, including block scale overhead: Q4_K_M ≈ 4.85, Q5_K_M ≈ 5.69, Q6_K ≈ 6.56, Q8_0 = 8.5, F16 = 16. Real GGUF files vary slightly because some tensors (embeddings, output) use different types. Source: llama.cpp quantization docs, github.com/ggml-org/llama.cpp.

KV cache

kv_GB = 2 × layers × kv_heads × head_dim × context × bytes_per_element ÷ 1e9

The factor 2 covers keys and values. Models that use grouped-query attention store only num_key_value_heads. F16 = 2 bytes; Q8_0 ≈ 1.0625 bytes.

Overhead

We add 0.5 GB plus 10% of weights for the compute buffer and runtime/driver overhead. This heuristic is rough, and actual overhead depends on the backend (CUDA, ROCm/HIP, Vulkan, Metal) and batch size.

Fit

"Fits" means total ≤ 90% of VRAM. "Tight" means 90–100%. Anything over 100% needs partial CPU offload, which is much slower.

Theoretical speed ceiling

Token generation is mostly memory-bandwidth bound: max_tok/s ≈ bandwidth_GB/s ÷ weights_GB. Treat it as an upper bound only. MoE models read only their active parameters per token, so this overestimates their weight reads; we do not adjust for that yet.

Sources

GPU VRAM and bandwidth come from the Wikipedia lists of Nvidia and AMD GPUs, retrieved 2026-10-15. Those tables in turn cite vendor spec sheets.