Estimate the download size and disk space a model will need based on its parameter count and precision.
💡 Quantized models (INT8, INT4) trade some accuracy for dramatically smaller size — useful for running large models on consumer hardware or mobile devices.
A model's parameter count stays fixed, but the disk space it occupies depends entirely on how many bits are used to store each individual parameter. FP32 (32-bit floating point) uses 4 bytes per parameter and was the original standard for training. FP16/BF16 (16-bit) halves that to 2 bytes, now the standard for both training and inference on modern hardware. INT8 quantization compresses to 1 byte per parameter, and INT4 pushes further to roughly half a byte, these aggressive compression levels are what make running large models on consumer laptops and even phones possible at all.
Quantization converts a model's continuous floating-point weights into a smaller set of discrete values (like 256 possible values for INT8, or just 16 for INT4), then stores an index into that smaller value set instead of the full-precision number. This introduces some approximation error, but well-implemented quantization techniques (like the GK-quantization methods used in the popular GGUF format) keep this error small enough that quality loss is often barely perceptible for many practical use cases, while cutting file size by 4-8x compared to full precision.
GGUF (and its predecessor GGML) became the standard format for running large language models locally on consumer hardware through tools like llama.cpp and Ollama. It supports a range of quantization levels (commonly labeled Q4, Q5, Q8, etc.) that let users choose their own trade-off between file size, inference speed, and output quality, Q4_K_M has become a particularly popular middle-ground choice, offering close to INT4-level compression with noticeably better quality retention than naive 4-bit quantization.
The download size calculated here represents the model weights on disk, but running a model requires additional memory for activations, KV cache, and framework overhead during actual inference, this is a separate and usually larger consideration covered by GPU VRAM estimation rather than pure model file size. A model that downloads as a 4GB file may need considerably more than 4GB of RAM or VRAM to actually run, especially with longer context windows or larger batch sizes.