A model with 7 billion parameters stored as 32-bit numbers occupies about 28 GB. Many consumer computers and phones cannot load that, and even servers pay a large price in memory and time. Quantization reduces the number of bits used to store each number in the model, shrinking its size and often speeding it up, usually with only a small loss in quality.
The basic idea
Neural network weights are typically trained as 32-bit or 16-bit floating-point numbers. Quantization maps them to lower precision, such as 8-bit integers (INT8) or even 4-bit values. Fewer bits per weight means:
- Smaller files and less memory use
- Less data moved between memory and processor, which is often the real bottleneck
- Faster computation on hardware with efficient low-precision operations
- Lower energy use, important for phones and edge devices
A simple numerical example
Suppose weights range from -2.0 to 2.0. With 8-bit integers, we have 256 levels. We choose a scale so that the range maps onto integers from -127 to 127, store the integers and multiply by the scale when we need approximate real values.
import numpy as np
weights = np.array([0.82, -1.93, 0.07, 1.41, -0.55], dtype=np.float32)
scale = np.abs(weights).max() / 127 # map largest magnitude to 127
q = np.round(weights / scale).astype(np.int8) # quantise
restored = q.astype(np.float32) * scale # dequantise
print("int8 values:", q)
print("restored: ", restored.round(3))
print("max error: ", float(np.abs(weights - restored).max()))The restored values are close to the originals; the small difference is the quantization error. A model has millions or billions of weights, and the network is usually robust to such small errors, though not infinitely so.
Number formats
| Format | Bits per weight | Typical use |
|---|---|---|
| FP32 | 32 | Classic training |
| FP16 / BF16 | 16 | Modern training and inference |
| INT8 / FP8 | 8 | Efficient inference |
| INT4 / NF4 | 4 | Compact LLM inference |
| 2 to 3-bit | 2-3 | Experimental and extreme compression |
Memory for 7 billion parameters: about 28 GB at FP32, 14 GB at FP16, 7 GB at INT8, around 3.5 to 4 GB at 4-bit.
Types of quantization
Post-training quantization (PTQ)
Take a trained model and convert it without retraining. It is fast and convenient. Variants:
- Dynamic quantization: weights are stored in low precision and activations are quantised on the fly. Simple and useful for CPU inference of models like transformers.
- Static quantization: uses a small calibration dataset to measure the range of activations so they can be quantised ahead of time, giving better speed.
Quantization-aware training (QAT)
Simulate quantization during training or fine-tuning so the model learns to be robust to it. This typically gives better accuracy at low bit-widths, but needs training resources.
Methods for large language models
LLMs have special challenges, such as outlier values in activations that make naive quantization harmful. Popular approaches include:
- GPTQ: layer-by-layer weight quantization using approximate second-order information to minimise error.
- AWQ (activation-aware weight quantization): protects the small fraction of weights that matter most according to activation patterns.
- GGUF formats (used by llama.cpp): several quantisation levels such as Q8, Q5 and Q4 variants for CPUs and GPUs.
- bitsandbytes and QLoRA: 8-bit and 4-bit loading in Hugging Face Transformers, and fine-tuning small adapter layers on top of a quantised model.
- KV-cache quantisation: compress the stored attention keys and values to support longer contexts with less memory.
Granularity matters
Instead of one scale for a whole layer, finer groupings (per-channel or per-group, for example one scale per 64 or 128 weights) capture local variation and reduce error, with a small overhead for storing scales.
Using quantisation in practice
Loading a 4-bit model with Hugging Face Transformers and bitsandbytes:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_id = "Qwen/Qwen2.5-1.5B-Instruct" # replace with a model you have access to
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb, device_map="auto")
inputs = tokenizer("Explain quantization in one sentence.", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))This requires a compatible GPU and the bitsandbytes library. On CPUs and Macs, tools like llama.cpp and Ollama use GGUF quantised files.
Trade-offs
- Accuracy: 8-bit quantisation usually preserves quality very well. 4-bit is often acceptable for general chat, with small drops that can matter for maths, code or rare languages. Below 4 bits, quality can fall noticeably.
- Speed: gains depend on hardware support. Without efficient low-precision kernels, a quantised model may not run faster, though it still uses less memory.
- Task sensitivity: evaluate on your own tasks, including Nepali text, because benchmarks in English may not reflect your results.
- Compatibility: not every layer or operation can be quantised equally.
- Calibration data: poor calibration data can hurt accuracy.
How to choose
- Start with the largest model that fits in memory at 8-bit or 4-bit and compare it to a smaller model at higher precision. Often a bigger 4-bit model beats a smaller 16-bit one.
- Test quality on 50 to 100 realistic prompts.
- Measure speed and memory on your actual hardware.
- Use higher precision for sensitive tasks, and quantise less aggressively where accuracy matters.
- Keep the original model so you can compare and revert.
Other compression techniques
Quantisation is often combined with pruning (removing unimportant weights), distillation (training a small model to imitate a large one) and efficient architectures. Together they make AI practical on phones, laptops and small servers.
Frequently Asked Questions
Does quantisation make a model less accurate?
Slightly, usually. At 8-bit the difference is often negligible; at 4-bit it is small for many tasks but should be tested.
Can I quantise any model?
Most common architectures are supported by popular tools, but results vary and some need special handling.
Is quantisation only for inference?
Mostly, though techniques like QLoRA use quantised weights during fine-tuning to save memory.
What does Q4_K_M mean?
It names a specific 4-bit quantisation scheme in GGUF files, with variants that trade file size and quality.
Conclusion
Quantization trades a little numeric precision for large savings in memory and often speed, making powerful models usable on modest hardware. Choose the bit-width by testing on your own tasks, and combine it with other optimisation techniques when needed.