Quantisation
Quantisation is a technique in machine learning that reduces the precision of numerical values used in neural networks, typically by representing weights and activations with fewer bits than standard floating-point formats. Standard neural networks commonly use 32-bit floating-point numbers, but quantisation can reduce this to 8-bit, 4-bit, or even lower representations. This compression directly decreases the memory footprint and computational requirements of models.
4-bit Quantisation
4-bit quantisation represents model parameters using only four bits per value, reducing memory usage by approximately 87% compared to 32-bit floating-point representations. This extreme compression enables deployment of large models on resource-constrained devices and significantly accelerates inference speed. However, the aggressive reduction in numerical precision introduces quantisation error, which can degrade model accuracy if not carefully managed through techniques such as calibration and fine-tuning.
Trade-offs and Applications
The primary advantage of 4-bit quantisation is enabling the use of large language models and other compute-intensive architectures on consumer hardware with limited memory and processing power. The trade-off is that some loss in model performance is typically unavoidable, though recent methods have shown this loss can be minimised to acceptable levels for many applications. 4-bit quantisation has become increasingly practical for AI agents and other systems requiring efficient deployment without access to high-end accelerators.