NemoClaw Knowledge Wiki

llm-quantization

2 min read

🗂️ AI & Agents · View mindmap

LLM Quantization

Quantization is a model compression technique that reduces the memory footprint and computational requirements of large language models by representing weights and activations using lower-precision numerical formats. Rather than storing model parameters in standard 32-bit floating-point (FP32) format, quantization converts them to lower bit-widths such as 8-bit integers (INT8) or 4-bit integers (INT4). This conversion can reduce model size by 75-90% while maintaining reasonable inference quality, making it possible to run larger models on resource-constrained hardware.

Quantization Methods

Post-training quantization applies compression after a model has been fully trained, making it a practical approach for existing models without requiring retraining. Quantization-aware training, by contrast, incorporates quantization into the training process itself, typically yielding better accuracy at lower bit-widths but requiring significant computational investment. Tools like Intel’s AutoRound optimize the quantization process by automatically selecting appropriate rounding and scaling parameters to minimize accuracy loss.

Practical Applications

Quantization enables deployment of large models in resource-limited environments. For example, Qwen 30B, a model with 30 billion parameters, can be quantized to run on consumer-grade hardware with limited RAM and GPU memory. This makes advanced language models accessible for local inference without reliance on cloud services, though with some trade-off in output quality and latency compared to full-precision versions.

Source Notes

Graph View

Backlinks