NemoClaw Knowledge Wiki

Adam Lucek - quantisation of LLM

5 min read

Adam Lucek - quantisation of LLM https://www.youtube.com/watch?v=3EDI4akymhA This video provides a detailed overview of quantization in the context of large language models (LLMs), explaining what it is, why it’s necessary, and how it’s implemented. 1. The Challenge of Large Language Models: LLMs like NVIDIA’s Llama 3.1 Nemotron 70B (70.6 billion parameters) are massive, often requiring gigabytes of storage (e.g., 30+ files, each ~5GB). Running these models at full precision demands significant computational resources, including expensive GPU clusters with large amounts of VRAM, making them inaccessible to most consumers. 2. What is Quantization?

  • General Definition: Quantization is the process of mapping input values from a large (often continuous) set to output values in a smaller (countable) set. This involves rounding and truncation, making it a form of lossy compression.
  • In Deep Learning/ML: It’s a technique to reduce the computational and memory costs of running inference on models. This is achieved by representing the model’s internal weights and activations using low-precision data types (e.g., 8-bit integers or 4-bit integers) instead of the standard 32-bit floating-point numbers (float32).

3. Why Quantize LLMs? Quantization makes LLMs significantly more accessible. By reducing the number of bits needed to represent weights:

  • Memory Footprint: The model requires less memory (VRAM).
  • Energy Consumption: It consumes less energy.
  • Inference Speed: Operations like matrix multiplication can be performed faster with integer arithmetic.
  • Hardware Accessibility: It allows models to run on less powerful, consumer-grade hardware (laptops, even CPUs) where previously only high-end GPUs or clusters were capable. Popular tools like OLLAMA and LM Studio leverage quantization to run LLMs locally.

4. How Numbers are Stored (Briefly):

  • Decimal (Base-10): How humans represent numbers (e.g., 254 = 4x10^0 + 5x10^1 + 2x10^2).
  • Binary (Base-2): How computers work (0s and 1s).
  • Floating-Point Numbers: A standard way to represent real numbers (including fractions) in a compact binary format, similar to scientific notation (sign, exponent, significand/mantissa). Common types in deep learning include: FP32 (float32): 32 bits (1 sign, 8 exponent, 23 fraction). FP16 (half-precision): 16 bits (1 sign, 5 exponent, 10 fraction). BF16 (bfloat16): 16 bits (1 sign, 8 exponent, 7 fraction). BF16 is often preferred in DL as it retains more exponent precision, which is crucial for the dynamic range of gradients during training.

5. Quantization in Practice (Reducing Precision): The core idea is that for inference, the exact numerical precision of every weight isn’t as critical as the relationships between weights.

  • BitsAndBytes Library: A widely used library that enables 8-bit (INT8) and 4-bit (INT4) quantization for transformers.
  • INT8 Quantization: Maps the larger floating-point range (e.g., FP16) into an 8-bit integer range (-127 to 127). This involves scaling the vector by its absolute maximum.
  • INT4 Quantization: Even more aggressive, using only 4 bits to represent values (16 distinct representations). This means mapping the floating-point values into an even smaller integer range (e.g., 0-15 for unsigned, or -7 to 8 for signed).

6. Performance vs. Size Trade-off:

  • Minimal Performance Degradation: Extensive research (like the Hugging Face BitsAndBytes blog posts) shows that quantizing models down to 8-bit or even 4-bit precision results in surprisingly little loss in accuracy or performance on common benchmarks. The key is preserving the relative differences between weights.
  • Dramatic Resource Reduction: The benefits in terms of memory and computational requirements are immense. For a 1-billion parameter model: FP16 (base model): ~4.9 GB VRAM INT8: ~1.7 GB VRAM 4-bit: ~1.2 GB VRAM This allows models to run on devices with as little as 4-8GB of VRAM (common in consumer GPUs or even integrated graphics on laptops) or purely on CPU RAM.

7. GGUF Format:

  • Developed by Georgi Gerganov for llama.cpp (a C/C++ inference framework for LLMs).
  • Single File: Stores all model data (weights, metadata, etc.) in a single, optimized file. This simplifies distribution and loading.
  • CPU Optimization: Specifically designed for efficient loading and running on CPUs (and other architectures supporting C++ inference).
  • k-quants Methods: GGUF offers various quantization methods (e.g., q4_K_M, q8_0, q2_K), allowing different parts of the model to be quantized with varying bit-depths based on their sensitivity to compression. q4_K_M is a popular choice for balancing performance and size.

8. Practical Implementation (Code Example): The video demonstrates using the bitsandbytes library within Hugging Face’s transformers to load a Llama 3.2 1B model in FP16, 8-bit, and 4-bit. It then shows how to convert a Hugging Face model to the GGUF format and run it using the llama.cpp command-line interface on a CPU, consuming minimal RAM (e.g., 800MB for a 1B 4-bit model). Conclusion: Quantization is a game-changer for LLM accessibility. By intelligently reducing the numerical precision of model weights, it significantly cuts down on memory and computational demands without a major drop in performance. This enables running powerful LLMs on a wide range of consumer hardware, fostering broader adoption and experimentation within the AI community.

Graph View