NemoClaw Knowledge Wiki

RotorQuant vs TurboQuant LLM KV Cache Compression Performance Reality Check

4 min read

RotorQuant vs TurboQuant: LLM KV Cache Compression Performance Reality Check

Clip title: RotorQuant vs TurboQuant: 31x Speed Claim - Reality Check (Local AI) Author / channel: Protorikis URL: https://www.youtube.com/watch?v=wSxsYjScRr0

Summary

This video provides an in-depth look at Key-Value (KV) cache compression techniques for Large Language Models (LLMs), focusing on Google’s TurboQuant and the open-source alternative, RotorQuant. The main topic revolves around increasing LLM context window size and improving inference speed by efficiently compressing the KV cache, which stores token representations. The video highlights that while TurboQuant effectively compresses KV cache memory by up to 5 times (from 16-bit to 3.5-bit), this significant compression comes with a hidden cost: dramatically increased prompt processing (prefill) and token generation latency.

The video explains TurboQuant’s mechanism using a suitcase analogy for vector quantization. A token’s vector, representing its meaning in the context, consists of many dimensions (numbers). Some are small “nuances” (like socks), while others are large “outliers” (like ski boots). Standard 4-bit quantization (Q4) would round these nuances to zero, effectively “lobotomizing” the vector and losing critical directional information (the “soul” of the vector). TurboQuant’s clever solution is to use a large 128x128 rotation matrix as a “blender.” This matrix multiplies the input vector, spreading its “spiky” energy (outliers) across all dimensions, making the values more uniform. This uniformity allows for 4-bit quantization to preserve the nuances (as non-zero “noise”) and, crucially, the original positive/negative signs of the dimensions, thus maintaining context accuracy.

However, the “blending” rotation matrix multiplication is computationally expensive. For a single 128-dimension vector, it requires 16,384 multiply-add operations. When scaled across multiple keys, values, attention heads, and model layers during the prefill phase, this amounts to billions of additional compute operations, creating a significant latency bottleneck. This is where RotorQuant, developed by the open-source community Scrya, steps in. RotorQuant proposes replacing the dense matrix rotations with block-diagonal rotations, exploiting geometric algebra. Its variants, IsoQuant and PlanarQuant, further optimize this by splitting the vector into smaller, independent chunks (e.g., 4 dimensions for IsoQuant) and applying simpler quaternion-based rotations to each chunk. This approach boasts a drastic reduction in computational load (32x less compute) and data movement (128x less) compared to TurboQuant’s dense matrix approach, with claims of 10-31x speedups on modern GPUs.

Despite RotorQuant’s impressive theoretical advantages, real-world testing on an Apple M3 Max revealed a practical challenge. When running IsoQuant with a large Qwen model, the prefill latency was unacceptably high, and the CPU was overloaded while the GPU remained underutilized. The core issue was identified as a high number of “graph splits,” indicating that the llama.cpp fork used for the test lacked proper Metal kernel implementations for IsoQuant. This forced computational tasks to fall back to the slower CPU, negating the architectural benefits. In contrast, the original TurboQuant implementation, which already has optimized GPU kernels, performed as expected with minimal graph splits and efficient GPU utilization. The video concludes that while RotorQuant and its variants represent a promising future for LLM efficiency by significantly reducing prefill latency, their full potential will only be realized once robust, hardware-optimized kernel implementations are widely available, especially for platforms like Apple Silicon.

Graph View