Storage Bandwidth
Storage Bandwidth refers to the rate at which data can be read from or written to storage devices or memory hierarchies. In the context of high-performance computing and large-language-model (LLM) inference, it is often the primary bottleneck limiting throughput, particularly during the decoding phase where inference-optimization access dominates compute requirements.
Key Characteristics
- Memory Wall: The disparity between processor speed and memory access speed.
- Bandwidth-Bound Operations: Tasks where performance is limited by data transfer rates rather than arithmetic operations (e.g., Matrix Multiplication with low arithmetic intensity).
- HBM vs. VRAM: High Bandwidth Memory (HBM) offers significantly higher throughput than standard GDDR VRAM, crucial for large model weights.
Consumer-Grade MoE Inference
Recent advancements in Mixture-of-Experts (MoE) architectures allow for massive parameter counts with lower active inference costs. The Colibri project demonstrates how to run the 744-billion parameter GLoM 5.2 model on consumer-grade laptops by optimizing data movement and leveraging sparse activation patterns. This approach directly addresses the Memory Wall by reducing the effective bandwidth demand per token generated, making large-scale large-language-model inference accessible outside of data-center GPU clusters.
See Colibri: Unlocking 744B MoE LLMs for Consumer-Grade Laptops for technical details on this implementation.