Inference Scaling
Inference Scaling refers to the strategies and architectural optimizations employed to increase the throughput, reduce latency, and improve resource efficiency of Large Language Model (LLM) inference. Unlike training scaling, which focuses on model capacity and data volume, inference scaling addresses the computational bottlenecks of serving models to users, particularly regarding vram constraints and Token Generation speed.
Core Challenges
- Memory Bandwidth Bound: Inference is often limited by the speed at which weights and activations can be moved from HBM (High Bandwidth Memory) to the compute units, rather than raw FLOPS.
- Variable Sequence Lengths: Dynamic input and output lengths lead to irregular memory access patterns and inefficient batching.
- KV Cache Growth: The Key-Value cache grows linearly with sequence length, consuming significant vram and limiting batch sizes.
Optimization Strategies
Memory Management & Caching
- inference-optimization: Storing previously computed key and value vectors to avoid redundant computation during autoregressive generation. Efficient management of this cache is critical for reducing latency.
- Paged Attention: A memory management technique that decouples logical memory from physical memory, allowing non-contiguous allocation of KV cache blocks. This reduces fragmentation and enables higher throughput by maximizing GPU utilization.
- Integration Note: Recent developments highlight how KV Cache and Paged Attention: Accelerating LLM Inference through VRAM Optimization addresses VRAM optimization to accelerate LLM inference.
Compute Efficiency
- speculative-decoding: Using a smaller draft model to propose tokens, which are then verified by the larger target model, reducing the number of expensive forward passes.
- model-compression: Reducing the precision of model weights (e.g., FP16 to INT8 or INT4) to decrease memory footprint and increase bandwidth efficiency.
- Kernel Fusion: Combining multiple operations into a single kernel launch to minimize memory read/write overhead.
Serving Architecture
- Continuous Batching: Dynamically adding new requests to the batch as soon as previous requests finish, rather than waiting for the entire batch to complete.
- Tensor Parallelism: Distributing model layers across multiple GPUs to handle models that exceed single-device memory capacity.