GPU Compute Throughput
GPU Compute Throughput refers to the rate at which a Graphics Processing Unit performs arithmetic operations, typically measured in FLOPS (Floating Point Operations Per Second). In the context of large-language-model (LLM) inference, throughput is constrained not just by raw compute power, but by memory bandwidth and the efficiency of data movement, particularly regarding the Key-Value Cache.
Key Bottlenecks
- Memory Bound vs. Compute Bound: LLM inference is often memory-bound during the prefill phase and compute-bound during the decoding phase.
- KV-Cache Overhead: The Key-Value Cache grows linearly with sequence length, consuming significant VRAM and limiting batch sizes, thereby reducing overall throughput.
Optimization Strategies
- DualPath Architecture: Recent advancements, such as those detailed in DeepSeek’s DualPath: Optimizing LLM GPU Compute Throughput via KV-Cache, propose separating the processing paths for key and value states to optimize memory access patterns and reduce latency.
- Speculative Decoding: Reduces the number of autoregressive steps required, effectively increasing tokens-per-second throughput.
- Quantization: Using lower precision formats (e.g., FP8, INT4) to increase batch size and reduce memory bandwidth pressure.