Autoregressive Decoding
Autoregressive decoding is the standard method for generating text in large-language-models (LLMs), where tokens are predicted sequentially, one at a time, conditioned on all previously generated tokens. This sequential dependency creates a computational bottleneck, as each step requires a full forward pass through the model.
Core Mechanism
- Sequential Generation: The model predicts the probability distribution of the next token given the context .
- Latency Constraint: Inference speed is limited by the time required to process each token individually, often referred to as “time-to-first-token” and “inter-token latency.”
- Common Strategies: Includes Greedy Search, Beam Search, and Sampling methods (e.g., top-k, top-p).
Optimization Techniques
To mitigate the latency of sequential generation, various acceleration methods have been developed:
- Speculative Decoding: Uses a smaller “draft” model to propose multiple tokens, which are then verified in parallel by the larger target model. This reduces the number of forward passes required for the large model.
- KV Cache Optimization: Techniques to manage the memory overhead of storing attention states for long contexts.
- Quantization: Reducing precision (e.g., INT8, FP4) to speed up matrix multiplications.
Recent Developments (2026)
- DSparK: A novel speculative decoding technique developed by DeepSeek and Peking University.
- Claims up to 85% acceleration in LLM inference.
- Focuses on lossless acceleration, maintaining output quality while significantly reducing compute time.
- See detailed analysis in DeepSeek’s DSparK: Lossless LLM Inference Acceleration via Speculative Decoding.