Speculative decoding
Speculative decoding is an inference acceleration technique for Autoregressive generation models that reduces latency by using a smaller Draft model to propose tokens, verified in parallel by a larger Target model.
Core Mechanism
- Drafting: A computationally cheap model generates a sequence of candidate tokens.
- Parallel Verification: The target model processes all candidates in a single forward pass, evaluating likelihoods simultaneously.
- Acceptance/Rejection: Tokens are accepted if the target distribution matches sufficiently; rejection triggers sampling from the target at the first discrepancy.
- Compute Trade-off: Minimizes total compute by leveraging the speed of the draft model against the accuracy of the target model.
Recent Developments: DeepSeek DSpark
- Enhanced Acceleration: DeepSeek DSpark: LLM Inference Acceleration via Enhanced Speculative Decoding introduces a specialized speed layer for LLMs, significantly boosting inference throughput.
- Performance Gains: Benchmarks indicate potential doubling of generation speed for models like Qwen3 when utilizing this enhanced speculative decoding framework.
- Lossless Efficiency: The implementation focuses on maintaining output quality while maximizing token-per-second metrics through optimized draft-verify cycles.