Speculative decoding
Speculative decoding is an inference acceleration technique for Autoregressive generation models that reduces latency by using a smaller Draft model to propose tokens, verified in parallel by a larger Target model.
Core Mechanism
- Drafting: A computationally cheap model generates a sequence of candidate tokens.
- Parallel Verification: The target model processes all candidates in a single forward pass, evaluating likelihoods simultaneously.
- Acceptance/Rejection: Tokens are accepted if the target distribution matches sufficiently; rejection triggers sampling from the target at the first discrepancy.
- Computational Efficiency: Reduces total FLOPs per token by leveraging parallel verification, though overhead depends on draft accuracy.
Related Implementations & Models
- DSparK: Recent implementation aiming for lossless acceleration via speculative decoding DSparK.
- Poolside’s Laguna S 2.1: An 118B parameter Mixture-of-Experts (MoE) agentic coding model optimized for local hardware. Its architecture and efficiency characteristics are relevant to the deployment of large-scale Target models in speculative decoding workflows. See Poolside’s Laguna S 2.1: Efficient Open-Source Agentic Coding for Local Hardware.