Draft Model
A Draft Model (also known as a “fast model” or “proposal model”) is a smaller, computationally cheaper neural network used in conjunction with a larger Target Model to accelerate text generation. This architecture is central to Speculative Decoding, a technique that allows the system to generate multiple tokens in parallel using the draft model, which are then verified by the target model in a single forward pass.
Core Mechanism
- Proposal Phase: The draft model predicts a sequence of tokens based on the current context.
- Verification Phase: The target model evaluates the entire sequence of proposed tokens simultaneously.
- Acceptance/Rejection: Tokens are accepted if the target model’s probability distribution aligns with the draft; otherwise, the sequence is truncated at the first mismatch, and the target model’s prediction for that position is used.
- Efficiency: Reduces the number of expensive target model forward passes per output token, significantly lowering latency without altering output quality.
Implementations & Case Studies
- DeepSpec DSparK: An open-source implementation by DeepSeek AI designed to accelerate LLM text generation.
- Demonstrated on local hardware using Qwen3 as the target model.
- Reproduced significant speedups in inference latency via speculative decoding strategies.
- See: DeepSpec DSparK: Local Qwen3 LLM Acceleration through Speculative Decoding
Related Concepts
- speculative-decoding
- Large Language Model Inference
- Model Distillation