🗂️ AI & Agents · View mindmap

Auto-regressive Models

Auto-regressive models are a class of generative models that generate data sequentially, predicting the next element in a sequence based on all preceding elements. They are the foundational architecture behind most large language models (LLMs) and many time-series forecasting systems.

Core Mechanism

• Sequential Generation: Outputs are generated token-by-token (or step-by-step). The probability distribution for the next token $x_t$ is conditioned on the history $x_{<t}$.
• Markov Property: Assumes the current state depends only on the immediate previous state(s), though in practice, attention mechanisms allow access to the entire context window.
• Training Objective: Typically trained via maximum likelihood estimation to maximize the probability of the ground-truth sequence.

Key Architectures

• Transformers: Dominant architecture for NLP, using self-attention to weigh the importance of different tokens in the context.
• RNNs/LSTMs: Older recurrent architectures, largely superseded by Transformers due to parallelization capabilities during training.
• Diffusion Models: Traditionally used for image generation, now being adapted for text. Unlike auto-regressive models, diffusion models denoise data iteratively and can potentially generate tokens in parallel.

Limitations

• Latency: Sequential nature prevents parallel generation during inference, leading to slower throughput compared to non-auto-regressive methods.
• Error Propagation: Mistakes in early tokens can compound, affecting subsequent predictions.
• Context Window: Limited by memory constraints and attention mechanism complexity.

Recent Developments & Alternatives

• Parallel Decoding: Techniques like speculative decoding aim to mitigate latency by generating multiple tokens in parallel and verifying them.
• Diffusion for Text: Emerging architectures challenge the auto-regressive paradigm by treating text generation as a denoising process, allowing for parallel token generation.
  • See NVIDIA’s TwoTower: Parallel Diffusion Architecture for Faster Text Generation for details on NVIDIA’s approach to parallel diffusion for text.

Related Concepts

• Transformer
• large-language-model
• Diffusion Model
• Tokenization
• Attention Mechanism

References

• NVIDIA’s TwoTower: Parallel Diffusion Architecture for Faster Text Generation