🗂️ AI & Agents · View mindmap

Full Attention

Full Attention refers to the standard Transformer mechanism where every token in a sequence attends to every other token, resulting in quadratic computational complexity $O(N^2)$ relative to sequence length. While effective for capturing long-range dependencies, this approach becomes a bottleneck for large-language-model (LLM) inference and training at scale.

Core Characteristics

• Global Context: Each position has direct access to all previous positions.
• Computational Cost: Memory and compute requirements scale quadratically with context window size.
• Baseline Standard: Serves as the reference point for efficiency improvements in Sparse Attention, Sliding Window Attention, and inference-optimization optimizations.

Recent Developments & Optimizations

Efforts to mitigate the quadratic cost of full attention have led to various architectural innovations. Notable recent work includes:

• Minimax M3 Optimizations:
  • Minimax M3’s Optimized Attention for Efficient LLM Inference details specific architectural adjustments in the Minimax M3 model.
  • Focuses on reducing inference latency while maintaining performance metrics comparable to standard full attention mechanisms.
  • Highlights the trade-off between computational efficiency and context retention in modern LLMs.

Related Concepts

• Attention Mechanism
• Transformer Architecture
• Inference Efficiency
• context-window

References

• Minimax M3’s Optimized Attention for Efficient LLM Inference