Dense Models
Dense Models are Large Language Models (LLMs) where every parameter is activated for every token during inference. This contrasts with Sparse Models and Mixture of Experts (MoE) architectures, which activate only a subset of parameters per token.
Characteristics
- Uniform Computation: All layers and weights participate in processing each input token.
- Hardware Efficiency: Generally easier to optimize for standard GPU clusters due to uniform memory access patterns compared to sparse routing.
- Parameter Efficiency: Typically require fewer total parameters than MoE models to achieve comparable performance, as there is no “dead” weight.
- Latency: Inference latency is predictable and scales linearly with model depth and width, without the overhead of expert routing logic.
Comparison with Sparse Architectures
- vs. Mixture of Experts (MoE): Dense models often outperform MoE models of similar active parameter counts in specific reasoning tasks due to better gradient flow and lack of routing noise, though MoE scales better in total parameter size.
- Recent Evaluations: Recent benchmarks suggest that for certain agentic workflows, dense models offer superior consistency. For instance, the Ornith 9B Agentic Coding LLM: Local Performance Evaluation on Consumer Hardware demonstrates how dense architectures like Ornith-1.0 can be effectively deployed on consumer hardware for specialized agentic coding tasks, highlighting the viability of smaller dense models for local inference.