Selective Quantization
Selective Quantization is an optimization technique in large-language-models (LLMs) that applies varying precision levels to different parts of a neural network. Unlike uniform quantization, it identifies and preserves higher precision for sensitive layers or parameters while aggressively compressing less critical components. This approach maximizes parameter efficiency and reduces memory footprint with minimal impact on model performance.
Key Applications & Developments
- DwarfStar (DS4) System: A system engineering project demonstrating extreme compression capabilities.
- Enables running the deepseek V4 Flash model (284 billion parameters) on standard laptop hardware.
- Utilizes selective quantization to bypass typical memory constraints associated with models of this scale.
- Detailed analysis available in: DwarfStar: Enabling 284B DeepSeek V4 Flash on Laptops via Selective Quantization
Technical Context
- Precision Trade-offs: Balances FP16/FP32 preservation in attention heads or residual connections against INT4/INT8 quantization in dense linear layers.
- Hardware Efficiency: Critical for deploying MoE (Mixture of Experts) architectures on consumer-grade GPUs with limited VRAM.