Memory Footprint
The total amount of computer memory required to store and execute a program or data structure. In the context of Large Language Models (LLMs), memory footprint is the primary constraint for deployment, dictating whether a model can run on consumer-grade hardware or requires specialized GPU clusters.
Key Components
- Model Weights: The static parameters of the model. Size is directly proportional to parameter count and precision (e.g., FP16 vs. INT8).
- Activation Memory: Temporary storage for intermediate calculations during inference or training. Scales with batch size and sequence length.
- Optimizer States: For training, memory required for gradients and optimizer states (e.g., AdamW) can significantly exceed weight storage.
- Quantization Impact: Aggressive quantization (e.g., Q1/Q2) drastically reduces weight footprint, enabling deployment on limited VRAM, though it may impact reasoning fidelity.
Case Study: Bonsai-2-27B on Consumer Hardware
Recent evaluations of the LLM Bonsai-2-27B (Ternary-Bonsai-2-27B-gguf) highlight the trade-offs between extreme quantization and hardware constraints.
- Hardware Constraints: Tested on a 16GB VRAM local setup, demonstrating that 27B parameter models can be deployed on consumer-grade GPUs via aggressive quantization.
- Quantization Comparison: Benchmarked Q1 (1-bit) vs. Q2 (2-bit) formats. While Q1 minimizes footprint, Q2 often provides a better balance of memory efficiency and reasoning performance.
- Performance Metrics: The re-evaluation focuses on reasoning capabilities and overall benchmark scores to determine viability for local inference.
For detailed benchmarking data and methodology, see Q2 Re-evaluation: Benchmarking Performance, Memory, Reasoning.