Model Size
The physical and computational footprint of a machine learning model, primarily determined by the number of parameters (e.g., 7B, 70B) and their precision (e.g., 32-bit, 8-bit). Larger models require more storage, memory, and computational resources for training and inference.
Key implications:
- Storage: Full-precision 70B models (e.g., NVIDIA’s Llama 3.1 Nemotron 70B) require ~150GB (30 files × 5GB each).
- Hardware demands: High memory bandwidth and VRAM needed for inference, limiting deployment on consumer hardware.
- Trade-offs: Larger size often correlates with better performance but increases latency and cost.
Quantisation as optimization technique:
- model-efficiency reduces memory footprint by lowering precision (e.g., FP16 to INT8), enabling larger models to fit within constrained hardware environments without significant performance degradation.
Behavioral Optimization vs. Parameter Scaling: Recent discourse challenges the paradigm of infinite scaling, suggesting that for specific enterprise tasks, optimizing model behavior is more effective than increasing size.
- Paradigm Shift: “Stop Making Models Bigger. Make Them Behave” — focusing on tool use and disciplined output in smaller models rather than relying on massive parameter counts for general capability Training Smaller Models for Disciplined Tool Use in Enterprise AI.
- Enterprise Efficiency: Smaller models, when properly constrained and fine-tuned, can achieve higher reliability in structured workflows (e.g., API calls, data extraction) compared to larger, unconstrained counterparts, reducing hallucination rates and inference costs.