Scaling laws

Empirical relationships between model size, data volume, and computational resources in AI, typically showing performance gains with increased scale (e.g., loss decreasing as ${\text{size}}^{-\alpha }$). Early work (e.g., Kaplan et al., 2020) established foundational scaling behavior.

Key principles:

• Performance scales predictably with model size and data
• Data efficiency and compute efficiency are critical factors

Recent debates:

• The 2025-2026 surge in large model releases (e.g., Granite models) has challenged traditional scaling law assumptions
• Panel discussions (e.g., IBM Mixture of Experts) debated whether scaling laws remain valid amid architectural innovations like Mixture of Experts (model)
• Amazon’s blocking of chatgpt’s shopping agent highlighted practical constraints on agent deployment tied to scaling limitations

Related concepts:

• Mixture of Experts (model)
• Agent (AI)
• large-language-models

2026 04 14 IBM Mixture of Experts

Source Notes

• 2026-04-23: # Mixture of Experts: The “Fun-cember” of Model Releases, Scaling Laws, and Agent Wars Host: Tim Hwang Panelists: * Gabe Goodhart: Chief Architect, AI Open Innovation * Abraham Daniels: Sr. Technical Product Manager, Granite * Aaron Baughman: IBM Fellow, Maste (Mixture of Experts: The “Fun-cember” of Model Releases, Scaling Laws, and Agent Wars)
• 2026-04-23: https://www.youtube.com/watch?v=_lZgapJzFho # Mixture of Experts: The “Fun-cember” of Model Releases, Scaling Laws, and Agent Wars Host: Tim Hwang Panelists: * Gabe Goodhart: Chief Architect, AI Open Innovation * Abraham Daniels: Sr. Technical Product Manager, G (Mixture of Experts: The “Fun-cember” of Model Releases, Scaling Laws, and Agent Wars)
• 2026-04-14: # IBM Mixture of Experts --- --- # Mixture of Experts: The “Fun-cember” of Model Releases, Scaling Laws, and Agent Wars Host: Tim Hwang Panelists: * **[[entities/gabe-goodhart|Gabe Goo (IBM Mixture of Experts)