Proxy Metrics
Proxy metrics are measurable indicators used as stand-ins for complex, unobservable, or difficult-to-measure target variables. In optimization contexts, particularly in machine-learning and AI Alignment, relying on proxies introduces the risk that the system will optimize the metric rather than the underlying goal, leading to Goodhart’s Law effects.
Core Dynamics
- Substitution Error: Treating the proxy as the target itself.
- Optimization Pressure: As pressure to improve the proxy increases, the correlation between the proxy and the true target often degrades.
- Gaming: Agents (human or algorithmic) exploit loopholes in the proxy definition to maximize scores without improving the actual outcome.
AI and LLM Context
In Large Language Models, token generation speed, perplexity scores, or benchmark pass rates often serve as proxies for “intelligence” or “utility.” Recent analysis highlights a critical divergence between these proxies and actual cognitive capability.
- Token vs. Intelligence: The industry has faced criticism for prioritizing token throughput and consumption over genuine reasoning capabilities. This misalignment is described as a “billion-dollar mistake” where infrastructure scales for volume rather than value Goodhart’s Law in AI: The Cost of Confusing Tokens with Intelligence.
- Benchmark Saturation: As models overfit to specific evaluation benchmarks, these benchmarks cease to be valid proxies for general intelligence, becoming mere measures of memorization or pattern matching.
Mitigation Strategies
- Multi-objective Optimization: Use diverse, uncorrelated proxies to reduce the surface area for gaming.
- Adversarial Testing: Actively attempt to break the proxy-target correlation.
- Human-in-the-Loop: Incorporate subjective human judgment where quantitative proxies fail.