Token Per Second
Token Per Second (tok/s) is a primary metric for Large Language Model (LLM) inference speed, measuring the rate at which a model generates output tokens. It is a critical determinant of user experience, latency, and computational cost in real-time applications.
Key Factors
- Model Architecture: Mixture of Experts (MoE) models often achieve higher tok/s by activating only a subset of parameters per token.
- Hardware Acceleration: GPU VRAM bandwidth and tensor core utilization directly limit maximum throughput.
- Context Length: Longer contexts increase memory overhead, potentially reducing tok/s due to attention mechanism complexity.
- Quantization: Reducing precision (e.g., FP16 to INT8) can significantly boost tok/s by lowering memory bandwidth requirements and compute intensity.
Strategic Trade-offs: Speed vs. Intelligence
Recent industry analysis highlights the tension between maximizing inference speed and maintaining model capability, particularly in proprietary systems.
- GPT-5.6 Sol Case Study: OpenAI’s GPT-5.6 Sol demonstrates a strategic optimization for speed, reportedly achieving up to 18.5x speed improvements in specific benchmarks. This highlights a deliberate trade-off where raw throughput is prioritized, potentially at the expense of peak reasoning capabilities or “intelligence” density per token.
- Revenue and Infrastructure Implications: High tok/s enables lower latency for end-users and reduced compute costs per query, directly impacting revenue strategy and scalability. However, this requires careful balancing to ensure that speed enhancements do not degrade the quality of complex reasoning tasks.
- Hardware Constraints: Achieving such high throughput often relies on specialized hardware configurations and efficient KV cache management, pushing the limits of current GPU architecture capabilities.