Token Per Second
Token Per Second (tok/s) is a primary metric for Large Language Model (LLM) inference speed, measuring the rate at which a model generates output tokens. It is a critical determinant of user experience, latency, and computational cost in real-time applications.
Key Factors
- Model Architecture: Mixture of Experts (MoE) models often achieve higher tok/s by activating only a subset of parameters per token.
- Hardware Acceleration: GPU VRAM bandwidth and tensor core utilization directly limit maximum throughput.
- Context Length: Longer contexts increase memory overhead, potentially reducing tok/s due to KV cache management.
- Quantization: Lower precision formats (e.g., INT4, FP8) reduce memory footprint and increase bandwidth efficiency, boosting tok/s.
Recent Developments & Benchmarks
- Qwopus Coder: The Qwopus Coder: Agentic Code Self-Correction and MTP-Driven Efficiency model (Qwopus 3.6-35B-A3B-Coder) demonstrates high efficiency in coding tasks.
- Built on the Qwen 3.6-35B A3B base with Mixture of Experts architecture.
- Achieves approximately 160 tok/s while performing agentic code self-correction.
- Highlights the trade-off between model size (35B parameters) and inference speed via MoE sparsity.
Related Concepts
- Inference Latency
- mixture-of-experts
- llm-quantization
- Throughput