Concurrent Users
Concurrent Users refers to the number of distinct sessions or requests processed by a system simultaneously. In the context of large-language-model (LLM) serving, this metric is strictly bounded by hardware resources, primarily vram capacity and compute throughput.
Key Constraints & Optimization
- Memory Bottlenecks: High concurrency leads to VRAM exhaustion if key-value states are not managed efficiently.
- Throughput vs. Latency: Increasing concurrent users often trades off individual request latency for higher aggregate throughput.
- VRAM Optimization: Techniques like inference-optimization management are critical for scaling concurrent user limits without linear hardware increases.
Related Concepts
- inference-optimization: Stores intermediate attention states to avoid recomputation.
- Paged Attention: A memory management technique that reduces fragmentation and allows higher concurrency by treating VRAM like virtual memory.
- llm-inference: The process of generating outputs from trained models.