- “vram”
- “gpu”
- “machine-learning”
- “quantization”
- “llm”
- “video-ram”
- “gpu-memory”
- “model-compression”
- “llm-inference”
- “kv-cache”
- “paged-attention” group: open-systems-local-models aliases:
- “Video RAM”
- “GPU memory”
VRAM
Video RAM (VRAM) is the dedicated memory on a GPU used to store model weights, activations, and intermediate data during inference and training. Its capacity directly limits the size of models that can be executed on a single GPU, especially for resource-intensive tasks like large-language-model (LLM) deployment.
- VRAM Constraints in LLMs: Full-precision (32-bit) LLMs like NVIDIA’s Llama 3.1 Nemotron 70B (70.6 billion parameters) require ~30GB+ of VRAM (e.g., 30+ files at ~5GB each), exceeding most consumer GPUs.
- Quantization as a VRAM optimization: Reducing precision (e.g., 4-bit or 8-bit) significantly lowers memory footprint, enabling larger models to fit within limited VRAM.
- Inference Memory Overhead: Beyond static weights, VRAM must accommodate dynamic data structures during generation:
- Activations: Intermediate tensor values computed during forward passes.
- KV Cache: Stores key and value vectors for previous tokens to avoid redundant computation in autoregressive generation. This cache grows linearly with sequence length, often becoming the bottleneck for long-context inference.
- Paged Attention: A memory management technique that treats VRAM as virtual memory, allowing non-contiguous allocation of KV cache blocks to reduce fragmentation and improve throughput KV Cache and Paged Attention: Accelerating LLM Inference through VRAM Optimization.