- “llm”
- “local-inference”
- “quantization”
- “instruction-following”
- “video-generation”
- “pinokio” group: model-efficiency-compression
Local Inference
Running large language models (LLMs) directly on user-owned hardware without cloud dependency, enabling privacy, offline use, and reduced latency.
Recommended Models for Instruction Following (48GB VRAM)
- Llama 3.1 70B (quantized): Meta’s model excels in instruction-following tasks when model-efficiency reduces its footprint for 48GB NVIDIA GPU deployment.
- Gemma 2 27B (quantized): Efficient balance of size and performance for instruction tasks on consumer-grade hardware.
- Ornith 9B: Specialized for agentic coding workflows; see Ornith 9B Agentic Coding LLM: Local Performance Evaluation on Consumer Hardware for local performance benchmarks on consumer hardware.