Local AI Tools
Software and frameworks enabling the execution, management, and deployment of Large Language Models (LLMs) and other AI workloads on local hardware.
Key Frameworks & Tools
- llamacpp: High-performance C/C++ library for running LLMs locally; serves as the foundational backend for many higher-level interfaces.
- Router Mode: A native feature for hot-swapping models without restarting the server. See llama.cpp Router Mode: Native Hot-Swappable Local LLM Switching.
- Ollama: Simplifies running LLMs locally via CLI and REST API; optimized for ease of use, background service management, and seamless model switching.
- LM Studio: GUI-based interface for downloading and running local LLMs; emphasizes user-friendly model browsing, hardware configuration visualization, and chat interfaces.
- Text Generation WebUI (oobabooga): Comprehensive web interface for local LLM inference, offering extensive extension support and fine-tuning capabilities.
Comparison & Use Cases
For a detailed breakdown of when to choose each tool, see Ollama, LM Studio, and llama.cpp: Local AI Tool Comparison and Use Cases. Key distinctions include:
- llama.cpp: Best for developers requiring low-level control, integration into custom applications, or maximum efficiency on constrained hardware.
- Ollama: Ideal for CLI users, automated pipelines, and those seeking a “set-and-forget” background server with simple model management.
- LM Studio: Preferred by non-technical users or those who benefit from visual hardware diagnostics, easy model search/filtering, and a polished chat UI without configuration files.
Concepts
- Model Quantization: Reducing