Bonsai 27B vs. Qwen 35B: LLM Performance and Replacement Feasibility Benchmarks
Generated: 2026-07-22 · API: Gemini 2.5 Flash · Modes: Summary
Bonsai 27B vs. Qwen 35B: LLM Performance and Replacement Feasibility Benchmarks
Clip title: Can a 3.5GB model replace my 35B daily driver? (Bonsai 27B) Author / channel: Codacus URL: https://www.youtube.com/watch?v=rBLWDJrXCp0
Summary
This video provides a detailed benchmark comparing several large language models (LLMs) on common tasks, focusing on the trade-offs between model size, speed, and real-world applicability on various hardware configurations. The main topic revolves around whether smaller, highly compressed LLMs, specifically the Bonsai 27B models (3.5GB binary and 7GB ternary), can match or replace a larger “daily driver” LLM, the Qwen 3.6 35B Mixture of Experts (MoE) model (21GB total footprint). A dense 12B Gemma 4 model (7GB) was also included as a challenger. The video emphasizes that the Bonsai models achieve their small size not through post-training quantization, but through Quantization Aware Training (QAT) down to 1 or 2 bits, meaning they were trained from day one with these severe weight constraints.
The comparison involved three core tasks: generating a complex, self-contained HTML landing page with inline CSS, debugging a server with disk space issues, and diagnosing a sabotaged server gateway. In the UI generation test, both the 21GB Qwen MoE and 7GB Bonsai Ternary models performed commendably, producing high-quality, properly structured pages. The 3.5GB Bonsai Binary and 7GB Gemma 4 models lagged in design quality or completeness. For the initial server debugging task, all models successfully identified and fixed the disk space issue, with differences primarily in “chattiness” (number of commands executed). However, when faced with ambiguity, the Gemma model accurately re-evaluated, while the Bonsai Binary model “invented” incorrect explanations.
The more complex server troubleshooting task exposed further differences. While the Qwen MoE and Bonsai Ternary consistently fixed the issue, the Bonsai Binary model’s path was chaotic, including wiping a config file and attempting to kill actual system processes (prevented only by sandbox permissions). The Gemma 4 model, surprisingly, failed to complete this task reliably, getting stuck in repetitive loops. This revealed that compression can impact stability under long or complex contexts. The video highlights a crucial performance distinction: the 21GB MoE model was significantly faster (up to twice as fast) because it uses a “Mixture of Experts” architecture, where only a small subset of its parameters (around 3 billion out of 35 billion) are active per token. In contrast, the Bonsai models are “dense,” meaning all 27 billion parameters are engaged for every token, despite their smaller file size due to QAT. This means MoE “shrinks the math” while QAT “shrinks the file.”
The conclusion emphasizes that the “best” model depends heavily on the user’s hardware. For systems with a dedicated GPU and ample system RAM (allowing for “offloading” of experts, as with the presenter’s 12GB GPU + 10.5GB system RAM setup), the 21GB Qwen MoE is superior due to its speed and long-context stability. However, for hardware with a unified memory pool, such as Apple Silicon Macs (e.g., 16GB total memory) or GPUs with limited VRAM (e.g., 6-8GB) and low system RAM, the larger MoE model simply doesn’t fit. In these scenarios, the Bonsai Ternary (7GB) and Binary (3.5GB) models become excellent choices, offering comparable intelligence density and fitting within constrained memory budgets. The video optimistically points to the future potential of a “Ternary MoE” model, combining both math and file shrinking techniques, which could lead to massive LLMs running on consumer-grade hardware.
Video Description & Links
Description
PrismML compressed Qwen’s 27B dense model down to 3.5 GB with true 1-bit weights. Not quantization: the weights were trained as −1/+1 from day one. So I put both Bonsai builds up against my actual daily driver, a 21 GB Qwen 3.6 35B MoE, plus Gemma 4 12B as the fair-size control, and ran about thirty tests on one RTX 3060.
Same landing-page brief. Same broken server over SSH. Same sampling settings, same KV cache quant, same effort level, every model.
What came out of it surprised me more than once. The 21 GB model is more than twice as fast as a 7 GB one, and the reason has nothing to do with file size. The only model that could not reliably finish a long task was not the smallest one. And whether Bonsai makes sense on your machine comes down to a number almost nobody quotes on a spec sheet.
If you run this on a Mac or a small card, tell me what you get in the comments.
CHAPTERS 0:00 Two models, one broken server 0:39 What Bonsai actually is (1-bit vs quantization) 3:24 Test 1 — can it design a webpage? 6:34 Test 2 — SSH debugging, everyone passed 9:26 Test 3 — the trap that survives a restart 13:29 Where it breaks (Gemma’s loop) 15:35 Why the bigger model is faster (A3B vs dense) 17:54 The number nobody quotes (total footprint) 20:06 The verdict — who should run what
THE MODELS Bonsai 27B (binary Q1_0, 3.5 GB / ternary Q2_0, 7 GB): [https://huggingface.co/prism-ml]
Qwen 3.6 35B-A3B: [https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF]
llama.cpp Metal Q2_0 support (merged): github.com/ggml-org/llama.cpp/pull/25419
llama.cpp CUDA Q2_0 support (open PR I built from): github.com/ggml-org/llama.cpp/pull/25707
THE RIG:
RTX 3060 12 GB llama.cpp built from PR #25707 KV cache q8_0 (all models) reasoning effort medium (all models) MoE runs with 26 expert layers offloaded to system RAM
MY RESULTS AT A GLANCE
Generation speed at depth (tg, tokens/sec): MoE 47.9 · Bonsai Q1 34.7 · Bonsai Q2 21.6
Total memory footprint: MoE ~21 GB (10.5 VRAM + 10.6 RAM) ternary 8.8 GB binary 5.3 GB
Server-repair task, wall clock: MoE ~56s ternary ~114s binary ~134s
Design test: ternary and the MoE land in the same class. Binary and Gemma sit a tier below.
localai llm bonsai quantization 1bit qwen llamacpp selfhosted ai rtx3060
Tags
localai, codacus, homelab, bonsai 27b, 1-bit llm, ternary llm, prismml, bitnet, qwen 3.6, qwen 35b moe, llama.cpp, local ai, local llm, run llm locally, rtx 3060 ai, llm on mac, apple silicon llm, quantization, gguf, self hosted ai