Qwen 3.8 27B AI Coding Harness Evaluation: DeepSeek Harness Outperforms

Clip title: Qwen3.8 27B: Same Model, Three Harnesses, One Clear Winner Author / channel: James Layne URL: https://www.youtube.com/watch?v=sSySOPGNdjw

Summary

This video evaluates the performance of three AI coding harnesses—Pi Coding Agent, Hermes Agent, and DeepSeek Harness—when used with the Qwen 3.8 27B model. The presenter conducts two primary tests: generating a playable Minecraft clone and creating a sophisticated color palette generator website. The goal is to determine which harness provides the best results in terms of code quality, functionality, and efficiency for these complex coding tasks.

For the Minecraft clone test, both Pi Coding Agent and Hermes Agent faced significant challenges. Pi Coding Agent required extensive back-and-forth prompting to correct numerous issues, such as missing textures, and while it eventually produced a somewhat playable Minecraft experience with basic elements like block breaking and hostile mobs, its crafting system remained non-functional. Hermes Agent performed even worse, needing many revisions to generate a game that still featured strange-looking animals, lacked proper block-breaking animations, and had an only partially working crafting table. Notably, for both these harnesses, the “low thinking level” of the AI model yielded better, albeit flawed, results compared to higher reasoning levels, which often produced unusable or bizarre outputs.

DeepSeek Harness, however, emerged as a clear leader in the Minecraft challenge. It managed to produce a remarkably complete and playable Minecraft clone with minimal revisions. This single-shot generation featured diverse biomes, mountains, functional crafting, correct block physics (like floating blocks), and appropriate mob behavior, including zombies and exploding creepers. Although minor issues like slow wood-block breaking and water not entirely filling were present, DeepSeek Harness delivered the most impressive and closest-to-spec Minecraft experience from a local AI model.

In the second test, generating a color palette website, the results largely mirrored the Minecraft performance. Pi Coding Agent created a functional website with various palette generation modes and export options, but its visual design was considered decent yet not top-tier. Hermes Agent, despite taking the longest to process, also delivered a functional and visually appealing website with responsive features like dark/light mode toggling. DeepSeek Harness once again excelled, producing the most professional and aesthetically pleasing website among all, closely resembling a high-quality custom-built application with all requested functionalities. Overall, DeepSeek Harness demonstrated superior capability in both code generation and adhering to complex specifications, securing its position as the top-performing harness in this evaluation.

Video Description & Links

Description

Qwen3.8 27B local LLM comparison: DeepSeek Harness vs Pi Coding Agent vs Hermes Agent. I tested the same Qwen 3.8 27B model across all three AI coding harnesses to compare coding quality, tool use, reliability, speed, and overall results.

Which is the best harness for Qwen3.8 27B, and how can the exact same local model produce completely different results?

Tested locally on an RTX 5090 using Unsloth Desktop.

Minecraft Prompt: https://docs.google.com/document/d/1gLob3fq25TWFdevQIWwsNCilOp7q4rlIqNiRvew4Sx4/edit?usp=sharing

Website Color Palette Prompt: https://docs.google.com/document/d/15fh1HQ0D8yIpJy-8Kab_0gDhmGN4VvlOooRLCuBPpr4/edit?usp=sharing

🖥️ My Setup:

Mac Mini running Openclaw
Mac Studio 128GB (For Local Inference)
RTX 5090 (32GB VRAM)
LM Studio for local LLM hosting

Purchases made through some links may provide some compensation to the creator.

00:00 - Qwen3.8 27B Best Harness 00:52 - Testing Process 01:29 - Minecraft Benchmark Test Prompt 02:09 - Pi Harness 02:38 - Pi Coding Agent Settings 05:18 - Pi Harness Prompting Qwen3.8 06:29 - Pi Harness Minecraft Results - Low Reasoning 09:24 - Pi Harness Minecraft Results - Medium Reasoning 10:05 - Pi Harness Minecraft Results - Xhigh Reasoning 10:53 - Hermes Agent Prompting Qwen3.8 11:39 - Hermes Agent Original Minecraft Generation 12:41 - Hermes Agent Minecraft Results - Low Reasoning 16:45 - Deepseek Harness Prompting Qwen3.8 17:23 - Deepseek Harness Original Minecraft Generation 18:48 - Deepseek Harness Minecraft Results - Low Reasoning 23:49 - Website Generator Benchmark Test Prompt 24:23 - Pi Harness Website Results - Low Reasoning 26:26 - Hermes Agent Website Results - Low Reasoning 28:23 - Deepseek Harness Website Results - Low Reasoning 30:13 - Codex Website Comparison 31:14 - Verdict

NemoClaw Knowledge Wiki

Explorer

Qwen 3.8 27B AI Coding Harness Evaluation: DeepSeek Harness Outperforms