NemoClaw Knowledge Wiki

Tencent AI Model Shrink with Sherry Quantization: AngelSlim Breakthrough Report

3 min read

🗂️ AI & Agents · View mindmap

Tencent AI Model Shrink with Sherry Quantization: AngelSlim Breakthrough Report

Clip title: Tencent Shrank a 1.5TB AI Model to 214GB — Here’s the Trick Author / channel: Fahd Mirza URL: https://www.youtube.com/watch?v=4y8WjfawRrk

Summary

This video discusses Tencent’s significant achievement in shrinking a 1.5 Terabyte (TB) AI model, specifically the 770-billion parameter Hy4 preview, down to 214 Gigabytes (GB) using a technique called Sherry Quantization, implemented through their AngelSlim toolkit. This represents a seven-fold reduction in model size with only a 0.7% performance drop in benchmarks. The speaker highlights this as a major breakthrough, enabling large AI models to run on more accessible hardware, such as laptops with 32GB of RAM or servers equipped with 4x 4400 16GB VRAM GPUs, thereby reducing computational and storage costs.

The core of this compression lies in Sherry Quantization, a clever engineering trick. It converts the full-precision, messy decimal weights of the AI model into “ternary weights,” which are simplified to one of three values: -1, 0, or +1. Crucially, Sherry Quantization enforces a specific sparsity pattern by guaranteeing exactly one zero in every group of four weights. This fixed pattern allows these four ternary weights to be efficiently packed into just 5 bits of storage, equating to 1.25 bits per weight. This method avoids wasted storage space and aligns perfectly with SIMD (Single Instruction, Multiple Data) hardware, leading to faster inference speed.

The speaker emphasizes that his channel had already covered AngelSlim and similar 2-bit quantization techniques months prior, expressing a sense of vindication rather than surprise at this trending news. He notes that Tencent’s official documentation for AngelSlim is dense and difficult to understand due to a mix of English and Chinese and complex academic explanations, which he believes has hindered its broader recognition. Despite the technical complexity, the speaker provides a step-by-step action plan with commands for viewers who possess the necessary high-end GPU hardware to implement and run these large quantized models themselves, or to re-quantize them.

In conclusion, Tencent’s Sherry Quantization via AngelSlim represents a substantial leap in AI model compression, making powerful large language models more efficient and accessible by dramatically reducing their size and computational demands with minimal impact on performance. This innovation is crucial for broader AI adoption and deployment in diverse environments. The video implicitly underscores the value of channels like the speaker’s in deciphering and disseminating complex technical advancements to a wider audience, often ahead of mainstream recognition.

Video Description & Links

Description

Tencent shrank a 770B model from 1.5TB to 214GB with Sherry quantization — here’s how it works, explained simply.

sherryquantization tencent agentslim

▶ LinkedIn: / fahdmirza
▶ YouTube: / @fahdmirza

▶ https://huggingface.co/AngelSlim/Hy4-preview-GGUF

All rights reserved © Fahd Mirza

URLs

Graph View