NemoClaw Knowledge Wiki

Google DeepMind's Gemma 4: Open-Source AI Models and Architectural Innovations

5 min read

Google DeepMind’s Gemma 4: Open-Source AI Models and Architectural Innovations

Generated: 2026-04-29 · API: Gemini 2.5 Flash · Modes: Summary

Google DeepMind’s Gemma 4: Open-Source AI Models and Architectural Innovations

Clip title: Open Models at Google DeepMind — Cassidy Hardin, Google DeepMind Author / channel: AI Engineer URL: https://www.youtube.com/watch?v=_A367W_qvc8

Summary

Google DeepMind has introduced Gemma 4, the latest generation in their family of open-source AI models, which the presenter highlights for its unprecedented performance at smaller scales. Launched under an Apache 2.0 License, Gemma 4 aims to be more accessible for everyday developers, fostering easier integration into various development workflows from testing to deployment. The release features a range of models designed for diverse applications and hardware, marking a significant step forward in open-source AI capabilities.

The Gemma 4 family includes four distinct models: two smaller, “effective” models (E2B and E4B) optimized for on-device applications like phones, iPads, and laptops, and two larger models (26B and 31B) for more complex reasoning tasks. The E2B and E4B support text, vision, and audio inputs, while the 26B and 31B focus on text and vision, with the 26B being the first Gemma Mixture of Experts (MoE) model. Notably, the larger 31B (Dense) and 26B (MoE) models have achieved top rankings on the LLM Arena, outperforming models significantly larger than themselves, setting a new benchmark for what small open-source models can achieve.

A core focus of Gemma 4’s enhancements lies in its refined architecture. Improvements to the attention mechanism include a strategic interleaving of local and global layers (5:1 ratio, or 4:1 for E2B), utilizing sliding windows for local attention while global layers attend to all preceding tokens. A key innovation, Grouped Query Attention (GQA), boosts efficiency by having groups of queries share key and value heads—two for local layers and eight for global layers (with doubled key/value head lengths for global layers). Furthermore, the 26B model introduces a Mixture of Experts (MoE) design, featuring 128 total experts with eight active per forward pass, alongside a larger, constantly active shared expert. For the on-device effective models (E2B and E4B), Per-Layer Embeddings (PLE) were implemented, storing embedding tables in flash memory instead of VRAM, drastically reducing memory overhead and enabling superior performance on constrained hardware.

Gemma 4 also extends its native multimodal capabilities, building upon the vision support introduced in Gemma 3, and now adding audio to the E2B and E4B models. The vision encoder supports variable aspect ratios and resolutions for images, allowing developers to choose from five different soft token budgets to optimize for specific tasks like OCR or object recognition. Images are processed by splitting them into 16x16 pixel patches, which are then pooled into single embeddings and passed to the model, ensuring spatial positional encoding is maintained. For audio, the E2B and E4B models incorporate a 305 million-parameter conformer and an audio tokenizer, enabling advanced speech recognition and translation by processing audio embeddings rather than raw tokens.

In conclusion, Gemma 4 represents a substantial leap in open-source AI, pushing the boundaries of performance, efficiency, and accessibility across a spectrum of model sizes and modalities. Developers can get started with Gemma 4 today by downloading and self-hosting all models via Hugging Face, Kaggle, or Ollama, or by accessing the larger 31B and 26B models through Google Cloud-hosted options like AI Studio and Vertex AI for prototyping and deploying advanced agential workflows.

Video Description & Links

Description

Open models are getting smaller, faster, and far more capable. In this talk, Cassidy Hardin walks through the latest advances in the Gemma family, with a focus on Gemma 4 and what it enables for developers building on-device and open-weight AI systems. She covers the architecture behind Gemma’s dense, effective, and mixture-of-experts models, including improvements to attention, multimodal support for text, vision, and audio, and the design decisions that make strong reasoning, coding, and agentic workflows possible at practical sizes.

Speaker info:

Timestamps: 00:00:28 - Introduction to the Gemma 4 model family and its four size categories 00:01:54 - Shift to Apache 2.0 licensing for developer accessibility 00:02:25 - Deep dive into the 31B dense reasoning and 26B mixture-of-experts (MoE) models 00:03:30 - Overview of on-device effective models (2B and 4B) with multimodal support 00:04:21 - Architectural updates: interleaved local/global attention and grouped query attention 00:06:51 - Explanation of the new MoE architecture (128 experts, 8 active) 00:07:44 - Implementation of Per Layer Embeddings (PLE) to optimize on-device memory 00:11:06 - Multimodal advances: variable aspect ratios and resolutions for vision encoders 00:16:31 - Audio processing enhancements via conformer architecture and audio tokenizers 00:18:07 - Getting started: self-hosting (Hugging Face, Ollama) and cloud deployment (Vertex AI)

Tags

ai, ai engineer, ai engineering, software development, tech, startups, software architecture, machine learning

URLs

Graph View

Backlinks