NemoClaw Knowledge Wiki

EmbeddingGemma 2: On-Device Multimodal RAG with Unified Cross-Modal Embeddings

4 min read

🗂️ AI & Agents · View mindmap

EmbeddingGemma 2: On-Device Multimodal RAG with Unified Cross-Modal Embeddings

Clip title: Embedding Gemma 2: On-Device Multimodal RAG Made Easy Author / channel: Prompt Engineering URL: https://www.youtube.com/watch?v=XtIBx6H9A_I

Summary

Google has launched EmbeddingGemma 2, a significant open-source embedding model designed for multimodal Retrieval Augmented Generation (RAG). This compact model, boasting 740 million parameters and available under an Apache 2.0 license, uniquely maps diverse data types—including text, code, images, video, and audio—into a single, unified vector space. Its small footprint allows it to run efficiently even on mobile devices, unlocking new possibilities for developers in creating AI applications that can understand and process information across various modalities. The video highlights that traditional RAG systems are predominantly text-based, leaving a vast amount of multimodal personal and organizational data unsearchable or requiring cumbersome conversion steps.

At its core, EmbeddingGemma 2 functions by processing different modalities through specialized encoders before feeding them into a shared text-based backbone. A 170-million-parameter vision encoder handles images and video frames, while a 300-million-parameter audio encoder processes sound. These encoders convert their respective inputs into tokens, which are then combined into a single sequence with text tokens. This combined sequence is processed by a 270-million-parameter backbone, based on Gemma 4, which then generates a unified 768-dimensional embedding. This architecture, combined with contrastive training on a massive mixed dataset, ensures that semantically similar items across different modalities (e.g., an image of a dog and a textual description of a dog) are positioned closely in the shared vector space, enabling seamless cross-modal search and understanding without needing intermediate transcriptions or captions.

The model offers two key deployment advantages for developers: Matryoshka Representation Learning and Modular Loading. Matryoshka allows users to truncate the full 768-dimensional embedding to smaller sizes (e.g., 128, 256, 512 dimensions), significantly reducing storage requirements for vector databases while retaining much of the model’s performance for text-based tasks. Modular loading further enhances efficiency by allowing developers to load only the specific encoders needed for their application (e.g., text-only for 270M parameters, or text + vision for 440M). Benchmarking reveals that EmbeddingGemma 2 outperforms its predecessor, EmbeddingGemma 1, particularly in code retrieval, and shows strong performance across various modalities, making it a powerful contender for real-world applications.

Practically, EmbeddingGemma 2 demonstrates impressive capabilities in tasks such as photo search (finding images from text or image queries), multilingual retrieval (understanding queries in over 100 languages), and pinpointing specific moments within videos without relying on captions. It can even search through PDF pages based purely on their visual content, bypassing the need for OCR. While its performance on abstract sound effects is noted as weaker, the overall promise of an open, multimodal embedding model that can run on-device and be fine-tuned for specific needs marks a significant step forward in making advanced AI accessible and efficient for a wider range of applications and private data scenarios.

Video Description & Links

Description

Let me know in the comments what you’d build with this, and whether you want a follow-up on fine-tuning EmbeddingGemma 2.

Colab notebook: https://colab.research.google.com/drive/1mXzvOCo-_y4r1yQ0AqAJJtU8BgU3LlCP Model card: https://huggingface.co/google/embeddinggemma-2 Launch blog: https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/ Developer guide: https://developers.googleblog.com/embeddinggemma-2-the-developer-guide/ On-device guide (LiteRT, MediaPipe): https://developers.googleblog.com/google-ai-edge-with-embeddinggemma-2/ Gemini Embedding 2 paper: https://arxiv.org/abs/2605.27295 CLIP paper: https://arxiv.org/abs/2103.00020 ImageBind paper: https://arxiv.org/abs/2305.05665 ColPali paper: https://arxiv.org/abs/2407.01449

My ColPali videos: https://www.youtube.com/watch?v=rhJJynv47Pw https://www.youtube.com/watch?v=DI9Q60T_054

LocalGPT: https://github.com/PromtEngineer/localGPT

My voice to text App: whryte.com

00:00 - EmbeddingGemma 2: Multimodal Embeddings 00:40 - Demo: Search Photos With Your Voice 01:20 - Why Multimodal Retrieval Matters for RAG 02:25 - Before: Convert Everything to Text 03:21 - Before: CLIP, ImageBind and ColPali 04:14 - One Backbone for Every Modality 04:33 - How It Works: Embeddings, Encoders, Tokens 06:07 - Mean Pooling and the 768-Number Vector 06:29 - Contrastive Training 07:15 - Matryoshka Embeddings: Smaller Vectors 08:05 - Modular Loading: 270M to 740M 09:00 - Google Colab Notebook 19:11 - Verdict

Credits: Big Buck Bunny © Blender Foundation (CC BY 3.0). ESC-50 by K. Piczak (CC BY-NC 3.0). Flickr30k test images. Benchmark numbers in the explainer are Google’s; results in the notebook section are from my own Colab T4 run.

Tags

prompt engineering, Prompt Engineer, LLMs, AI, artificial Intelligence, Llama, GPT-4, fine-tuning LLMs

URLs

Graph View