NemoClaw Knowledge Wiki

Multimodal AI: Concepts, Approaches, and Data Processing by LLMs

3 min read

Multimodal AI: Concepts, Approaches, and Data Processing by LLMs

Clip title: What is Multimodal AI? How LLMs Process Text, Images, and More Author / channel: IBM Technology URL: https://www.youtube.com/watch?v=J51oZYcNvP8

Summary

The video, presented by Martin Keen of IBM, introduces and explains the concept of Multimodal AI. It begins by defining “modality” in the context of AI as a data modality, referring to different types of data such as text, images, audio, lidar, and thermal imaging. Multimodal AI models are distinguished by their ability to ingest and/or generate multiple data modalities, moving beyond the single-modality limitations of earlier AI systems like Large Language Models (LLMs) that primarily process text.

Keen illustrates two primary approaches to achieving multimodality. The first, Feature-Level Fusion, involves connecting separate, specialized models. For example, a text-based LLM might be paired with a vision encoder. The vision encoder processes image data, extracts numerical features (a feature vector), and then passes these summarized features to the LLM. While still used for specialized enterprise tasks due to cost-effectiveness and modularity, this method has a significant drawback: information can be lost or compressed during the transfer between models. The LLM only “sees” a numerical description of the image, not the raw visual data itself.

The second, more advanced approach is Native Multimodality. This method integrates different data types into a single, unified model by embedding all modalities into a “shared vector space.” In this space, different types of data that represent similar concepts (e.g., the word “cat” and an image of a cat) are positioned closely together. This eliminates the need for separate models and intermediate translations, allowing the AI to reason about all modalities cohesively and without significant information loss.

The video further explores the application of native multimodality to video data, introducing the concept of Temporal Reasoning. Older methods for processing video involved sampling individual frames and running them through a vision encoder, often losing the crucial temporal context of motion and sequence. Native multimodal models, however, embed video with its temporal dimension intact. They process “spatial-temporal patches,” essentially 3D cubes of information that capture both visual data and movement over a short window of time. This means the model doesn’t have to infer motion; it’s inherently part of the tokenized data.

A significant advantage of native multimodal models, especially those incorporating temporal reasoning, is their capability for Any-to-Any Generation. This allows the model to accept any combination of input modalities (e.g., text, image, video) and generate coherent output in any combination of modalities. For instance, one could ask the model (via text and an image of a phone problem) how to fix something, and it could respond with text instructions and a generated video demonstrating the solution. This holistic understanding and generation across diverse data types represent the gold standard for multimodal AI today, enabling more comprehensive and intuitive interactions.

Graph View