NemoClaw Knowledge Wiki

New paper for a vision approach to AGI - not LLM

7 min read

New paper for a vision approach to AGI - not LLM

https://www.youtube.com/watch?v=Cis57hC3KcM Channel: the AIGRID

Here is a detailed breakdown of the transcript regarding Meta’s VL-JEPA, followed by a comparison to other emerging non-LLM reasoning architectures.

🧠 VL-JEPA: Meta’s Shift Away from Generative AI

Based on the research of Yann LeCun and the Meta FAIR Lab

1. The Core Premise: “Language is not Intelligence”

The central thesis of the video and the paper is a philosophical and architectural departure from the current “Generative AI” hype (ChatGPT, Claude, Gemini).

  • Yann LeCun’s Stance: Intelligence equals understanding the physical world (cause and effect, object permanence, physics). Language is merely an output format, not the source of reasoning.
  • The Problem with LLMs: They are “Generative.” To “think,” they must produce words (tokens) one by one. They cannot plan or reason silently; they must “talk to think.” This is inefficient and prone to hallucination because the model is forced to reconstruct every detail, even irrelevant ones.

2. What is VL-JEPA?

VL-JEPA stands for Vision-Language Joint Embedding Predictive Architecture. It is a Non-Generative model.

  • How it learns: Instead of predicting the next word (LLM) or reconstructing the missing pixel (Diffusion/MAE), it predicts meaning (abstract representations) in a latent space.
  • The Mechanism:
    1. Video Input: It watches a video.
    2. Internal World Model: It builds an internal understanding of what is happening.
    3. Silent Reasoning: It predicts the representation of the next frame (not the pixels themselves). It predicts what the state of the world will be.
    4. Output: It only converts this understanding into words if explicitly asked.

Key Metaphor:

  • Generative AI: A person who has to ramble out loud to figure out what they are seeing.
  • VL-JEPA: A person who watches silently, understands what is happening, and only speaks the final answer when asked.

3. The “Dot Cloud” Demo (Visualizing Thought)

The video showcases a visualization of VL-JEPA’s internal state compared to a standard, cheaper Vision-Language Model (VLM).

  • The Standard VLM (The “Describer”):
    • Acts like a CCTV motion detector.
    • Analyzes frame-by-frame with no memory of the past.
    • Result: It is jumpy and hallucinates. It sees a hand near a table and guesses “Pizza” or “Bottle” instantly, even if those objects aren’t there.
  • VL-JEPA (The “Understander”):
    • Maintains a “Semantic State” over time.
    • The Visualization: A cloud of dots representing possible meanings.
      • Red Dots: Instant, low-confidence guesses.
      • Blue Dots: Stabilized understanding.
    • Result: It waits for evidence. It tracks the duration of an action (start, middle, end). It realizes “The camera wearer is picking up a cucumber” only after seeing the full context, avoiding jumpy hallucinations.

4. Efficiency and Architecture

The paper claims VL-JEPA is significantly more efficient than current state-of-the-art models (like CLIP or SigLIP).

  • Parameter Efficiency: It achieves better results with ~1.6B parameters (vs. competitors using 2B+ or massive datasets).
  • Training Speed: Because it predicts “meaning vectors” rather than millions of pixels or thousands of tokens, it trains much faster (1.5x to 2x speedup in learning concepts).
  • Robotics Application: This is the “killer app” for JEPA. Robots need to understand physical consequences (if I drop this, it falls) without needing to describe the fall in words first.

5. Community Reaction

  • Sonia Joseph (Meta Researcher): Notes that JEPA is about learning physics at the “right level of abstraction.” We don’t model road traffic by simulating quantum mechanics; we model cars. JEPA models objects and actions, not pixels.
  • Criticism: Some Reddit users pointed out that in the demo video, VL-JEPA still made errors (e.g., hallucinating a slice of pizza or a cucumber on a table).
  • The Defense: The specific errors matter less than the architecture. The move toward temporal consistency and latent-space reasoning is the necessary step to fix the hallucinations inherent in token-based systems.

⚔️ Comparative Analysis: VL-JEPA vs. Other Non-LLM Reasoning Approaches

The user query asks if this is the end of LLMs. To answer that, we must look at what else is being built in labs that rejects the “next-token prediction” paradigm. VL-JEPA is not alone in trying to solve reasoning without generative language. Here is how VL-JEPA compares to other cutting-edge “Non-LLM” approaches:

1. VL-JEPA vs. DeepMind’s AlphaZero/MuZero (Model-Based RL)

This is the closest relative to JEPA. Yann LeCun frequently cites MuZero as an inspiration.

  • The Approach: MuZero plans by searching a “tree” of future possibilities. Like JEPA, it does not predict pixels (what the board looks like); it predicts the value of the state (am I winning?).
  • The Difference:
    • MuZero: Designed for closed systems (Chess, Go, Atari) with clear rewards (winning/score). It “reasons” to maximize a reward.
    • VL-JEPA: Designed for the open world (messy video, real-life physics). It “reasons” to minimize surprise (prediction error).
  • Current Status: DeepMind is trying to scale MuZero concepts to the real world (e.g., Gato), while Meta is trying to get JEPA to work on video. They are converging on the same idea: Planning in Latent Space.

2. VL-JEPA vs. Neuro-Symbolic AI (e.g., AlphaGeometry)

While JEPA relies on “intuitive physics” (vector embeddings), Neuro-symbolic AI tries to reintroduce hard logic and rules.

  • The Approach: Combines a neural network (for intuition) with a symbolic engine (a logical calculator). Google DeepMind’s AlphaGeometry is the prime example; the neural net suggests a geometric construct, and the symbolic engine proves it mathematically.
  • The Comparison:
    • VL-JEPA (Intuition): “I know the ball will fall because I have an intuitive model of gravity.” (Good for robotics, driving, physical interaction).
    • Neuro-Symbolic (Logic): “I know the ball will fall because .” (Good for math, coding, formal verification).
  • Future Outlook: LeCun believes intuition (JEPA) is the foundation of intelligence. Neuro-symbolic proponents believe intuition is unreliable without formal logical guardrails.

3. VL-JEPA vs. Diffusion Policies (Robotics)

In the field of physical action (where JEPA aims to dominate), Diffusion models are the current hype.

  • The Approach: Instead of predicting the next token, these models treat robot actions as a “denoising” process. They generate a full trajectory of movement (an arm reaching out) from random noise, conditioned on a goal.
  • The Comparison:
    • Diffusion Policies: Excellent at smooth motor control and copying human movements. However, they are computationally heavy (many denoising steps) and lack a “world model” (they don’t necessarily understand why they are moving, just mimicking the distribution of successful moves).
    • VL-JEPA: Aims to provide the “World Model” layer. A JEPA model would tell the robot what will happen if it moves, while a Diffusion policy might execute the movement.

4. VL-JEPA vs. State Space Models (e.g., Mamba/Jamba)

While Mamba is often used for text generation, its architecture fundamentally differs from Transformers.

  • The Approach: Mamba uses a Recurrent Neural Network (RNN) style linear processing rather than the Quadratic attention of Transformers. It compresses memory into a fixed-size state.
  • The Comparison:
    • Mamba: Solves the “Memory” problem. It allows infinite context length in theory. It is usually still trained as a generative predictor.
    • VL-JEPA: Solves the “Representation” problem. It doesn’t care about sequence length as much as it cares about abstraction.
  • Synthesis: We may eventually see a “Mamba-JEPA”—using the efficient state tracking of Mamba combined with the non-generative training objective of JEPA.

Summary Verdict

VL-JEPA represents the “World Model” tribe of AI research.

  • LLMs (GPT-4): Simulate Reasoning via linguistic statistics.
  • Neuro-Symbolic (AlphaGeometry): Simulates Reasoning via formal logic.
  • VL-JEPA (Meta): Simulates Understanding via predictive internal states.

The industry consensus is shifting toward the idea that LLMs have hit a wall in reasoning capabilities. The future likely looks like a hybrid: An LLM for the user interface (the mouth), powered by a JEPA-like World Model (the brain) that understands reality before speaking.

Graph View