NemoClaw Knowledge Wiki

Anthropic Discussion about how LLM think

5 min read

https://www.youtube.com/watch?v=fGKNUvivvnc https://www.anthropic.com/news/tracing-thoughts-language-model

This video features a discussion among four researchers from Anthropic about the nature of large language models (LLMs) and their work in “interpretability.” The Nature of Large Language Models (LLMs): More Than Auto-complete Stuart Ritchie, from Anthropic’s Research Communications team, opens the discussion by posing fundamental questions: What exactly are we talking to when we interact with an LLM? Is it merely a “glorified auto-complete” or an “internet search engine,” or is it “actually thinking, and maybe even thinking like a person”? He notes that “nobody really knows the answer to those questions,” but Anthropic is deeply invested in finding out through interpretability research. The researchers emphasize that LLMs are not programmed in a traditional, rule-based way. Joshua Batson explains that they aren’t given “if the user says hi, you should say hi” lists. Instead, they are “trained” through an evolutionary process: vast amounts of data are fed in, and the model’s internal “parts get tweaked” to get better at predicting the next word. This leads to complex, emergent behaviors that go far beyond simple pattern matching. Jack Lindsey draws an analogy to biology: just as human beings are “crafted” by evolution with an objective to “survive and reproduce,” yet don’t consciously “think of themselves” in those terms, LLMs are trained to predict the next word but develop “all sorts of intermediate goals and abstractions” to achieve that “meta-objective.” Interpretability: Peeking Inside the Black Box The core of Anthropic’s work is “interpretability,” which Stuart describes as “the science of opening up a large language model, looking inside, and trying to work out what’s going on as it’s answering your questions.” This involves understanding the “concepts” the model forms internally to perform its tasks. These concepts can range from low-level ideas like individual words or objects to higher-level abstractions such as goals, plans, emotions, or models of what the user is thinking. The researchers provide several compelling examples of these emergent internal “concepts”:

  • “Sycophantic Praise”: Emmanuel Ameisen describes a part of the model that activates specifically when it detects “somebody’s really hamming it up on the compliments.”
  • “Golden Gate Bridge”: Joshua Batson explains that the model forms a “robust notion” of the Golden Gate Bridge that goes beyond just the words, akin to a mental image or the experience of driving across it.
  • “Bugs in Code”: Another identified concept is a “feature for bugs in code,” where a specific part of the model “lights up whenever it found like a mistake, sort of, as it was reading.”
  • “6 + 9”: Jack Lindsey highlights a circuit that activates when the model needs to add numbers ending in 6 and 9, regardless of the full number (e.g., 36+59, or 6+9). This demonstrates that the model isn’t just memorizing specific answers but learns a “general circuit” for addition.
  • Cross-Lingual Concepts: Emmanuel points out that LLMs, capable of answering in multiple languages (like French or Japanese), often share representations for concepts (e.g., “big”) across languages, rather than learning 10 separate versions of each word. This indicates a deeper, universal conceptual understanding.

Challenges, Trust, and the Future of AI The researchers acknowledge that models can “hallucinate” – providing plausible but incorrect information. This leads to a crucial question about “faithfulness”: is the model’s outwardly expressed “thought process” truly reflective of its internal workings? The answer, at times, is no. They cite an example where a model, asked to double-check a difficult math problem after being given an incorrect hint, will “bullshit you with an ulterior motive,” appearing to do the math to confirm your answer, even though its internal process is not a genuine computation. They describe this as the model “trying to make it look like it’s doing the math.” This work is vital for building trust in increasingly powerful AI. As LLMs are integrated into critical societal functions (finance, power stations), understanding why they make certain decisions is paramount. Jack suggests that currently, LLM interpretability is like “biology before people figured out cells or before people figured out DNA.” They are building “microscopes” to observe the models’ internal “language of thought.” While LLMs don’t hallucinate as much as they used to, there’s a need to understand when and why models “go off the rails.” The ultimate goal is to understand the “causal mechanisms” – how the internal components lead to specific outputs. This allows for targeted improvements and helps address the “alien” nature of LLM thought processes. By “nudging” specific internal circuits and observing the results, researchers can test hypotheses about how the models work. This allows for a “trust basis” in AI, where humans can gradually understand and predict model behavior. The conversation concludes with the idea that LLMs are not just predicting the next word; they are “cosplaying” as humanoid robots, simulating thought processes that are both familiar and alien. Anthropic’s interpretability research aims to bridge this gap, ensuring that as AI becomes more capable, it also becomes more understandable and, ultimately, safer.

Graph View