Context Window
The maximum number of Tokens an LLM can process within a single Inference cycle, representing the model’s functional “working memory.”
Core Mechanics
- Capacity: Defines the boundary of information the model can “attend to” simultaneously.
- Complexity: Historically limited by the Attention Mechanism, where computation scales quadratically with sequence length.
- Limitations: Finite capacity necessitates external memory strategies for long-term retention and complex reasoning tasks.
Persistent Memory & Librarian Systems
To overcome the ephemeral nature of the context window, local AI agents increasingly rely on externalized memory architectures:
- Librarian System: A specialized subsystem that manages persistent, evolving memory for local AI, decoupling long-term storage from immediate inference context Developing Persistent, Intelligent Memory for Local AI with a Librarian System.
- Integration: These systems allow agents to retrieve relevant historical data dynamically rather than loading entire histories into the context window, optimizing token usage.
- Standards: Protocols like OKF facilitate structured interoperability between these external knowledge bases and the LLM.