Agentic Search
Agentic search is an information retrieval approach where autonomous agents actively explore file systems or knowledge bases to locate relevant information, rather than relying on static, pre-indexed retrieval methods. In this model, an agent navigates through documents and directory structures iteratively, evaluating relevance based on discovered content and deciding which paths to explore next. This transforms information discovery from a lookup operation into an exploration process, allowing agents to adapt their search strategies dynamically as new context becomes available.
This methodology contrasts with traditional Retrieval-Augmented Generation systems, which typically depend on fixed embeddings and vector databases. By treating search as a sequential decision-making problem, agentic search can handle complex queries that require traversing hierarchical structures or synthesizing information from disparate sources without prior indexing. The process involves continuous feedback loops where the agent assesses the relevance of retrieved chunks and refines its search strategy.
Key Insights & Evolution
- Lexical vs. Semantic: While vector search dominates modern rag implementations, lexical methods remain critical in agentic-search contexts.
- BM25 Resurgence: Recent analysis highlights the “unreasonable effectiveness” of BM25 for agentic-search, defined specifically as “search inside an agent loop” BM25’s Unreasonable Effectiveness in LLM-Driven Agentic Search.
- Agent Loop Integration: Unlike static retrieval, agentic search leverages BM25’s precision within the iterative decision-making process of an ai-agent, allowing for more robust handling of specific entity names and technical terms that embeddings might obscure.
- Hybrid Approaches: Effective fs-explorer implementations often combine the semantic understanding of LLMs with the exact-match precision of BM25 to optimize the exploration phase.
References
- Jo Kristian Bergum, CEO of Hornet Dev. “The unreasonable effectiveness of BM25 for agentic search.” BM25’s Unreasonable Effectiveness in LLM-Driven Agentic Search