Lexical Scoring Function
A mathematical function used in Information Retrieval (IR) to calculate the relevance of a document to a given query based on the frequency and distribution of terms. While modern search often relies on dense vector embeddings, lexical scoring functions remain critical for precision, interpretability, and handling rare or compound terms.
Core Concepts
- Term Frequency (TF): Measures how often a term appears in a document.
- Inverse Document Frequency (IDF): Measures how rare a term is across the entire corpus.
- Sparse Representations: Relies on explicit term matching rather than semantic proximity.
BM25: The Standard Lexical Scorer
bm25 (Best Matching 25) is the most widely used lexical scoring function. It improves upon basic TF-IDF by:
- Normalizing document length to prevent bias toward longer documents.
- Using saturation functions to dampen the impact of extremely high term frequencies.
Recent Developments in Agentic Search
Despite the dominance of dense retrieval, lexical methods are seeing a resurgence in agentic-search contexts.
- Unreasonable Effectiveness: Recent analysis highlights that BM25 remains highly effective for “search inside an agent loop,” often outperforming or complementing vector-based approaches in specific agentic workflows BM25’s Unreasonable Effectiveness in LLM-Driven Agentic Search.
- Hybrid Approaches: Modern agentic systems often combine lexical scoring for exact match precision with vector search for semantic generalization.
- Interpretability: Lexical scores provide transparent reasoning for why a document was retrieved, which is crucial for debugging agent behavior.
Comparison with Dense Retrieval
| Feature | Lexical (BM25) | Dense (Vector) |
|---|---|---|
| Basis | Term frequency & rarity | Semantic embedding proximity |
| Strengths | Exact matches, rare terms, speed | Semantic understanding, synonymy |
| Weaknesses | Vocabulary mismatch, no semantics | Computationally heavier, less interpretable |
References
- Jo Kristian Bergum, “The unreasonable effectiveness of BM25 for agentic search,” Hornet.dev. BM25’s Unreasonable Effectiveness in LLM-Driven Agentic Search