Vector Databases
Specialized databases optimized for storing, indexing, and retrieving high-dimensional vector embeddings. Enable efficient similarity search (e.g., nearest neighbor queries) for applications like semantic search, recommendation systems, and LLM-powered retrieval.
Core Functionality
- Approximate Nearest Neighbor (ANN) Search: Uses algorithms like HNSW, IVF, or FAISS for scalable similarity matching.
- Embedding Support: Stores vectors generated from text, images, or other modalities via models like BERT, CLIP, or Sentence Transformers.
- Scalability: Handles millions/billions of vectors with low-latency queries.
Key Limitations
- Model Constraint: Requires the same embedding model for both vector generation and retrieval to ensure semantic consistency.
- Context Loss: Pure vector search may lack structured metadata filtering capabilities found in traditional relational databases.
Integration with Knowledge Standards
- Open Knowledge Format (OKF): Emerging standards like Google’s OKF aim to standardize personal knowledge bases for AI agents, evolving from concepts like Andrej Karpathy’s “LLM Wiki” to enable better interoperability. See Google’s OKF: Standardizing Karpathy’s LLM Wiki for AI Interoperability for details on this shift.
- Structured Retrieval: Combining vector databases with standardized formats allows for hybrid retrieval systems that leverage both semantic similarity and structured metadata.