Bigram Language Model
A Bigram Language Model is a statistical Language Model that predicts the next token in a sequence based solely on the immediately preceding token. It operates under the Markov assumption that the probability of a word depends only on the previous word, ignoring longer-range context.
Core Mechanics
- Probability Calculation: Estimates by counting co-occurrences in a training corpus.
- Vocabulary: Requires a defined set of unique tokens (words or characters).
- Generation: Samples from the conditional probability distribution of the next token given the current one.
- Limitations: Lacks long-term memory; cannot capture complex syntactic structures or semantic dependencies beyond adjacent pairs.
Role in Modern NLP
- Serves as the pedagogical foundation for understanding Transformer architectures and GPT models.
- Demonstrates the basic principle of next-token prediction, which scales to deep neural networks via Attention Mechanisms.
- Often used in introductory tutorials to visualize how language models learn distributions from text data.