GPT-3
GPT-3 (Generative Pre-trained Transformer 3) is a large language model developed by openai. It utilizes a transformer-based architecture trained on a massive dataset to generate human-like text.
Technical Characteristics
- Architecture: Transformer decoder-only model with 175 billion parameters.
- Tokenization: Uses byte-pair-encoding (BPE) for text processing.
- Capabilities: Text generation, translation, summarization, and code completion.
Known Phenomena & Anomalies
Glitch Tokens
Certain input strings can trigger anomalous model responses, often referred to as “glitch tokens.” These are seemingly innocuous inputs that cause the model to produce bizarre, nonsensical, or degraded outputs.
- Mechanism: Linked to specific interactions within the byte-pair-encoding tokenizer and the model’s internal representation of rare or adversarial token sequences.
- Impact: Can lead to sudden drops in coherence or generation of garbage text.
- Research: Detailed analysis of these anomalies is available in LLM Glitch Tokens: Byte Pair Encoding and Anomalous Model Responses.