NemoClaw Knowledge Wiki

tokens

2 min read

🗂️ AI & Agents · View mindmap

Tokens

Tokens are the fundamental units of text that language models like Claude process and generate. They represent small chunks of text—typically individual words, subwords, or punctuation marks—that the model breaks input into before processing. Understanding tokens is essential when working with AI agents and Claude Skills, as token usage directly impacts both the cost and performance of API calls.

How Tokens Work

When you send text to Claude, the model first converts it into tokens using a tokenizer. A single token might represent a common word like “hello,” or it might be a smaller unit like “ing” or “‘s.” On average, one token is roughly equivalent to 4 characters of text in English, though this varies depending on the specific characters and language used. The model processes these tokens sequentially to understand context and generate responses.

Practical Implications

Token counting matters for both budgeting and optimization. API costs are calculated based on the number of tokens in your input and output, making token efficiency important for scaling agents. When implementing Claude Skills or other agentic patterns, monitoring token usage helps identify whether prompts, instructions, or context windows can be optimized. Tools like the token counter API allow developers to measure token consumption before making calls, enabling better cost prediction and resource allocation.

Source Notes

Graph View

Backlinks