What Is a Token in AI? (2026)

Before a model processes text, a tokenizer splits it into tokens using a learned vocabulary. Most modern models use subword tokenization (variants of byte-pair encoding), which means common words are usually a single token while rare words, long words, code, and non-English text get broken into several pieces.

A few practical consequences fall out of this. Common English words like 'the' or 'time' are one token each. A word like 'tokenization' may be two or three tokens. Whitespace and punctuation count too — a leading space is often part of a token. Emoji and many non-Latin scripts cost more tokens per visible character, so the same sentence in, say, Japanese or Arabic can use noticeably more tokens than its English equivalent.

The 4-characters-per-token rule is only a planning heuristic. For anything cost-sensitive, count real tokens with the provider's own tooling rather than estimating from word count.

Every major API prices per million tokens (MTok), and almost always charges more for output (the tokens the model generates) than for input (the tokens you send). That asymmetry is the single most important thing to internalize: a long answer can cost several times more than a long prompt of the same length.

As of June 2026, for example, GPT-5.4 is $2.50 in / $15.00 out per 1M tokens and the cheaper GPT-5.4-mini is $0.75 / $4.50 (see OpenAI pricing). Claude Sonnet 4.6 is $3 / $15 and Claude Haiku 4.5 is $1 / $5 (see Anthropic pricing). Gemini 2.5 Flash is $0.30 / $2.50 (see Google Gemini pricing). Prices change, so treat these as directional and check the live pages.

Two features cut token costs sharply when your prompts repeat. Prompt caching lets you reuse the processed form of a stable prefix — on Anthropic, a cache read is about 10% of the base input price. Batch APIs trade latency for a discount — Anthropic's Batch API is 50% off both input and output. Both are documented on the Claude API pricing detail page.

The other reason tokens matter is the context window: the maximum number of tokens — prompt plus output — a model can consider in a single request. Everything you send (system prompt, instructions, retrieved documents, conversation history) and everything the model generates has to fit inside that budget.

Run out of room and you have to truncate, summarize, or retrieve only the most relevant pieces rather than dumping everything in. That is exactly why estimating token counts up front is a core skill. For the full picture on limits and 1M-token windows, see what is a context window.

For quick planning, divide your character count by 4, or multiply your word count by about 1.33. That gets you close enough to decide whether something will fit and roughly what it will cost.

For real budgeting, count actual tokens. Providers expose tokenizers and token-counting endpoints, and most SDKs report exact input/output token usage on every response — log those numbers and you will know your true cost per call. When in doubt, measure on your own representative text rather than trusting a generic ratio, because code, JSON, and non-English content all skew the estimate.