AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
Language models don't read words. They read tokens: chunks produced by a tokenizer, which can be whole common words, pieces of rarer words, punctuation, digits or bytes. Common English words often take one token each. Rare words, scientific names, numbers, dates, code and many non-English scripts get split into several. So the same 100 words can cost very different numbers of tokens depending on the text, and different models use different tokenizers, so counts aren't portable between them. Context limits, prices and rate limits are set in tokens, so count with the right tool rather than estimating from words.
The evidence
How tokenizers split text (documented behavior, OpenAI tiktoken README, last checked 2026-10-16). OpenAI's open-source tokenizer, tiktoken, uses byte pair encoding (BPE). The README explains that "language models don't see text like you and I, instead they see a sequence of numbers (known as tokens)." BPE "works on arbitrary text, even text that is not in the tokeniser's training data," and "on average, in practice, each token corresponds to about 4 bytes." It "attempts to let the model see common subwords... BPE encodings will often split 'encoding' into tokens like 'encod' and 'ing'" (openai/tiktoken README; read).
Tokenizers change between model generations (documented behavior, Anthropic docs, last checked 2026-10-16). Anthropic says that "Claude 4.7 and later models ... use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content." Its advice is to "recount prompts against the model you plan to use rather than reusing counts measured against earlier models." It also notes that "the token count is an estimate" that may "differ by a small amount" from actual usage, and that counts can include system-added tokens that are not billed (Anthropic, Token counting; read).
A small box test (observed result, tiktoken 0.14.0 with the o200k_base encoding, run 2026-10-16). Words here are counted as whitespace-separated strings:
| Text | Words | Tokens |
|---|---|---|
| "The honey bee colony survived the winter." | 7 | 8 |
| "Varroa destructor mites parasitize Apis mellifera brood." | 7 | 14 |
| "2026-10-16 14:05 UTC, $1,234.56" | 4 | 18 |
| "Die Honigbienenvölker überwinterten erfolgreich." (German) | 4 | 11 |
| "ミツバチの群れは冬を越した。" (Japanese, no spaces) | 1 | 13 |
Plain English came close to one token per word. Latin names roughly doubled the count. A date, time and price string used 18 tokens for 4 "words". The Japanese sentence counts as one "word" by spaces but 13 tokens. This is a five-sentence illustration for one tokenizer, not a general rate.
Why it matters in practice (explanation)
- Context budgets: a document's word count can understate its token size if it is full of tables, numbers, code or names.
- Cost and rate limits are metered in tokens.
- Cross-model comparisons: "this prompt is 2,000 tokens" is only true for one tokenizer.
- Rules of thumb ("about ¾ of a word per token") hold best for ordinary English prose.
How to count properly (editorial suggestion)
- Use the provider's own counter: Anthropic's token-counting endpoint, or tiktoken with the encoding for your OpenAI model.
- Count with the exact model you will call, especially after model upgrades.
- Count the whole request: system prompt, conversation, documents, tool definitions, plus room for the output.
- Keep a margin, since counts are estimates.
Limits
- Exact ratios vary with text, language and tokenizer. The table is a single test, not a benchmark.
- Provider tokenizer details change; claims are as documented on 2026-10-16.
Reader takeaway
A token is a model's unit of text, not a word. Numbers, names, code and many non-English scripts use more tokens per word. Count with the provider's tool for the model you'll actually use.
Sources read for this answer
- OpenAI. tiktoken README. https://raw.githubusercontent.com/openai/tiktoken/main/README.md (last checked 2026-10-16)
- Anthropic. Token counting. https://platform.claude.com/docs/en/build-with-claude/token-counting (last checked 2026-10-16)
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.