What is a token?
A token is a unit used when a model processes content. In text, a tokenizer divides the input into units that are not a fixed one-to-one match with words or characters.
One token is not guaranteed to equal one English word or one Chinese character. The same passage may be divided differently by different tokenizers.
Why word count does not determine token count
Tokenization can use words, characters or subword units. The Hugging Face source explains these approaches. Counting the visible words therefore does not necessarily tell you the model's input size.
Providing full retrieved pages usually sends more content to the model than providing selected passages. Selection can reduce the amount processed, but removing context does not guarantee a better answer.
Separate model usage from retrieval costs
The Claude documentation describes counting tokens before a request to estimate usage. An estimate still needs the model and input scope to be meaningful.
Search API requests, model processing and other workflow operations may have separate charges. A reduction in one component does not establish the total cost of the task.
Sources and scope
Source-language entry records a source check on 2026-10-21. This adaptation preserves that provenance and does not represent a new audit of the cited pages.