On-Prem AIGlossary

What Is Token?

Also known as: LLM Token, Text Token

Definition

A token is the basic unit of text a language model processes, usually a word fragment rather than a whole word. In English, one token averages roughly four characters, so about 100 tokens covers 75 words.

Token Explained

Tokenizers are built by learning which character sequences appear together most often in a training corpus, typically using byte-pair encoding or a similar subword algorithm. Frequent words become single tokens, rare words split into several pieces, and unusual strings can fragment heavily. The vocabulary is fixed at training time, commonly somewhere between 32,000 and 250,000 entries depending on the model family and how many languages it targets.

Token counts are not proportional to character counts in the way people assume. Ordinary English prose sits near the four-characters-per-token average. Code, JSON, and XML tokenize far less efficiently because braces, indentation, and punctuation each consume tokens. Part numbers, serial numbers, and alphanumeric identifiers fragment badly: a string like 44-7192-AX may consume six or more tokens. Non-English text, especially non-Latin scripts, often uses two to four times more tokens for the same meaning.

Everything that matters commercially is measured in tokens. Context windows are expressed in tokens, cloud API pricing is per token, throughput is tokens per second, and KV cache memory scales with token count. Because output tokens are generated one at a time while input tokens are processed in parallel, output is generally the more expensive side in both latency and cloud pricing, which is why concise output instructions have a real effect on cost.

For estimation, useful rules of thumb hold well enough. A page of prose is roughly 500 to 700 tokens. A typical work instruction or procedure runs a few thousand. A long technical specification can reach tens of thousands. Multiply by the number of retrieved chunks a RAG prompt carries and the arithmetic of context budget becomes concrete quickly. Always verify with the actual tokenizer for your model family, since counts differ meaningfully between families.

Why It Matters

Context window limits are token limits, so document chunking and retrieval budgets must be planned in tokens rather than pages.
Cloud AI pricing is per token, making token efficiency a direct line item rather than an abstract engineering concern.
Part numbers and identifiers fragment into many tokens, quietly consuming context in exactly the ERP workloads that rely on them.
KV cache memory scales with tokens per request, tying prompt length directly to how many concurrent users a GPU can serve.

In Practice

A team estimated that fifty retrieved records would fit comfortably in a 32,000-token window based on character count. In practice the records were JSON with long field names and alphanumeric part codes, tokenizing at nearly double the expected rate, and half the records were silently truncated out of the prompt.

Frequently Asked Questions

How many tokens are in a page of text?

A standard page of English prose, roughly 400 to 500 words, comes to about 500 to 700 tokens. Dense technical documents with tables, part numbers, and specifications run higher, sometimes 1,000 or more per page. Code and structured data such as JSON can double these figures because punctuation and identifiers tokenize inefficiently.

Why do input and output tokens cost different amounts?

Input tokens are processed in a single parallel prefill pass, which uses the GPU efficiently. Output tokens are generated sequentially, each requiring a full pass over the model weights, so they cost far more compute time per token. That is why cloud providers typically price output tokens several times higher than input tokens and why limiting response length reduces cost meaningfully.

Related Terms

Context Window

A context window is the maximum number of tokens a language model can hold in view at one time, covering the system instructions, conversation history, retrieved documents, and the generated response combined.

LLM (Large Language Model)

An LLM (Large Language Model) is a neural network with billions of parameters trained on huge volumes of text to predict the next token in a sequence, which gives it broad language understanding, reasoning, and generation ability.

Inference

Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.