Document Utilities

Tokenization & BPE Explained

Language models do not read words the way humans do. Before inference, text is split into tokens, which can be full words, sub-words, punctuation fragments, or byte-like units. Byte-Pair Encoding (BPE) is one of the most widely used approaches for this segmentation.

Why "ChatGPT" Might Be 2-3 Tokens

BPE builds a vocabulary from frequent symbol pairs. Common words may become single tokens, while rare or compound terms are broken into smaller reusable pieces. Depending on tokenizer version and training corpus frequencies, "ChatGPT" can be represented as chunks like "Chat", "G", and "PT" instead of one token.

How BPE Works (Simplified)

  1. Start with small units (characters/bytes).
  2. Count frequently co-occurring adjacent pairs.
  3. Merge the most frequent pair into a new symbol.
  4. Repeat until desired vocabulary size is reached.

Visual Token Breakdown

Word: "ChatGPT"
Tokenized As
Chat
G
PT
Or Sometimes
Ch
at
GPT

Practical Impact for Builders

  • Cost: Billing is usually token-based, not word-based.
  • Limits: Context windows cap total tokens (prompt + response).
  • Latency: More tokens means more compute steps and longer generation time.
  • Prompt design: Compact prompting often improves both speed and reliability.
# Token budgeting mindset
Total context budget = input tokens + output tokens + system instructions

If prompt size grows too large:
- latency increases
- truncation risk rises
- answer quality may degrade
Educational takeaway: Tokenization is the accounting layer of LLM systems. Understand it early to avoid hidden performance and cost surprises.