How AI Reads Your Words

You type a sentence. The AI doesn’t read it the way you do.

Before any processing happens, your text is broken into tokens — chunks that might be a word, part of a word, a space, or a punctuation mark. The model never sees letters. It sees a list of integers.

Why Tokens, Not Words?

Words are inconsistent. “run”, “running”, “ran” — three different words that share a root. Handling every possible form of every word in every language would require a vocabulary of millions.

Tokens are a middle ground: a fixed vocabulary of ~100,000 chunks that cover the patterns in language efficiently. Common words are single tokens. Rare words get split. Code has its own patterns. Emoji are usually one token each.

The Practical Stuff

Cost — APIs charge per token. A short message might be 20 tokens. A full document pasted into the prompt might be 5,000. This is why “be concise” is genuinely good advice when using AI.

Context limits — every model has a maximum token count it can process at once. This is the “context window.” More tokens in = more expensive and eventually impossible once you hit the limit.

Names and unusual words — common English words are single tokens. “Abhishek” might be tokenized as Ab + hi + shek. This is partly why AI can be surprisingly bad at word games, counting letters, or handling proper nouns.

Next up: What happens to those tokens once the model has them? They get compressed into meaning. That’s the context window — and it’s also why AI forgets things in long conversations.