RolloAcademy
Under the hood

Token

A chunk of text the AI reads and counts, roughly three-quarters of a word. Tokens drive limits, cost, and speed.

A word split into tokens The word unbelievable broken into three token chunks, with a rule of thumb underneath. ONE WORD, SPLIT INTO TOKENS Un believ able ~4 characters ≈ 1 token · about ¾ of a word

A token is a small piece of text. AI does not read whole words; it breaks text into tokens, each roughly four characters or three-quarters of a word. A long word can split into several tokens, and a page is about 500.

Tokens are the unit working behind the scenes: the context window, speed, and cost are all measured in them. You rarely count tokens by hand, but knowing they exist explains why very long chats and huge files start to strain.

Often confused with: A word. Most short words are one token, but long or unusual words split into several, and spaces and punctuation count too.

See it in a lesson: Context and tokens

Related terms

Context windowContext

Further reading

← All terms