AI glossary

What is a token in AI?

A unit a model uses to represent text or other input.

Text tokens can correspond to whole words, parts of words or punctuation. The exact split depends on the tokenizer and language. Some models also represent other media with token-based units.

A practical example

A short sentence can contain more tokens than words because a word may be split into several pieces.

What to remember

Do not treat a token limit as an exact word limit. Input and output allowances can be separate, and product interfaces may impose additional limits.

Put it into practice

Use the answer-checking guide to keep evidence and uncertainty visible when working with AI.

Found something that needs updating?

Tools change quickly. Send a correction and include this page’s address.

Keep exploring.

View all