Definition
A token is the basic chunk of text that an AI model uses to understand and process human language. Computers can't do math on letters directly, so the text is split into small pieces called tokens, and each piece gets its own ID number to calculate with.
The bite-sized pieces AI chews on
You cut a long loaf of bread into slices before eating it. AI does something similar. To turn text into numbers it can calculate with, it first needs pieces of a set size. So it cuts each sentence into pieces a computer can handle, and each piece is called a token.
A token isn't always one word or one letter. A short word like "apple" is a single token. But a longer word like "unbelievably" may be split into several pieces, such as "un", "believ", and "ably", based on letter groups that often show up together.
Languages get split differently, too. Many common English words become one token, or just a few. Korean's Hangul has over 10,000 possible syllable characters, and many of them are rare in training data. So, depending on the tokenizer, a single rare character may be split into several tokens.
Just as we understand a sentence by breaking it into words, AI handles text by breaking it into tokens. Splitting text into the right pieces makes the next steps, analyzing and generating, much easier.
How text pieces turn into numbers
AI can't work with letters as they are. It has to turn them into numbers first. So after splitting text into tokens, it opens up a giant vocabulary list and gives each piece its own ID number, like a claim ticket.
Say the token "apple" is listed as number 4201. The AI doesn't receive the word "apple." It receives 4201 and starts its complex math from there. This whole process of splitting text into tokens and converting them to numbers is called tokenization.
When the AI writes a reply, the same idea works in reverse. It uses math to find the most likely next number, then looks up that number and sticks the matching text piece back on, building a finished sentence one piece at a time.
Even in the few seconds you spend chatting with a bot, a huge number of text pieces are being turned into numbers and put back together as words, over and over, behind the scenes.
A bit more precisely
To be a bit more precise, tokens are both the billing unit for AI services and the yardstick for how much an AI can "remember." When you build a chatbot or use an API, what you pay isn't based on plain character counts. It's based on the number of input tokens and output tokens. Input and output usually have different prices.
The length of conversation a model can hold in mind at once is also strictly capped by token count. This limit is called the context window. Some models can read several books' worth of tokens at once, while others forget the start of a conversation after just a few pages.
That's why a smart way of splitting text, grouping frequently used letter patterns into single tokens, matters so much. It speeds up the AI's work and cuts running costs. Even for sentences with the same meaning, the fewer tokens it takes to express them, the faster and cheaper the AI can run.
This is also why cutting-edge AI research focuses on algorithms that compress many languages into fewer tokens.
Common misconceptions
One token is exactly one word or one letter.
A token can be smaller or bigger than a word. One English word can become several tokens, and depending on the tokenizer, a single rare Korean character can be split into as many as 2 to 3 tokens.
Where you meet it
The longer the question you type into an AI chatbot, the more tokens it uses.
When you build a program with an AI API, you pay for the number of tokens you send and receive.
In one sentence
A token is the basic small piece of text that AI splits language into so it can calculate with it and understand it.