Context Window

Picture how many documents you can spread out across a desk. When studying for an exam, if your desk is too small, you have to push older notebooks onto the floor just to open a new textbook. Artificial intelligence works in a similar way—it does not permanently etch every past conversation into a deep, human-like brain.

Every time you send a new message, the AI spreads out the previous chat transcript onto its desk and reads it all over again from the start. However, this workspace has a strict limit. Once you exceed this space, most services push the oldest messages off the desk, meaning the AI can no longer see them. Some services handle this by summarizing earlier chats, while others simply stop accepting new input.

This desk limit is a major reason why an AI might forget your name or a rule you established earlier in a long conversation. In short, the context window acts like an AI's short-term working memory, keeping the conversation's context intact.

While humans typically measure reading volume by word counts or page numbers, the AI world counts in tokens. A token is the bite-sized chunk of text an AI uses to break down and understand language. In English, a standard word is roughly one token, while rarer words or punctuation marks might split into several tokens.

Crucially, your prompt (input) and the AI's reply (output) must share this exact same space. If a model's limit is 8,000 tokens and your prompt plus chat history already eats up 7,000 tokens, the AI only has room left to generate a short, 1,000-token answer.

Recently, context windows have grown dramatically, with modern models supporting hundreds of thousands—or even millions—of tokens. Thanks to this expansion, an AI can now lay an entire textbook or a hundred-page financial report flat on its desk to analyze and summarize everything at once.

If your desk were as large as a football field, you could lay out hundreds of books, but finding one specific sentence would become a serious challenge. AI researchers test this retrieval ability using 'Needle in a Haystack' evaluations. Studies have shown that when documents get very long, AI models often suffer from 'lost in the middle' syndrome, missing key details buried in the center compared to information near the start or the end.

Computing power presents another major hurdle. When the text an AI must evaluate doubles, the calculations required to map relationships between every word jump drastically. Processing huge context windows demands powerful graphics hardware, massive electrical power, and often leads to slower response times.

Expanding the context window endlessly is not a silver bullet. That is why modern AI systems increasingly pair large windows with smart search technologies that fetch only the most relevant pages and place just those onto the desk.