Tokens and Context Windows Explained: A Simple LLM Guide

A context window is the amount of text an AI model can consider at one time, measured in tokens. Tokens are the small chunks of text a model reads and writes, often a piece of a word. Everything the model works with in a single request, including your instructions, the conversation so far, any pasted documents, and its own reply, has to fit inside that window.

Understanding these two ideas explains a lot: why long chats get forgetful, why some tasks cost more than others, and why AI agents work the way they do. Here is the simple version.

What is a token?

Language models do not read letters or whole words. They read tokens, which are chunks of text from a fixed vocabulary the model learned to use.

A token might be:

• A whole common word, like “house”
• Part of a longer word, like “un” + “believ” + “able”
• A space plus a word, a punctuation mark, or a symbol
• A fragment of code, a number, or a piece of a non-English word

The exact split depends on the model’s tokenizer, and different model families tokenize differently. As a rough rule of thumb for English prose, one token is a bit less than one word, so a page of text is several hundred tokens. Treat that as an estimate, not a conversion rate.

Why tokens, not words?

Tokens give the model a manageable vocabulary that can still spell out anything. Common words get a single token, which is efficient. Rare words, names, and typos can still be represented by combining smaller pieces. This is also why models sometimes stumble on tasks like counting letters in a word: they never see individual letters, only tokens.

What uses more tokens

Some content is “token-heavy” compared with plain English:

• Code, especially with lots of symbols, indentation, and long identifiers
• Numbers and data, such as long tables, IDs, or logs
• Many non-English languages, depending on the tokenizer
• Unusual formatting, like repeated characters or long URLs

What is a context window?

The context window is the model’s working memory for a single request. It is the maximum number of tokens the model can take in and produce together.

Think of it like a desk. Everything the model needs to look at has to be on the desk at once. If the pile gets too big, something must come off before anything new goes on.

What goes on that desk:

Part of the context	Example
System prompt	Hidden instructions from the app about how the model should behave
Your messages	Questions, instructions, requests
Earlier replies	The model’s previous answers in the conversation
Attached content	Files, documents, web pages, search results
Tool results	Output from commands, tests, or API calls (for agents)
The new reply	The tokens the model is generating right now

Context window sizes vary widely between models, from a few thousand tokens to very large windows that can hold entire books or codebases. Larger is not automatically better, as you will see below.

Why the model “forgets”

An LLM does not remember anything between requests by itself. Each time you send a message, the app sends the relevant conversation back to the model inside the context window. The model’s sense of memory is just that text.

When a conversation grows past the window, the app has to do something:

1. Cut older messages, so the model no longer sees them.
2. Summarize earlier parts and keep the summary instead.
3. Refuse or warn that the input is too long.

This is why a long chat can suddenly “forget” a detail from the start. It was not forgotten in a human sense. It simply was no longer on the desk.

Bigger windows have trade-offs

Large context windows are useful, but they come with costs:

• More tokens, more cost. Most paid AI services measure usage in tokens, so a huge context on every request adds up.
• Slower responses. Processing more text takes more time.
• Attention gets spread thin. Models can miss or underweight details buried in the middle of a very long input. Relevant information placed clearly and close to the question tends to work better.
• More room for noise. Irrelevant text can distract the model and lower the quality of the answer.

The practical lesson: give the model the right context, not all the context.

Context windows and AI agents

Context is even more important for AI agents. An agent does not just answer once. It loops: read files, run a command, read the output, decide the next step. Every file it opens and every test log it reads goes into the context window.

On a big task, that fills up fast. Good agent tools manage this by:

• Reading only the files that matter instead of the whole project
• Summarizing or compacting older steps as the session grows
• Splitting big goals into smaller tasks with fresh context each time
• Using separate agents for separate jobs, so no single context gets overloaded

That last idea is one reason teams of agents can handle bigger projects than one agent alone. In BotQuarters, for example, a team lead agent breaks a goal into tasks and gives each teammate its own focused piece of work, with its own context. For more on how agents run, see what is an AI agent.

Context also affects cost directly, since every step re-sends tokens. Our guide to AI agent costs covers how that adds up.

Practical tips for working within the window

Whether you are chatting or running agents, these habits help:

1. Start fresh for new topics. A new conversation clears out old, irrelevant context.
2. Paste only what matters. Share the relevant section of a document, not the whole thing.
3. Put key instructions where they are easy to find. Clear, direct instructions near your actual question work well.
4. Summarize long threads. Ask the model to summarize the decisions so far, then continue in a new chat with that summary.
5. Use files and tools instead of pasting. Agents and assistants that can search or read files can pull in just the parts they need.
6. Restate important constraints. In a long session, repeat must-follow rules rather than assuming they are still in view.

If you want to see where tokens fit into the bigger picture of how models generate text, read how LLMs work.

FAQ

What is a context window in AI?

A context window is the maximum amount of text, measured in tokens, that a language model can consider in one request. It includes your instructions, the conversation history, any attached documents, and the model’s own response.

How many words is a token?

It depends on the model and the text. For typical English writing, a token is usually a bit less than a word on average, so a given number of tokens covers somewhat fewer words. Code, numbers, and many other languages usually take more tokens per word.

What happens when you exceed the context window?

The model cannot see anything outside the window. Depending on the app, older messages are dropped or summarized, or you get an error asking you to shorten the input. Either way, details that fall outside the window no longer affect the answer.

Is a bigger context window always better?

Not always. A bigger window lets you include more material, but it can cost more, respond more slowly, and make it harder for the model to focus on what matters. Well-chosen, relevant context often beats simply stuffing in everything.