Context Windows Explained: Why AI Forgets Long Conversations

Have you ever told a chatbot something at the start of a long conversation, only to find it ignoring that detail 50 messages later? That is usually the context window at work. Understanding it explains many puzzling behaviours and helps you get better results.

What is a context window?

A language model does not have memory the way a person does. For each reply, it reads a block of text, the context, and predicts what comes next. The context window is the maximum length of that block, measured in tokens. It includes everything the model must consider:

• System instructions
• The conversation so far (your messages and its replies)
• Any documents, search results or tool outputs you provide
• The reply it is currently writing

Once the total exceeds the window, something must be dropped or shortened. Anything outside the window is simply not visible to the model.

Tokens, not words

Text is split into tokens before processing. In English, a token averages roughly three-quarters of a word, so 1,000 tokens are about 750 words. Other languages and scripts, including Nepali in Devanagari, may require more tokens per word, so the same content fills the window faster.

A rough estimator:

def estimate_tokens(text: str, chars_per_token: float = 4.0) -> int:
    """Very rough English estimate. Use the provider's tokenizer for accuracy."""
    return int(len(text) / chars_per_token)

essay = "word " * 3000
print(estimate_tokens(essay))      # about 3,750 tokens

For exact counts, use the tokeniser provided by the model's library or API.

How big are context windows?

They vary widely. Early models handled a few thousand tokens. Many current models support tens of thousands, and some support hundreds of thousands or more. Larger windows allow long documents or big codebases, but they cost more and run slower.

Why the AI "forgets"

1. The oldest messages fall out. Chat apps often trim early parts of the conversation when it gets too long, or summarise them, losing details.
2. Attention gets diluted. Even inside the window, models can pay less attention to information buried in the middle of a very long context. This effect has been observed in research as "lost in the middle".
3. Conflicting instructions. If you changed your mind, earlier instructions can compete with later ones.
4. No true memory. When you start a new chat, the model begins fresh unless the application stores notes for you.

Practical tips for users

• Put key instructions near the start and restate them near the end of long prompts.
• Start a new chat for a new topic, pasting a short summary if needed.
• Summarise periodically: "Summarise our decisions so far in 10 bullet points", then continue from that summary.
• Keep prompts focused. Remove irrelevant text.
• Label sections of pasted material (for example, "Document A", "Document B") and refer to them by name.
• Ask for quotes. When working with a long document, ask the model to quote the relevant sentence to confirm it found the right place.
• Check important details against the source rather than trusting recall from a long chat.

For developers: managing context

If you build apps with language models, you control what goes into the window.

Trimming history

Keep the system prompt, the latest messages and a summary of older ones.

def build_messages(history, summary, max_recent=6):
    messages = []
    if summary:
        messages.append({"role": "user", "content": f"Summary of earlier conversation: {summary}"})
        messages.append({"role": "assistant", "content": "Understood."})
    messages.extend(history[-max_recent:])        # keep only the latest turns
    return messages

Periodically ask the model to update the summary from the older turns you are about to drop.

Retrieval instead of stuffing

Instead of pasting an entire knowledge base, retrieve only the relevant passages (RAG). This keeps prompts short and focused.

Chunking long documents

For a document longer than the window, split it into sections, process each one (for example summarise it) and combine the results, a pattern called map-reduce.

Caching

Some providers offer prompt caching, which reduces the cost and delay of repeated long prefixes such as system prompts and reference documents.

Count tokens before sending

Check the length to avoid errors and surprise bills, and reserve space for the reply.

Cost and speed

Many services charge per token for both input and output. A long context sent on every turn of a conversation adds up. Longer contexts also mean slower responses. Efficient prompts are cheaper and faster.

Long context versus RAG

	Long context	RAG
Setup	Simple: paste the documents	Needs indexing and retrieval
Best for	A few documents, deep cross-references	Large or changing collections
Cost per query	Higher	Lower
Accuracy risk	Details may get lost in the middle	Retrieval may miss the right chunk

Many systems use both: retrieve a few relevant long passages and place them in a large window.

Frequently Asked Questions

Does the model remember past chats?

Not by itself. Some apps add a memory feature that stores selected facts and inserts them into future prompts.

Is a bigger context window always better?

Not necessarily. It costs more, can be slower and may dilute attention. Use only what you need.

What happens if I exceed the limit?

The application may return an error, cut the oldest text or refuse to continue. Behaviour depends on the product.

Why does Nepali text use more tokens?

Tokenisers are often optimised for English, so other scripts split into more pieces. The effect varies by model.

Conclusion

The context window is the model's working memory, measured in tokens. Keep important information inside it, restate key points, summarise long chats and use retrieval for large knowledge sources, and the AI will seem to remember much better.