Context window
The context window is everything a language model can have in front of it at once to produce an answer: the system instruction, the conversation history, the passages retrieved from documentation and the data looked up. It is measured in tokens and it has a ceiling.
What is context window?
The context window is everything a language model can have in front of it at once to produce an answer: the system instruction, the conversation history, the passages retrieved from documentation and the data looked up. It is measured in tokens and it has a ceiling.
Also: context · context length
More window is not more memory
A model with an enormous window does not remember previous conversations: every request is sent to it in full again. The window is how much fits in that request, not what the model knows about you. Memory across conversations has to be built separately, by storing and re-sending whatever matters.
Why it matters
What changes in a SaaS
Because it is where you decide which information reaches the model and which is left out, and that weighs more on answer quality than which model you use. When the window fills you have to choose, and that choice — which passages, how much history, which account data — is the real work of building an agent.
Context window in detail
What competes for the space
Four things: the instructions, the conversation history, the documentation passages and the account state. In a long conversation the history eats the space meant for passages, and the agent starts answering worse without anybody having touched anything.
The middle gets lost
Models attend better to the beginning and the end of what they are sent than to what sits in the middle. Putting the important part at the end of the context is one of the cheapest improvements available.
It costs money and time
It is billed per token, so filling the window with everything you can makes each answer more expensive and slower. Retrieving five good passages beats twenty mediocre ones, on the invoice too.
Questions about context window
Does a bigger context window give better answers?
Not on its own. It gives room for more information, but if what goes in is noise, the answer gets worse and costs more. What improves answers is choosing well what goes in.
Does the model remember what we talked about yesterday?
No, unless the system sends it again. Every request is independent; the feeling of memory is built by the application storing the history and re-sending it.
Related terms
A term on its own is only half understood. These come up in the same conversation.
LLM — Language model
A language model is a system trained on enormous amounts of text that, given a fragment, predicts how it continues. Writing, summarising, translating and holding a conversation all come out of that one simple capability: they are all ways of continuing a text plausibly.
RAG — Retrieval-augmented generation
Retrieval-augmented generation, or RAG, is the technique of finding relevant information in your own sources and handing it to a language model so it writes the answer with that. The model does not memorise the content: it looks it up at answer time.
System prompt
The system prompt is the fixed set of instructions a model receives before every conversation: who it is, what it can and cannot do, what tone it answers in and what to do when it does not know something. The user never sees it and it is sent on every request.
Latency
Latency is how long an AI system takes to return an answer from the moment it receives the question. It includes the information retrieval, the lookups against other systems and the text generation, and it is perceived whole even though the model is only one part of it.
Hallucination
A hallucination is an answer from a language model that sounds right and is not: an invented figure, a feature that does not exist, a plausible number nobody checked. It is not a bug, it is a direct consequence of how a model works — it produces the most probable continuation, not the verified one.
Tool calling
Tool calling is the mechanism by which a language model can run functions you define instead of only writing. You describe the available tools, the model decides which one to use and with what arguments, and it receives the result to fold into its answer.
