More window is not more memory
A model with an enormous window does not remember previous conversations.
Every request is sent to it in full again. The window is how much fits in that request, not what the model knows about you. Memory across conversations has to be built separately, by storing and re-sending whatever matters.
Why it matters
Because it is where you decide which information reaches the model and which is left out, and that weighs more on answer quality than which model you use.
When the window fills you have to choose, and that choice (which passages, how much history, which account data) is most of the work of building an agent.
What competes for the space
Four things, and what one takes up another cannot.
- The system instructions.
- The conversation history.
- The passages retrieved from the documentation.
- The data looked up in other systems, such as the customer’s account.
The middle gets lost
Models attend better to the beginning and the end of what they are sent than to what sits in the middle.
Putting the important part at the end of the context is one of the cheapest improvements available.
It costs money and time
It is billed per token, so filling the window with everything you can makes each answer more expensive and slower.
Retrieving five good passages beats twenty mediocre ones, on the invoice too.
