Skip to content

Context window

Adrià Castany 2 min

What is the context window?

The context window is everything a language model can have in front of it at once to produce an answer. The system instruction, the conversation history, the passages retrieved from documentation and the data looked up. It is measured in tokens and it has a ceiling.

Also called

context, context length

Topic
AI in support
On this page
  1. More window is not more memory
  2. Why it matters
  3. What competes for the space
  4. The middle gets lost
  5. It costs money and time
  6. How Intake handles it
  7. Frequently asked questions

More window is not more memory

A model with an enormous window does not remember previous conversations.

Every request is sent to it in full again. The window is how much fits in that request, not what the model knows about you. Memory across conversations has to be built separately, by storing and re-sending whatever matters.

Why it matters

Because it is where you decide which information reaches the model and which is left out, and that weighs more on answer quality than which model you use.

When the window fills you have to choose, and that choice (which passages, how much history, which account data) is most of the work of building an agent.

What competes for the space

Four things, and what one takes up another cannot.

  • The system instructions.
  • The conversation history.
  • The passages retrieved from the documentation.
  • The data looked up in other systems, such as the customer’s account.

The middle gets lost

Models attend better to the beginning and the end of what they are sent than to what sits in the middle.

Putting the important part at the end of the context is one of the cheapest improvements available.

It costs money and time

It is billed per token, so filling the window with everything you can makes each answer more expensive and slower.

Retrieving five good passages beats twenty mediocre ones, on the invoice too.

How Intake handles it

Frequently asked questions

Does a bigger context window give better answers?

Not on its own. It gives room for more information, but if what goes in is noise, the answer gets worse and costs more. What improves answers is choosing well what goes in.

Does the model remember what we talked about yesterday?

No, unless the system sends it again. Every request is independent; the feeling of memory is built by the application storing the history and re-sending it.

What is a token?

It is the unit a model counts text in, a piece of a word. A short word is usually one token and a long one, several. The context window and the usage price are measured in tokens.

What happens when a conversation does not fit in the window?

The system has to choose. The usual approach is to summarize the older messages or leave them out, which is why in very long conversations an agent can forget something said at the start.

Glossary

Keep reading

Glossary

Related terms

See the full glossary

Add it without dropping what you use

15 minutes, no commitment