Latency
Latency is how long an AI system takes to return an answer from the moment it receives the question. It includes the information retrieval, the lookups against other systems and the text generation, and it is perceived whole even though the model is only one part of it.
What is latency?
Latency is how long an AI system takes to return an answer from the moment it receives the question. It includes the information retrieval, the lookups against other systems and the text generation, and it is perceived whole even though the model is only one part of it.
Also: model response time
Why it matters
What changes in a SaaS
Because in support it competes with the expectation of a chat, which is measured in seconds. An agent taking fifteen seconds to reply loses the person asking even when it is right, and in practice it feels worse than a human taking two minutes who says they are looking into it.
Latency in detail
Where the time goes
Rarely in one place: retrieving passages, checking the account, generating the answer and, if the model calls a tool, another full round. Measuring the stretches separately comes first, because optimising the wrong one changes nothing.
Showing the text as it is generated
Writing the answer out as it comes cuts perceived waiting a lot without touching the real figure. It is one of the best effort-to-effect changes available.
The trade-off with quality
More context and more steps usually give better answers and always give more latency. Where the middle ground sits depends on the channel: in chat speed weighs heavily, in email it does not.
Questions about latency
How much latency is acceptable in a support chat?
Under three or four seconds to the first words, the conversation feels fluid. Above ten, people leave or write again, and both make the outcome worse.
Does a smaller model answer faster?
It usually does, and for classifying or routing it is a good option. For writing the answer to the customer, the quality difference nearly always justifies the extra second.
Related terms
A term on its own is only half understood. These come up in the same conversation.
Context window
The context window is everything a language model can have in front of it at once to produce an answer: the system instruction, the conversation history, the passages retrieved from documentation and the data looked up. It is measured in tokens and it has a ceiling.
LLM — Language model
A language model is a system trained on enormous amounts of text that, given a fragment, predicts how it continues. Writing, summarising, translating and holding a conversation all come out of that one simple capability: they are all ways of continuing a text plausibly.
AI agent
An AI agent is a system that, beyond generating text, reads data and takes actions to finish a task. In support that means it checks the actual state of the customer account before answering, and when the answer involves doing something, it does it instead of explaining how.
FRT — First response time
First response time is how long passes between a customer sending their question and receiving the first reply from a person. It measures the initial wait, not the resolution, and it is the metric that weighs most heavily on whether support feels good or not.
Guardrails
Guardrails are the restrictions that stop an AI system doing or saying certain things: which sources it may take data from, which actions it may run, which topics it does not touch and what requires confirmation from a person. They are applied outside the model rather than by trusting it to obey.
Evals — Evaluation
An evaluation is a set of test cases used to check, repeatably, whether an AI system answers as it should. Each case carries an input and a success criterion, and the whole set is re-run every time something changes: the prompt, the model or the documentation.
