Latency and first response time
Latency measures how long the system takes to answer a message, in seconds.
First response time measures how long the team takes to answer a new conversation, and is counted in minutes or hours. An AI agent takes the second close to zero, and latency then becomes what the customer notices.
Why it matters
Because in support it competes with the expectation of a chat, which is measured in seconds.
An agent taking fifteen seconds to reply loses the person asking even when it is right, and in practice it feels worse than a human taking two minutes who says they are looking into it.
Where the time goes
Rarely in one place.
Retrieving passages, checking the account, generating the answer and, if the model calls a tool, another full round. Measuring the stretches separately comes first, because optimizing the wrong one changes nothing.
Showing the text as it is generated
Writing the answer out as it comes cuts perceived waiting a lot without changing the total time.
It is one of the best effort-to-effect changes available.
The trade-off with quality
More context and more steps usually give better answers and always give more latency.
Where the middle ground sits depends on the channel. In chat speed weighs heavily, in email it does not.
How to reduce it
There are five levers, and none requires changing vendor.
- Showing the text as it is generated, so the wait feels shorter.
- Passing the model only the context it needs.
- Running at the same time the lookups that do not depend on each other.
- Using a smaller model to classify and keeping the large one for writing.
- Saying what is being done when a lookup takes a while.
