Skip to content

Latency

Adrià Castany 2 min

What is latency?

Latency is how long an AI system takes to return an answer from the moment it receives the question. It includes the information retrieval, the lookups against other systems and the text generation, and it is perceived whole even though the model is only one part of it.

Also called

model response time

Topic
AI in support
On this page
  1. Latency and first response time
  2. Why it matters
  3. Where the time goes
  4. Showing the text as it is generated
  5. The trade-off with quality
  6. How to reduce it
  7. How Intake handles it
  8. Frequently asked questions

Latency and first response time

Latency measures how long the system takes to answer a message, in seconds.

First response time measures how long the team takes to answer a new conversation, and is counted in minutes or hours. An AI agent takes the second close to zero, and latency then becomes what the customer notices.

Why it matters

Because in support it competes with the expectation of a chat, which is measured in seconds.

An agent taking fifteen seconds to reply loses the person asking even when it is right, and in practice it feels worse than a human taking two minutes who says they are looking into it.

Where the time goes

Rarely in one place.

Retrieving passages, checking the account, generating the answer and, if the model calls a tool, another full round. Measuring the stretches separately comes first, because optimizing the wrong one changes nothing.

Showing the text as it is generated

Writing the answer out as it comes cuts perceived waiting a lot without changing the total time.

It is one of the best effort-to-effect changes available.

The trade-off with quality

More context and more steps usually give better answers and always give more latency.

Where the middle ground sits depends on the channel. In chat speed weighs heavily, in email it does not.

How to reduce it

There are five levers, and none requires changing vendor.

  • Showing the text as it is generated, so the wait feels shorter.
  • Passing the model only the context it needs.
  • Running at the same time the lookups that do not depend on each other.
  • Using a smaller model to classify and keeping the large one for writing.
  • Saying what is being done when a lookup takes a while.

How Intake handles it

Frequently asked questions

How much latency is acceptable in a support chat?

As a guide, a few seconds to the first words feel fluid. When the wait drags on with no sign of progress, people leave or write again, and both make the outcome worse.

Does a smaller model answer faster?

It usually does, and for classifying or routing it is a good option. For writing the answer to the customer, the quality difference nearly always justifies the extra second.

Which matters more, latency or the quality of the answer?

Quality, as long as the wait is explained. A customer accepts a few more seconds when they can see their account is being checked and the answer resolves. They do not accept a fast answer that makes them ask again.

Glossary

Keep reading

Glossary

Related terms

See the full glossary

Add it without dropping what you use

15 minutes, no commitment