In the first month with an AI agent, the dashboard looks great. First response time falls from four hours to eleven seconds. The volume reaching people halves. Deflection rate comes out at 40%.
Then somebody asks whether support is getting better, and those numbers can't answer the question.
The metrics aren't lying. Most of them were designed for a team of people with limited capacity, and once part of the conversations stop going through a person, they no longer measure what they used to.
The ones that break
First response time
It measured how long you waited until somebody read your message. With an agent that answers instantly, it drops to zero and stays there.
The number keeps showing up, and that is what makes it dangerous: it looks like a metric, but it has stopped being one. A dashboard where first response time reads eleven seconds every month tells you nothing, whether support is going well or badly.
What to measure instead: the time to the first useful response, meaning a response the customer doesn't have to rephrase. If the agent answers instantly and the customer asks the same question again in different words, that first response doesn't count.
Separately, track the one that matters most: the time to reach a person when a person is needed. That is where things can get worse without anyone noticing, because the cases that need somebody are a minority and they hide behind the average.
Deflection rate
It is the metric used to justify the investment, and the easiest one to inflate without meaning to.
The trouble is the denominator. If you count any conversation that never reached a person as deflected, you are also counting the customers who gave up. Hiding the contact button pushes deflection rate up and hurts the business.
How to measure it honestly: conversations closed without a person stepping in and without the customer coming back within seven days, divided by all conversations started. The second condition is what turns a vanity number into a useful one.
A quick check: if deflection rate rises and CSAT falls in the same month, conversations were hidden, not resolved. The same applies if it rises alongside the number of customers writing through other channels, such as the account executive's email, LinkedIn or the sales chat.
CSAT
It still works, but it measures something different and has to be read differently.
When a person answers, CSAT measures the whole interaction: tone, judgment and resolution. When an agent answers, it splits into two cases that shouldn't be averaged:
- The agent resolved it: the score usually rises, because the answer came sooner.
- The agent didn't resolve it: the score drops further than if nothing had been there, because the customer sees the failed attempt as an obstacle put in their way.
Averaging the two gives a number in the middle that describes neither. Splitting by who resolved it (agent, person, or agent then person) is the minimum for CSAT to keep telling you something.
Tickets per agent
It no longer measures workload. If the AI agent takes the repeated, documented work, what reaches people is harder by definition: cases that need judgment, angry customers, problems nobody had seen before.
A team handling half the tickets it did six months ago may well be working harder. Measuring volume per person without looking at what is left leads you to the opposite of the right conclusion.
The ones that start to matter
True resolution rate
Of all the conversations the agent answered, how many were closed for good: the customer didn't write again, didn't rephrase and didn't ask for a person.
It is the central metric of an AI system, and it almost never comes in dashboards by default, because it requires looking at the following days and not just the moment the conversation closed.
Handoff quality
When the agent passes a conversation to your team, does the case arrive with everything in place or blank?
One figure measures it: how many handed-over conversations force the customer to repeat something they already said. It is the metric that best separates a handoff that saves time from one that costs time, and it explains why two companies with the same deflection rate can have very different satisfaction.
Rephrasing rate
How often the customer asks the same question again in different words after an agent response. It is the earliest sign that an answer didn't land, and it shows up weeks before CSAT moves.
It is also very useful for your documentation: the list of rephrasings, grouped by topic, is the list of articles that are badly written or missing.
Cost per resolved conversation
The number that settles the argument, and the only one you can compare before and after.
It has two parts: what the agent costs (subscription plus usage) and what the human time still needed costs, using handling time that includes looking for information and recovering from interruptions.
Without this metric, the question of whether AI pays off gets decided on impressions.
Coverage by question type
Look at the breakdown instead of one global percentage: how many billing questions get resolved by the agent, how many configuration questions, how many integration questions.
That is what turns the metric into a decision. A global 30% doesn't tell you what to do. "80% of billing questions and 10% of integration questions get resolved" tells you exactly where next week's work is.
How to build the dashboard
Four numbers are enough:
| Metric | What it answers |
|---|---|
| True resolution rate | Is it working? |
| Cost per resolved conversation | Does it pay off? |
| CSAT split by who resolved | Does it help the customer? |
| Coverage by question type | Where is the next piece of work? |
One warning matters more than all four: none of this is comparable with last month if the mix of questions has changed. A launch, a price increase or a campaign that brings in a different kind of customer changes what arrives, and with it every average.
Before switching anything on
The only way to know whether something improved is to have a "before". And you have to measure it by hand, because it isn't in any dashboard.
Take a hundred conversations from your inbox and sort them into three groups: those answered with general information, those that depend on the account, and those that need someone's judgment. That split sets the ceiling of what can be automated, and it decides whether a target is reachable or a promise somebody will break in three months.
With that split in front of you, the discussion moves from which tool to buy to which part of the work comes off your plate. That is the version of the conversation that leads to a decision.
Frequently asked questions
What resolution rate is good for an AI agent?
It depends on the split. If half your conversations need judgment, the ceiling is the other half. An agent that closes 70% of the documentation and account questions, measured over seven days, is working well. A "90%" without the seven-day condition says nothing.
How do I know if my deflection rate is inflated?
If it rises while CSAT falls, if messages through other channels go up, or if the vendor counts any conversation without a customer reply as resolved. Measure it with the condition that the customer doesn't come back for seven days.
Is first response time useless now?
It is still useful for what reaches your team, measured up to the first reply from a person, with the clock starting at the handoff. As a global average, it isn't.
How do I calculate cost per resolved conversation?
(Monthly AI cost + people's time × hourly cost) divided by the conversations resolved that month. Compare it with what each ticket cost you before.
What should I ask the tool for so I can measure this?
That it records who closed each conversation (agent, person or both), whether the customer came back in the following days, which actions the agent ran and why it handed the conversation over. Without that record, none of the new metrics can be calculated. Intake's insights include it.
