Before AI handles inquiries: count wrong resolutions and handoffs
Decide whether AI should take your inquiries by counting wrong resolutions and handoff effort, not by resolution rate.
Key points
- AI resolutions may include silent exits
- Set hand-back rules from past inquiries
- Count wrong resolutions via follow-ups
- Gauge handoff effort by repeat questions
Contents
Resolution rates may include silent exits
From the sources
Intercom, which makes customer service software, counts conversations in which the customer quietly left among the resolutions of Fin, its AI agent product. By Intercom's definition, the resolution rate is the share of conversations that Fin closes without any human involvement.
From the sources
According to Intercom's help center, a resolution takes one of two forms. After Fin's last answer, either the customer confirms it helped, for example by replying "That helped," or the customer leaves the conversation without asking for further help. If the customer later comes back to the same conversation for more help, that resolution is deducted.
From the sources
Fin costs $0.99 per resolution. A conversation in which the customer asks to talk to a person counts neither as a resolution nor toward the bill. The help center also says that when a customer unhappy with Fin's answer simply leaves, the conversation is counted as a resolution and charged.
A&A perspective
Automate & Augment (A&A), the company writing this article, reads this way of counting as follows. Customers who leave without a word include people who were satisfied with the answer and people who got a wrong answer and gave up. The resolution rate alone cannot tell these two groups apart.
From the sources
According to Anthropic's customer story, Intercom itself does not look only at resolution rate when it tests models. It also examines answer accuracy, customer satisfaction and response quality.
A&A perspective
Whether you buy a product or build your own support channel with Claude or a similar model, the number changes with what gets counted as a resolution. When you look at the resolution rate of your own support channel, first check how it is counted.
Set hand-back rules first, from past inquiries
A&A perspective
Set your hand-back rules by reading your own past inquiries, before you choose an AI product. Hand-back rules are the types of inquiry that the AI does not answer and instead passes to a staff member.
From the sources
In "Building effective agents," its guidance on designing systems built around AI, Anthropic describes sorting inquiries by type and sending each type to a different process. Its examples are general questions, refund requests and technical support. The guidance recommends starting with the simplest setup and adding complexity only when that falls short.
Hypothetical example
From here, take a hypothetical example. An online shop is run by four employees and two contractors. One of the contractors answers inquiries during weekday business hours. The owner wants AI to take the inquiries that arrive at night and on weekends.
Hypothetical example
Suppose the shop reads the last two weeks of inquiries and marks the ones that the contractor who answers them could not decide alone and passed to the owner: whether an item can be returned, whether to discount shipping, whether to change the delivery address. These become the hand-back rules: for inquiries that match them, the AI only replies that the inquiry has been received. The AI answers the other types.
From the sources
Anthropic's customer story says that letting the AI agent process refunds and account changes on its own is Intercom's next goal.
A&A perspective
Setting the rules first narrows what you compare between products. A setup in which the AI also answers the types you do not hand to it does not suit this shop, even if its resolution rate comes out high. Compare two things: answer accuracy on the types you do hand over, and how a conversation is passed on when it goes back to a person.
A&A perspective
How to prepare the answers for the types you hand over is covered in "Try AI on customer inquiries: start by settling your top 20 answers."
Count wrong resolutions via follow-ups on other channels
A&A perspective
Count wrong resolutions from follow-up contacts that arrive outside the AI channel. A wrong resolution is a conversation in which the AI gave a wrong answer but which was counted as resolved because the customer left without a word.
From the sources
Intercom's help center says that a resolution is deducted when the customer comes back to the same conversation for more help.
A&A perspective
When a customer gets in touch again by email, phone or social media, A&A thinks the AI channel's numbers may still show the original conversation as resolved. The help center does not say how contacts that arrive through other channels are treated.
Hypothetical example
In the shop, suppose the AI answers a return question with "This item is not eligible," and the customer closes the chat on the spot. Three days later the same customer emails: "That is not what I was told." The AI channel's numbers now hold one resolution, and the inbox holds one complaint.
Hypothetical example
One way to count: for the inquiry types handed to the AI, collect contacts from the same customer that reach another channel within one week. One week is a placeholder; adjust it to how long customers take to notice that the answer was off. During the trial only, the contractor collects these once a week from the email and phone records.
From the sources
Anthropic's customer story also mentions Fundrise, which runs a real estate investment service and uses Fin. The company is said to have automated more than half of its support volume while keeping answer accuracy at 95%. The story does not say how that accuracy was measured.
A&A perspective
Here the share handled automatically and answer accuracy appear as separate numbers. Do the same in your own company: put the count of wrong resolutions next to the resolution rate.
Gauge handoff effort by counting repeat questions
A&A perspective
Gauge the effort of a handoff by how often the staff member has to ask the customer again. A handoff is when the AI, instead of answering, passes the conversation to a staff member.
From the sources
Anthropic's guidance describes agents that return to a person partway through a task for information or a judgment call. They stop at checkpoints set in advance, or when they cannot go further, and wait for the person's reply.
A&A perspective
In a support channel, the point where an inquiry meets a hand-back rule is that stopping point. What should reach the staff member then is three things: the customer's first message, the full text of the AI's answer, and the reason the AI decided to hand it back. Without them, the staff member asks the customer the same things again, and the customer ends up explaining twice.
Hypothetical example
In the shop, suppose the AI passes a return question to the contractor. If the order number was not passed along, the contractor first has to ask for it again. Without the full text of the AI's answer, the contractor also cannot tell whether the AI answered from an outdated return policy.
A&A perspective
At the end of the trial period, line up three numbers: the resolution rate, the count of wrong resolutions, and the number of repeat questions. Add the types with many wrong resolutions or repeat questions to the hand-back rules. While those two stand out, do not widen what you hand to the AI.
Summary
A&A perspective
AI resolution rates may include conversations the customer silently left. Before choosing an AI, read your past inquiries and set your hand-back rules. During the trial, count wrong resolutions from follow-ups that arrive on other channels. Alongside that, gauge handoff effort by how often staff had to ask the customer again.
AI resolution rates may include conversations the customer silently left. Set hand-back rules from your past inquiries before choosing a product, and during the trial put the count of wrong resolutions and of repeat questions next to the resolution rate to decide how much to hand over.
FAQ
Is it worth counting wrong resolutions with only a few dozen inquiries a month?
Yes. But while volume is low, reading each conversation handed to the AI, and each contact that came in on another channel, one by one tells you more, and faster, than lining up counts.
If you pay per resolution, is a higher resolution rate always better value?
Not necessarily. Intercom's Fin counts conversations the customer silently left as resolutions and charges $0.99 for each. Its help center says a conversation in which an unhappy customer simply leaves is charged too, so a conversation that ended in a wrong answer can be billed as long as the customer does not come back to the same conversation.
Will refunds and delivery address changes eventually be safe to hand to AI?
For now, keep them on the hand-back side. Anthropic's customer story says that letting the AI agent process refunds and account changes on its own is Intercom's next goal. Start by settling what gets passed to a person at handoff.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Fin AI Agent outcomes
Intercom · Updated 2026-09-23
Accessed 2026-10-23 - Intercom provides customer service tech that delivers up to 86% resolution rates with Claude
Anthropic · Publication date not stated on page
Accessed 2026-10-23 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-10-23