AI service complaints to tests: sort first, then add examples in pairs
Sort AI service complaints first, then add the ones you test as pairs: a case that should happen and one that must not.
Key points
- Sort complaints first by concrete cases
- Add examples in pairs: should, must not
- Fix service description for vague ones
- Rerun all pairs on every AI prompt edit
Contents
Sort complaints first by whether you can write a concrete case
A&A perspective
Once AI is part of the service you sell, sort every customer complaint before you change anything: will it become a test case or not? A test case here is a concrete input and the result it must produce, rerun every time you change the AI's instructions or settings.
A&A perspective
The sort has one rule: can you write the complaint down as a concrete input and result? "This receipt should have been booked as entertainment" can be written. "The accuracy feels low overall" cannot.
From the sources
In January 2026, Anthropic published a guide to building evaluations for AI agents. If you are already in production, it advises looking at your bug tracker and support queue, and it says that converting user-reported failures into test cases makes the test set reflect actual usage.
From the sources
The same guide sets a bar for a good task: two domain experts, working independently, would reach the same pass or fail verdict. Ambiguity in how a task is written, it says, becomes noise in the scores.
A&A perspective
Automate & Augment (A&A), the company writing this article, combines those two points into one sorting rule. Complaints are good material for tests. But add complaints that people would grade differently, and the test set itself starts to wobble.
A&A perspective
So before adding anything, sort by whether you can write the case. Deciding the type of a problem before investigating it already happens in support work.
From the sources
Gumloop, a company that offers tools for automating work, described in February 2026 how it runs its own support with a two-person team.
From the sources
Its AI first decides whether an incoming issue is a workflow failure, a how-to question, or an agent issue, and only then picks the tools to investigate with, the post says.
A&A perspective
A&A thinks the same order, type first and investigation second, works when turning complaints into tests. Gumloop's post, however, is about running support, not about writing test cases, and it is a company describing its own product, not an independent review.
One-sided tests skew, so add examples in pairs
A&A perspective
When a complaint is going to become a test, add it as a pair: one example where the behavior should happen and one where it must not. A single complaint usually points one way, either "it didn't do it" or "it overdid it". Add only that side and the tests grow thick on one side.
From the sources
Anthropic's guide says to test both the cases where a behavior should occur and the cases where it should not. One-sided evaluations, it says, create one-sided optimization.
From the sources
Its example is search: test only whether the agent searches when it should, and you might end up with an agent that searches for almost everything.
From the sources
Anthropic also writes that it ran into this with web search in Claude, its own AI assistant. Balancing searching too little against searching too much was difficult, and it took many rounds of changes to both the prompts and the evaluation.
A&A perspective
A&A thinks adding complaints one at a time walks into the same imbalance. Fix the AI for an "it didn't do it" complaint and it leans toward doing it. The "it overdid it" complaint a few weeks later can look like a new issue, when the earlier fix may have caused it.
Hypothetical example
Say a bookkeeping firm has AI assign account categories to receipts. This is a made-up example, not a real client's record. One client complains that a dinner with a business partner was booked as a meeting expense, so the firm adds a line to the AI's instructions: "Meals that entertain business contacts are entertainment expenses."
Hypothetical example
Three weeks later, another client writes in: "Even tea bought for an internal meeting is booked as entertainment."
Hypothetical example
Adding a pair means writing two cases when the first complaint arrives. A 12,000-yen dinner with two people from a client company is an entertainment expense. 800 yen of drinks for a meeting of two staff members is not.
Hypothetical example
Side by side, the two show the condition this firm actually uses: whether someone from outside the company was there. The line added to the instructions changes too: "Meals with outside guests are entertainment; meals among staff only are meeting expenses."
A&A perspective
The must-not side rarely shows up in your records. An overdone result raises no error; it is logged as a normal, finished job. Someone who knows the work writes that side when the pair is written.
| Shape of the complaint | Can you write the pair? | Where it goes |
|---|---|---|
| This input should give this result, but it doesn't | Yes. The complaint is the should-happen case; write the must-not case yourself | Add the pair to the tests |
| This input must not give this result, but it does | Yes. The complaint is the must-not case; pick the should-happen case from jobs that went right | Add the pair to the tests |
| The accuracy feels low overall | No. Nobody has decided which input must not give what | No test. Ask for one case first; if none comes, rewrite what the service covers in its description |
| I want to use it for this too | No. It is outside what you promised | No test. State it is out of scope, or keep it as a request |
Fix service description for vague ones, or exclude the use
A&A perspective
A complaint you cannot write as a pair of cases does not become a test. It goes one of two ways: you fix how the service is described, or you state that the use is out of scope.
A&A perspective
A vague complaint is one where you cannot write a single concrete input with the result that would be right, or the result that must not appear. With "it's weird sometimes" or "accuracy feels low", neither the client nor you has decided yet what must not come out.
From the sources
Anthropic's guide also lists weak points of user feedback such as bug reports. Users, it says, rarely explain why something failed.
A&A perspective
A&A thinks that is one reason complaints arrive as "it's weird sometimes". When one comes in, first ask the client for one case: which document, and what result came out. Once a case appears, the complaint moves to the side you can write.
A&A perspective
If the promise was worded loosely, fix the description. For example, add whether you read handwritten receipts, or whether you handle invoices in foreign currencies. If it is a use you never took on, state that it is out of scope and keep it separately as a request.
A&A perspective
A&A thinks that mixing in items people grade differently means every failure sends you back to ask whether the AI is wrong or the item is vague. Keeping vague complaints out lets you read a failing test the moment it fails.
A&A perspective
Deciding right after a request comes in whether it needs a fix, a clearer explanation, or a new feature is covered in another article, "Sort AI service support requests into three: build next or fix the docs". This article starts after you have decided to fix something: what to keep as a test.
Rerun all pairs on every AI prompt edit
A&A perspective
Run every pair, all together, each time you change the AI's instructions or the model it uses. If both sides pass, you know the earlier complaints have not come back.
From the sources
Anthropic's guide splits evaluations into two kinds. The first measures what an AI can do well and is meant to start from a low pass rate. The second checks whether the AI still handles the tasks it used to, and should pass nearly 100% of the time.
From the sources
The second kind protects against backsliding: a drop in score signals that something is broken. The guide adds that while teams keep improving what the AI can do, it is important to also run these checks so changes don't cause problems elsewhere.
A&A perspective
A&A thinks pairs built from complaints belong in that second kind. Both sides should pass when you add them. If they don't, either the fix is not finished or the pair is written wrong. Sort that out before adding them.
A&A perspective
If only one side fails later, the fix has leaned toward the other side. Which side failed also tells you which way to adjust the instructions next.
A&A perspective
Neither source measures how many opposite complaints one-sided tests actually produce. A&A has not measured complaint counts or recurrence rates either.
A&A perspective
The same approach works when a contractor edits the instructions. Ask whoever made the change to report whether every pair passed, and the owner does not have to reread each fix. Write the pass-or-fail cases in house; hand the machinery that runs them to an outside contractor.
Summary
A&A perspective
Sort incoming complaints first by whether you can write them as concrete cases. Add the ones you test as a pair: a case where the behavior should happen and one where it must not. Don't test complaints you cannot write; fix the service description or state that they are out of scope. Rerun all pairs every time you change the AI's instructions.
Turning complaints into tests comes down to two choices: which complaints to keep, and in what unit to add them. Sort by whether a concrete case can be written, add the writable ones as should and must-not pairs, and rerun them all on every change to check that both sides pass.
FAQ
Do I still write pairs if only a few complaints arrive each month?
Yes. If you can write only one side, keep it out of the tests and note the missing side in one line. When the same complaint comes again, you can start the pair from that note.
Can an outside contractor write the test cases?
The cases that decide pass or fail are best written in house, by someone who knows the work. Anthropic's guide describes a good test as one where two domain experts would reach the same pass or fail verdict. Setting up the machinery that runs the tests can go to a contractor.
Do calculations or data transfers with one right answer need pairs?
Often not, A&A thinks. There is no overdoing side, so fixing and adding one case at a time rarely tips the balance. Pairs matter where the AI decides whether to act at all.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-10-24 - Supporting the world’s most AI-native companies with a 2-person team
Gumloop · 2026-02-17
Accessed 2026-10-24