What to test in a paid AI PoC: 4 things to decide before you outsource
Order a paid AI PoC only once your own staff can judge pass or fail on inputs and failures your company picked.
Key points
- You, not the vendor, pick past cases
- Include failures. Correct handoffs pass
- Decide who checks and how long it takes
- Go/stop rules on one page for approval
Contents
You pick past cases as the test inputs
A&A perspective
In a paid PoC, your company, not the vendor, picks the cases to test from its own past records. A paid PoC (proof of concept) is the stage where you pay to test whether AI can handle your work before you build it for real.
A&A perspective
The demo a vendor shows in a sales meeting runs on inputs the vendor chose. Picking examples that work well is only natural. But if the paid test runs on the same inputs, the question "what happens with our own cases?" stays open to the very end.
From the sources
Anthropic, the company that makes the AI model Claude, published a guide to evaluating AI agents on January 9, 2026. It says that 20 to 50 simple tasks drawn from real failures is a great place to start.
From the sources
For a system that is already running, the guide recommends looking at the bug tracker and the support queue. Turning failures that users reported into test cases, it explains, makes the tests reflect how the system is actually used.
A&A perspective
Automate & Augment (A&A), the company writing this article, reads this as saying that where the cases come from matters more than how many there are. Bug reports and support tickets pile up on the side that used the system, not the side that built it. In a paid PoC, the side that uses it, and so the side that picks the inputs, is your company.
A&A perspective
Do not carry over the guide's 20 to 50 as a target. The guide is about evaluating a product, and a single company's test may turn up only 6 cases of the same kind. What belongs on the page for approval is not a big count but one line: "our company picked these 10 cases."
Hypothetical example
For example, say an industrial machinery maintenance company outsources a paid PoC in which AI drafts service reports. Field technicians send photos and short notes from their phones, and the AI turns them into a draft report for the customer.
Hypothetical example
The vendor's demo ran on 3 cases with clear photos. The buyer asks the service manager to pick 10 reports from the past six months that had to be rewritten. This company and its numbers are a made-up example for illustration.
Include failures in the test as well
A&A perspective
Mix past failures into the 10 test cases, and count a case as a pass when the AI correctly hands it back to a person instead of writing it.
From the sources
Anthropic's guide says to test both the cases where a behavior should occur and the cases where it should not. Testing only one side, it explains, makes the system lean toward that side.
From the sources
Its example is search. If you only test whether the AI searches when it should, the guide says, you can end up with an AI that searches for almost everything.
A&A perspective
A&A reads this as deciding in advance, when you set the scope of the order, which cases the AI should leave alone. If you test without deciding, a case handed back to a person counts as a failure. If your manager or the board reads that as "if a person does it in the end, what's the point," behavior that works as designed becomes the reason to stop.
A&A perspective
Write it down first, and the same result counts as a pass. Pass or fail depends on what comes with the handoff. If it lists the items it could not read and the information that is missing, it passes. If it comes back blank with nothing written, it fails. Write what a handoff should carry into the scope of the order as a pass condition, too.
Hypothetical example
In the example, 3 of the 10 are cases where handing back is the right answer: 2 where the photo is too dark to read the model number, and 1 that contradicts the previous report. The handoff passes if it names the item, such as "model number not readable from the photo," and says in one sentence what to ask the technician.
Hypothetical example
The other 7 pass if the manager can send them to the customer without edits. The 7 and the 3 are placeholder numbers for illustration.
| Row | What to decide | Example (made up) |
|---|---|---|
| Task | The one job given to the AI | Drafting customer reports from technicians' photos and short notes |
| Who picks the inputs, and how many | Who in your company picks them, from which records | The service manager picks 10 reports from the past six months that had to be rewritten |
| Cases that should pass | The state that is usable as is | 7 cases the manager can send to the customer without edits |
| Cases that should be handed back | What a handoff should carry | 3 cases, each listing the unreadable item and what to ask |
| Checker, method, time | Role, current step, time | The service manager, the current sign-off step, 15 minutes a case, two and a half hours in all |
| Go rule | The line for moving to a full contract | At least 6 of the 7 pass without edits, and the 3 come back correctly |
| Stop rule | The line for stopping | The reasons for failing spread across 3 or more kinds |
| When the AI is updated | How to re-check after an update | Check the same 10 cases; the vendor bears the cost during the PoC |
Decide who checks pass or fail, and for how long
A&A perspective
The people who judge pass or fail are your own. So before ordering, decide in-house who will check and how much time it will take, and write that into the scope of the order together with how they will check.
From the sources
Lovable is a service that turns a plain-language description of an app into production-grade software.
From the sources
Each time a new Claude release comes out, Lovable measures quality again with the same evaluation it has run from the start. According to Anthropic's customer story, that evaluation measures how often the system produces an app that is broken or is not what the user asked for.
From the sources
The page gives the reason this check matters: many Lovable users cannot read the code themselves, so they are trusting the output to work.
A&A perspective
A&A thinks a buyer is in a similar position. A buyer who cannot read the AI's internals can only judge pass or fail with the checks it already has at hand. So do not invent a new check; use the steps of your current work as they are.
From the sources
Anthropic's guide defines the outcome as the state actually left behind when the trial ends. Its example: what counts is not whether the AI said "I booked it," but whether the booking record actually exists. It also says a good task is one where two experts looking separately would reach the same pass or fail verdict.
A&A perspective
So write the pass condition not as "a draft was produced" but as "the manager sent it to the customer without edits." Then show the same draft to two people in your company and check that they reach the same verdict. If they split, the sentence is not yet ready to use as a pass condition.
A&A perspective
Put the time in the scope too. If the checker has no time, the test stops no matter how well the AI does. When time is short, cut the number of cases. Judging 5 cases for certain leaves more you can attach to the approval request than 10 cases left unjudged.
Hypothetical example
In the example, the checker is the service manager. The method is the current step, unchanged: the manager reads each report and signs it before it goes to the customer. At 15 minutes a case, 10 cases take two and a half hours. Decide which days of the two-week period that time will come from. All of these numbers are placeholders.
Put go and stop rules on one page for approval
A&A perspective
Add go and stop rules to the 3 decisions so far, turn them into a one-page table, and attach it to the approval request for the order. It answers the questions your manager or the board will ask, "What are we testing?" and "Where do we stop if it fails?", on a single page.
A&A perspective
The page's rows are the task, who picks the inputs and how many, the cases that should pass, the cases that should be handed back, and the checker, method and time. Below them come the go rule, the line for moving to a full contract, and the stop rule, the line for stopping. The last row says how to re-check when the AI is updated.
Hypothetical example
Filled in with the example, the go rule is "at least 6 of the 7 pass without edits, and the 3 come back correctly." The stop rule is "the reasons for failing spread across 3 or more kinds." When the AI is updated, the same 10 cases are checked again, and the page says the vendor bears that cost during the PoC.
A&A perspective
The stop rule is the row that protects the buyer. If you have written "go if 6 pass," then when only 4 pass you can calmly choose between adding budget to keep going and taking it back to look for the cause. A test with no stop rule has no set end.
A&A perspective
If a result meets neither rule, do not extend the period. Decide again with your manager or the board, on the results at that point, whether to continue.
From the sources
Anthropic's guide says evals get harder to build the longer you wait. Wait too long, it explains, and you end up reverse-engineering success criteria from a live system.
A&A perspective
Even after the full contract, each time the AI version is upgraded or the instructions are edited, cases that used to pass can start to fail. The 10 cases and pass conditions from the test can be reused for that check as they are. That is why the last row of the page settles that updates are checked on the same 10 cases, and who bears the cost.
Summary
A&A perspective
You, not the vendor, pick the test inputs from your past cases.
A&A perspective
Mix in failures, and count a case the AI correctly hands back to a person as a pass.
A&A perspective
Decide in-house who checks pass or fail and how long it takes, and write that into the scope with the method.
A&A perspective
Add go and stop rules to these 3 to make the one page for the approval request.
What a buyer gets from a paid PoC is not a replay of the vendor's demo but a result your own people can judge on inputs your company chose. Decide the inputs, the failure cases, the checker and their time before ordering, put them on one page with the go and stop rules, and attach it to the approval request. If the budget and the checker are not settled yet, skip the paid PoC and start with a free short demo or a consultation.
FAQ
Isn't the vendor the one who writes the pass conditions?
The vendor can polish the wording. But only the buyer can supply the inputs and the way of checking. Past failures and the checker's time exist only inside your company. When you receive the vendor's draft, check that these 2 are in it, in your company's own words.
Should we make this page even before the budget is set?
No. At that stage a free short demo or a consultation moves faster. If all the approval needs is evidence that it is technically possible, making this table can stall the conversation instead. Order a paid PoC with this page attached once the budget and the checker are settled.
What if no one in-house can write this scope?
One option is to ask someone who stands on the buyer's side, separate from the vendor. A scope written only by the vendor tends to miss your failure cases and your way of checking. Even then, picking the inputs and judging pass or fail stay with your own people. Ask that person to help write those 2 into the scope.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Lovable Claude Platform (API) case study
Anthropic · Publication date not stated
Accessed 2026-10-23 - Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-10-23