AI acceptance tests: separate tuning from final checks
Split tuning and final test examples when feasible; with too few examples, record the held-out check as unperformed.
Key points
- Split if enough examples
- Share criteria; retain final examples
- A pass covers only the tested scope
Overview diagram in preparation
Contents
Separate inquiry examples by purpose
A&A perspective
A final test set is a collection of examples kept out of AI tuning and reserved for checking before launch. When a contractor reports that every test passed, distinguish results on examples used for tuning from results on examples that were not. If there are too few examples to cover routine requests and important exceptions in both sets, checking them without a split is an option.
A&A perspective
Repeatedly editing a prompt against the same inquiry lets you check the response to that inquiry. That alone does not establish how it will handle a different one. A pass alone does not prove overfitting either.
From the sources
In its engineering account of improving tools for AI agents, Anthropic reports evaluating on held-out problems that were not used for tuning. The purpose was to avoid overfitting to the evaluations used during optimization.
A&A perspective
The account also reports improvements beyond tools designed by people in that evaluation. However, it is not an experiment showing that splitting the problems into two sets itself improved performance.
A&A perspective
Automate & Augment (A&A), the organization writing this article, proposes applying this distinction when accepting AI built for your own business. The transferable practice is separating how test examples are used. The engineering team’s results are not a forecast of benefits for a small company.
Hypothetical example
Suppose a creative production company commissions AI that drafts estimates from inquiries. It supplies examples containing prices and requested deadlines for tuning, while retaining other inquiries for final checks. This is a hypothetical scenario.
Hypothetical example
The final set also includes an inquiry with no deadline. The expected result is to flag the missing deadline for the responsible person, rather than invent one. Both sets include routine requests and exceptions, so the final check does not consist only of easy examples.
From the sources
Anthropic also recommends realistic evaluation tasks with outcomes that can be verified.
A&A perspective
The proposed test sheet records the inquiry, expected outcome, tuning or final-check purpose, custodian, conditions for opening the final set, and execution results. If examples merely rephrase similar inquiries, record that similarity too. Separate sheets do not establish that different situations were tested.
Share pass criteria and retain final examples
A&A perspective
One option is to share pass criteria with the contractor while retaining the specific inquiries for final checks in your company. Concealing what counts as success would assess the contractor against requirements that were never agreed.
Hypothetical example
The hypothetical production company would state in advance: use the price list for prices, and flag an unknown deadline for clarification. It also explains that routine and incomplete inquiries will be checked. What it retains is the specific wording of the examples used for those checks.
A&A perspective
Choose a custodian who can track whether examples have been used for tuning. If final-check examples are included in materials given to the contractor or folders the AI reads, a different filename does not make them unused.
A&A perspective
When building the system yourself, record not only whether you saw an example but whether you edited the prompt in response to it. Examples whose results informed changes subsequently count as tuning examples. If no independent check has been performed, record that status.
A&A perspective
For example, start final checks after choosing the version and pass criteria and reaching a stopping point in tuning. Relaxing the criteria after seeing results obscures what you originally intended to establish.
From the sources
Anthropic explains that results can vary between runs of the same task, making multiple trials a consideration.
A&A perspective
Reserving examples until the final check does not mean running them only once. Decide the number of repeated trials and how to handle failures in advance. If inputs or settings change between runs, do not mix those results with trials conducted under unchanged conditions.
A&A perspective
The proposed custodian and conditions for opening the final set are A&A’s operating design. The sources do not prescribe buyer custody procedures or a split ratio. Their developers and an owner accepting a commissioned system make decisions from different positions.
| Treatment | Tuning set | Final-check set |
|---|---|---|
| Purpose | Use results to make changes | Keep examples out of tuning |
| Sharing with the contractor | Share concrete cases and pass criteria | Share criteria; retain concrete cases in your company |
| After a failure informs a fix | Check that the known defect is fixed | Treat that case as a tuning example thereafter |
Read the results before choosing rollout scope
A&A perspective
Even after the final check passes, the evidence is limited to the inquiries and conditions tested. Compare the pass/fail list with what the system actually did, record what remains unchecked, and then decide which work to allow in production.
From the sources
Anthropic says it does not accept scores at face value until someone examines evaluation details and execution transcripts. It also warns that evaluations unlike actual use can create false confidence.
Hypothetical example
In the hypothetical production company, the reviewer reads the estimate draft, the price list it used, and the clarification items. Correctly writing “deadline unknown” does not satisfy the expected outcome if another field promises a delivery date.
A&A perspective
Once a failed final-check example informs a fix, a later pass on that example shows a known defect was fixed. Keep it distinct from a fresh result on an unused example. Preserve the initial failure alongside the revised version and the change made.
From the sources
Reusing a fixed example still has value. Anthropic explains that evaluations the system has learned to pass can check that previously working capabilities have not regressed.
A&A perspective
With only a handful of examples, choosing not to split is reasonable. If either set would omit routine requests or important exceptions, first improve the system using the available cases. A&A proposes recording the final held-out check as unperformed and limiting use to a trial in which a person reviews every result.
A&A perspective
Collecting examples, defining expected outcomes, managing cases, and reviewing execution records all require work. Consider whether your company can sustain that review, alongside contractor fees. Reductions in defects or costs from this proposed split have not been measured.
A&A perspective
Keep a human-handled scope for inquiries not covered by the trial. For defining output criteria and repeated-trial conditions first, see “AI deliverable acceptance criteria: 4 terms for output that varies.”
Keep tuning and final-check results separate
A&A perspective
Treat inquiry examples used for tuning separately from those reserved for final checks.
A&A perspective
Share pass criteria with the contractor and decide who retains the specific examples.
A&A perspective
Read final-check results and distinguish the tested scope from what remains unchecked.
Keep tuning and final-check inquiry examples separate. Share pass criteria with the contractor and decide who retains the concrete cases. Read final-check results and distinguish tested work from what remains unchecked.
FAQ
What proportion should be reserved for final checks?
There is no universal ratio here, and the sources prescribe none. Check whether both sets can cover routine requests and important exceptions. If there are too few examples, testing without a split is an option.
Can I give a failed final-check example to the contractor?
Yes, to support a fix. Subsequently treat it as a known example; a later pass is not an unused-case result. If no other unused examples have been tested, leave that check recorded as unperformed.
Can I use this approach when building AI by myself?
Yes, but changing the system after examining reserved examples removes their independence from tuning. Record which cases informed tuning and keep their results separate from unused cases. If managing the split is too burdensome, another option is a trial where a person reviews every output.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Writing effective tools for agents — with agents
Anthropic · 2025-09-11
Accessed 2026-10-30 - Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-10-30