When to retest an AI feature you judged "not ready yet"
Retest a feature you set aside when the failure behind that verdict is reported fixed, not on a schedule.
Key points
- Name the failure, not the model
- Log input, pass line, failure that day
- Retest on news, not on a schedule
- Append to the record, reset the trigger
Contents
Write the reason as the name of the failure, not the model
A&A perspective
A set-aside record is the single page you keep on the day you decide a given task cannot be handed to AI. It holds three things: the input you gave, the result you expected, and the failure that happened.
A&A perspective
Write the reason as the name of the failure you saw at the time, not as a verdict on the model. "Not ready yet" looks like a summary judgement about capability, but what sits behind it is a summary of the failures you saw across a handful of attempts.
A&A perspective
If only that summary survives, six months later you cannot say what would have to change for the verdict to change. So every retest starts again from inventing a way to check.
From the sources
Genspark is a company that sells an AI service turning a single instruction into slides, spreadsheets and documents. A customer story Anthropic publishes about it describes the same idea being tested for two years.
From the sources
The person running those tests was the co-founder who serves as chief technology officer. He had been thinking since 2023 about a way for the model to decide each next step for itself. The story says he tested it against every new frontier model that came out, roughly every three months, for two years.
From the sources
The failures were always the same. Falling into infinite loops. Hitting the same error repeatedly. Failing to recognise when enough information had been gathered to stop. The page says it was one of those three.
A&A perspective
Automate & Augment (A&A), the organisation writing this article, reads the account this way. The reason for setting the idea aside stayed fixed as three concrete failures. That is why the run in which those failures stopped appearing could be told apart from the rest.
From the sources
The same page describes trying again in early 2025 with Claude Sonnet 3.7, the newest Claude release at the time. It knew when to stop, and it knew which tools to call. When a tool returned an error, it would try another way. That recovery behaviour, the page says, was the piece that had been missing.
A&A perspective
A&A reads that passage as meaning the three failures behind the earlier decision did not appear on that run. Across those two years the yardstick used for the judgement did not change. Only the model side changed. Because the yardstick stayed the same, the difference is visible.
Hypothetical example
Suppose, as a hypothetical, that you have a job that pulls out only the clauses that changed when a client revises its internal rules, and lists them. This example is invented; it is not a record of a real client. The numbers that appear in it from here on are not measurements either.
Hypothetical example
If you tried it six months ago and concluded that "extraction accuracy is still not good enough", that one sentence is all that remains. How many items of how many failed, and on which field, is gone.
Keep three fields on one page on the day you set it aside
A&A perspective
What you keep on that day is three things: the input you gave, the result you expected, and the failure that happened. With those three, six months later you can push the same input through unchanged and see whether the same failure appears.
A&A perspective
A check that judges how well an AI performs by a fixed procedure is called an evaluation here.
From the sources
In January 2026 Anthropic published an engineering article on how to build evaluations for systems in which a model decides each next step for itself.
From the sources
That article states that evals get harder to build the longer you wait. It also says that if you wait too long you end up reverse-engineering success criteria from a live system.
From the sources
The same article recommends starting with what you already test manually during development. Hundreds of tasks are not needed. It says 20 to 50 simple tasks drawn from real failures make a good start, and adds that more mature systems may need larger and harder evals.
A&A perspective
The day you set the task aside is the day the pass criteria cost least to write. What you checked by hand is still in your head, and the failing input is still on your desk.
A&A perspective
In the input field, write what you actually handed over. Not a summarised description, but the kind of file, the number of items, and the instruction text you sent with them.
A&A perspective
In the expected-result field, write the line that separates pass from fail. Not "extracts it correctly", but something like "all 20 of the 20 revised clauses are listed, with zero extras", so that two people looking separately reach the same verdict.
A&A perspective
In the failure field, write what went wrong and how. Not "accuracy is low", but something like "in 3 of the 20 clauses, a clause that had not changed was listed as changed", so that you can later say the same thing happened again.
A&A perspective
When you are tracking several capabilities at once, record the failures separately for each one. The company above could use one yardstick for two years because it was following a single idea. Watch three tasks together and you cannot tell which failure disappeared.
Hypothetical example
For the same hypothetical job, the page would read like this. Input: "the internal rules before and after revision, 2 PDFs, plus the extraction instruction text". Expected result: "all 20 of the 20 revised clauses listed, with zero extras". Failure: "in 3 clauses, an unchanged clause was listed as changed; in 2, a changed figure inside a table was missed".
| Point of comparison | Retest on a schedule | Retest on a trigger |
|---|---|---|
| What starts it | A date, such as every three months | News that the failure behind the decision is fixed |
| When a busy month lands on it | It is skipped, and the skipped round does not return | It only moves when news arrives, so nothing is skipped |
| Work per round | Rebuild the way of checking from scratch | Push the recorded input through unchanged |
| When it fails | You cannot say what is missing | You can say which failure remains |
Let the retest be triggered by news that the failure is fixed
A&A perspective
Start the retest from a piece of news that arrives from outside, not from a date. Decide on every three months and the retest is skipped in a busy month. A skipped round does not come back.
From the sources
Over those two years the company above tested against every new frontier model as it came out. On its revenue after the early-2025 rebuild, the same page carries two statements: that annual recurring revenue passed 200 million dollars, and that it passed 250 million dollars. Which of the two is current cannot be read off the page.
A&A perspective
Both are figures the company itself put out, not figures a third party verified. They are quoted here for neither of the obvious reasons, and certainly not to recommend the same cadence.
A&A perspective
A setup that can test every time a new model appears belongs to a company of that size. A&A takes the view that a one-person or few-person company will struggle to hold the same frequency. So the design swaps frequency for movement when news arrives.
A&A perspective
What works as a trigger is an update note that says, by name, that the failure behind your decision has been fixed. An announcement that something "got smarter overall" is not a trigger. Use only wording you can hold up against the failure name written in your record.
A&A perspective
Two more triggers can be set. One is a new system appearing that takes on the same task. The other is someone asking what that task would cost. Both arrive from outside rather than from your own calendar.
A&A perspective
The order of work once a trigger arrives is fixed. Push the recorded input through unchanged. Compare it against the expected result. Sort the outcome into same failure, different failure, or no failure, and write it back into the record. Not having to invent a new input is the difference from last time.
Hypothetical example
For the same hypothetical job, the trigger would be "an update note saying that the error of missing a change inside a table has been fixed". All you look for is whether the failure name in your record appears in the wording of the note.
Write the result back and rewrite the next trigger
A&A perspective
Write the result of a retest back into the record whether it passed or failed. Without the write-back, the next retest starts from zero again.
A&A perspective
Results fall into three. The same failure appeared, the failure went but a different one appeared, or the failure is gone.
A&A perspective
When the same failure appears, add only the date and the name of the model you used. Leave the trigger alone. The body of the record is still usable.
A&A perspective
When the failure goes but a different one appears, write the new failure name and rewrite the trigger to that name. If the old trigger is not removed here, the same news will set you moving again for nothing.
A&A perspective
When the failure is gone, the record becomes material for your reply. You can put the content of the "still difficult" answer you gave six months ago next to the input that now passes.
Hypothetical example
In the same hypothetical job, suppose the missed table changes are gone and the error of listing unchanged clauses remains. Write "listing an unchanged clause as changed" into the record as the new failure name. Rewrite the trigger to "news that accuracy in telling similar wording apart has improved". At this stage the answer is still not yes.
From the sources
The same engineering article notes that automated evals also have downsides. Evals that do not match real usage patterns can create false confidence, it points out.
A&A perspective
Meeting the pass criteria written on that one page is still not proof that the capability works in real operations. A&A treats the page as the yardstick for deciding whether a retest is worth running. Whether it can be delivered to a client, or switched on internally, is decided separately.
A&A perspective
The three fields, the placing of the trigger and the three-way sort are all A&A's design. Neither source says anything about keeping a written record of a set-aside decision, or about how to build a trigger.
A&A perspective
There are cases where this way of deciding does not fit. If you only set aside a few things a year and only need to try them when asked, the cost of keeping the records up is the higher one.
A&A perspective
If a few hours a week is all you have for retesting, those hours go further on trying the one thing you were asked about from scratch than on maintaining records. The record starts paying off from the second enquiry about the same task.
A&A perspective
How many times A&A has rerun a retest on a real engagement, and how long one run took, are not shown in this article. Neither the time from setting something aside to adopting it, nor a recommended retest interval, is given as a number.
Summary
A&A perspective
Write the reason for setting a capability aside as the name of the failure that happened, not as a verdict on the model.
A&A perspective
On the day you set it aside, keep the input you gave, the result you expected and the failure that happened on one page. That day is when the pass criteria cost least to write.
A&A perspective
Trigger the retest with news that the failure is fixed, not with a date. Because the news arrives from outside, it is not skipped in a busy month.
A&A perspective
Write the result back into the record whether it passed or failed, and rewrite the next trigger. Each use of the record lowers the effort of the next judgement.
When to retest an AI capability you set aside is decided by news that the failure behind the decision has been fixed, not by a date on a calendar. If the reason survives only as a verdict on the model, six months later you rebuild the way of checking from scratch. Keep the input you gave, the result you expected and the failure that happened on one page on the day you set it aside, and the retest becomes a matter of pushing the same input through. The only usable trigger is an update note you can hold up against the failure name in your record. Write the result back whether it passed or failed, and rewrite the next trigger. Meeting the pass criteria on that page is still not proof that the capability works in real operations, and if you only set aside a few things a year, not keeping the records is the faster route.
FAQ
How detailed does a set-aside record need to be?
Detailed enough that two people looking at it separately reach the same verdict. The same engineering article says a good task is one where two domain experts independently reach the same pass or fail verdict. What decides it is whether the input field and the expected-result field clear that line. Volume of prose matters less than verdicts not splitting.
If a capability passes the criteria in the record, can it go straight into use?
Not straight away. The record is the yardstick for whether the earlier decision has changed, which is separate from judging whether it works in real operations. The same article notes that automated evals can create false confidence when they do not match real usage patterns. Keep a separate step that checks against production volumes.
With several set-aside capabilities, which record should come first?
The one someone is asking about now. If someone is asking, the input examples sit on their side and the expected result is easier to agree on. Because failures blur together once you track several things at once, keep one record per capability.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Genspark's Super Agent orchestrates 150+ tools with Claude
Anthropic · undated customer story page; no publication or update date shown
Accessed 2026-10-23 - Demystifying evals for AI agents
Anthropic · 2026-01-09 published; page metadata shows a 2026-10-18 update
Accessed 2026-10-23