Check AI's “done” and what arrived separately: 4 client-side steps
An AI's “done” means the process ended, so reread where client reports landed to confirm delivery separately.
Key points
- Tie “done” to the client's side
- AI says “done”? Reread where it landed
- Inbox out of sight? Mark it “unverified”
- Check what landed. Resend only the rest
Contents
Tie “done” to the client's side
A&A perspective
Write the condition for a task you gave to AI as one sentence about what must happen on the client's side for it to be finished. Here, an AI's “done” means the report an AI or a process sends back, such as “Finished” or “Shared.”
Hypothetical example
Say a company sends a monthly operations report to 20 clients. The AI gathers the numbers, builds each report, puts it in that client's shared folder and emails them to let them know. This is a made-up example for illustration.
Hypothetical example
If you write done for this flow as “the report was created,” the AI's report alone can close the task. If you write “Client A's contact can open this month's version in the shared folder,” the place you check moves to the client's side.
Hypothetical example
Written this way, that one sentence turns out to hold 4 points to check. Are these this month's numbers? Is it the version checked in-house? Is it in the agreed folder? Can the contact open it? If any one fails, the work is not done from the client's point of view.
From the sources
In an engineering article on evaluating AI agents, Anthropic describes a good test task as one where two experts in the field, looking independently, would reach the same pass or fail verdict.
A&A perspective
Automate & Augment (A&A), the company that writes this blog, applies the same idea to how the condition for done is written. With “the client can open this month's version,” anyone in the company who checks reaches the same verdict.
A&A perspective
With wording like “the report was delivered properly,” what counts as “properly” differs from person to person. Each person who checks may give a different answer on whether the work is finished.
AI says “done”? Reread where it landed
A&A perspective
When the AI replies “done,” take it as the signal to reread, from your own side, the place where the work landed. The report marks the end of a process; it is not proof that the work reached the client.
From the sources
The same article separates the record of what the AI did from the result. The transcript is the complete record of one trial, including the AI's outputs and tool calls. The outcome, in the article's terms, is the final state left in the environment where the AI worked at the end of the trial.
From the sources
Its example is a flight booking. The AI might end by saying “Your flight has been booked.” Even so, the article says, the outcome is whether a reservation exists in the database that records bookings.
From the sources
It also gives an example from tests of AI that operates screens: confirm in the backend records that an order was actually placed, not just that the confirmation page appeared.
Hypothetical example
Back to the report example: the AI's “Shared” only reports that a file was put somewhere. It may have gone to a working folder while last month's version is still in the client's folder. This is also a made-up example.
A&A perspective
So, apart from the AI's report, open the client's folder and read what is in it. The question is not whether a file exists. Check that the month, the version and the main numbers match the version checked in-house.
A&A perspective
Open it with access close to the client contact's. If it opens with the creator's access, that does not mean the client's contact can open it. For example, keep an in-house checking account that, like the client's contact, can only view, and open it with that.
| Sign the AI or process returns | What the sign shows | What to check where it landed |
|---|---|---|
| The report-building process ended normally | The building process ran to the end | Is this month's version in the client's folder? |
| The AI answered “Shared” | The AI reported so | Can that version be opened with access close to the client's? |
| The email send was accepted | The sending process was accepted | Did it reach the client's inbox? |
| A confirmation screen appeared | The screen changed | Is the order or booking in the backend records? |
Can't see where it landed? Mark it “unverified”
A&A perspective
Where your company cannot read the destination, count it as neither success nor failure, and record it as “unverified.” The client's email inbox is the typical case.
A&A perspective
Write the status by how far it got. For example: “Send accepted. Arrival unverified.” Written that way, the next person can decide whether to resend or to ask the client.
From the sources
The evaluation article recommends building in partial credit for tasks with several parts. Its example is a customer support AI.
From the sources
An AI that identifies the problem and verifies the customer but fails to process a refund is meaningfully better than one that fails immediately, the article says. It also says it is important to show this range of success in the results.
A&A perspective
A&A reads this as: do not record work only as “finished” or “not finished.” Keep, step by step, how far it could be confirmed.
Hypothetical example
In the report example, write the steps for each client: “Created,” “This month's version confirmed in the client's folder,” “Email send accepted, arrival unverified.” List how far each one was confirmed.
A&A perspective
Record these 5 things: the client and version, what the AI processed, what could be read at the destination, what is not yet confirmed, and who looks next. This helps the next person act more than attaching the whole process log.
A&A perspective
When you report a delivery rate, also write what you counted as the denominator and the numerator. If unverified items count as successes, the ones that did not arrive vanish from the number.
Check what landed, then resend only the rest
A&A perspective
When work stops partway, look at the destination first, then send only what has not arrived. Redoing everything from the start can send the same client 2 copies.
From the sources
In another article, Anthropic writes about building AI that does its work by using tools (AI agents). It says it is crucial for the AI to get real results from the environment at each step, such as what a tool call returned, to assess its progress.
From the sources
It also says the AI can pause for human feedback at checkpoints or when it hits a blocker. To keep control, it is also common to include stopping conditions, such as a maximum number of iterations.
Hypothetical example
In the report example, say the process stopped after sending to 12 of the 20 clients. Run it again from the start, and those 12 get the same report twice. These numbers are made up as well.
Hypothetical example
Checking the client folders and the send log shows the remaining 8 clients. Send only to those 8.
A&A perspective
Clients whose send was accepted but whose arrival is unverified are not counted in the rest. A person decides whether to resend or to ask the client, so that no client receives the report twice.
A&A perspective
If the version that arrived was wrong, decide who will recheck the version before resending. Put a limit on retries too; past the limit, a person takes over. Do not build it to repeat until the AI says “done.”
A&A perspective
A&A thinks the checking time alone does not decide whether checking is worth it. Compare it together with the time spent correcting wrong sends and handling duplicates.
Summary
A&A perspective
Tie “done” to the client's side, in one sentence.
A&A perspective
Apart from the AI's “done,” reread where it landed, with access close to the client's.
A&A perspective
Where you can't see whether it arrived, mark it “unverified” and do not count it as a success.
A&A perspective
Check what landed, then resend only the rest.
An AI's “done” marks the end of a process, not proof that the work reached the client. Pick one task that reaches clients, write its condition for done as one sentence about the client's side, and decide who rereads the destination and when.
FAQ
Do we reread every task we give to AI, all the way to where it landed?
Start with work that reaches clients. For work like internal drafts, where mistakes can be fixed in-house right away, checking that the process finished can be enough.
Can the rereading itself be given to AI?
AI can open the destination itself, not the AI's own report, and match the month, version and main numbers. A person decides pass or fail and what stays unverified. Anthropic's article on building AI agents says automated testing can verify that code works, but human review remains crucial for whether it fits what the software must do. A&A reads it as: hand over the matching, keep the verdict.
We can't see the client's inbox. How do we check email?
While testing, also send to an address your company controls and check how it arrives. In production, record arrival as unverified, and add to the record when a client's reply or similar tells you.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-10-24 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-10-24