Explaining why your AI agent did that: 4 decisions before you outsource
Whether you can tell staff or clients why the AI did something depends on 4 decisions made before ordering.
Key points
- Log uncalled tools, plus calls (lookups)
- Set retention by the longest query delay
- Pick in-house readers, not vendor-only
- Put a cap in the quote for reading work
Contents
Log uncalled tools, not just the tools called
A&A perspective
Have the vendor build the AI agent so that it logs the tools it called and their arguments, what came back, and the tools it could have called but did not. A run record here means the step-by-step record of what the AI called and received while doing one job.
A&A perspective
An AI agent moves a job forward by choosing and calling operations on its own, such as looking up a client ledger or sending an email. Each of these operations is a tool, and the values passed to a tool are its arguments.
From the sources
In a technical post on writing tools for agents, Anthropic lists four ways an agent can go wrong: calling the wrong tools, calling the right tools with the wrong parameters, calling too few tools, or processing tool responses incorrectly.
A&A perspective
Automate & Augment (A&A), the company writing this article, focuses on calling too few tools. A tool that was never called appears nowhere in the output, so unless the record lists the tools that were available at the time, no one can notice the gap later.
Hypothetical example
Say, for example, that you outsource an AI agent that routes incoming invoices to staff by client and amount. This is a hypothetical example.
Hypothetical example
For each invoice, the record shows lines such as “looked up the ledger, got the client number”, “called the amount check, got a category” and “could have called the exceptions list, did not”. With that third line, when someone asks why the invoice was not treated as an exception, you can answer that the exceptions list was never checked.
From the sources
The same post says that what agents omit in their feedback and responses can often be more important than what they include.
From the sources
To investigate, it recommends reviewing the raw transcripts, including tool calls and tool responses. The aim is to catch behavior not described in the reasoning the agent wrote out. It also says that the AI does not always say what it means.
A&A perspective
A&A's view is that asking the AI for its reasons afterwards does not replace the record. Asked, an AI will give an explanation that sounds reasonable, but nothing guarantees that it matches what the AI actually did.
Set retention by how late questions can arrive
A&A perspective
Decide how many days of records to keep by the longest time it takes staff or clients to come back and ask why a case was handled as it was. There is no general right answer for retention, so this article does not recommend a number of days.
A&A perspective
The longest time here means the longest gap between the day the AI handled a case and the day someone asks about it.
Hypothetical example
In the invoice example, suppose a client sometimes asks in the middle of the next month about an invoice closed at month end. Then records must be kept longer than the days from closing to that question.
A&A perspective
When someone asks why the agent acted differently last month and this month, you need both months' records. If you also want to answer questions that compare months, choose a number of days that keeps both months' records.
A&A perspective
Keeping records longer is not automatically better. Records can include client names and amounts, and the longer you keep them, the more information you hold. If you decide to keep records up to the longest delay and delete them after that, the vendor can also estimate the storage cost.
A&A perspective
Some work does not need a record. If it takes one input and makes one decision, running it again on the same input and showing the result is cheaper. A record pays off when the work has several steps and a re-run is not guaranteed to follow the same path.
From the sources
The post on writing tools also says that agents can give different responses even when they start from the same conditions.
Hypothetical example
The invoice example has several steps: look up the ledger, check the amount, and choose whether to check the exceptions list. Even if a re-run routes the invoice correctly, that does not show how the case was handled last month.
A&A perspective
The table below lists common questions, whether the output alone can answer each, and the record each answer needs.
| What you are asked | Answerable from output alone | Record the answer needs |
|---|---|---|
| Why did this case go to this person? | Only partly | Tools called and arguments |
| Why was it not treated as an exception? | No | Tools it could have called but did not |
| Why did it act differently last month and this month? | No | Both months' records |
| How much does this work cost each month? | No | Call counts, time and failures |
Pick in-house readers so the record is not vendor-only
A&A perspective
Before ordering, decide who in your company can read the record, and where, instead of leaving it only in the vendor's environment. If only the vendor has it, you wait each time someone asks, and you may lose access when the contract ends.
From the sources
In a separate post on evaluating AI agents, Anthropic defines the transcript. It is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions. In the same post, Anthropic says it invested in tooling for viewing transcripts and regularly takes the time to read them.
A&A perspective
What the ordering side needs is a record you can look up by case number, with the tools called, arguments, results and uncalled tools in time order. If the owner can open and read it, simple questions can be answered without waiting for the vendor.
A&A perspective
At acceptance (checking delivered work before taking it on), pick any one case and check with the vendor that all four of these show up. Correct output passes acceptance, so a gap in the record tends to surface only when someone asks for an explanation.
Hypothetical example
In the invoice example, entering an invoice number opens that case's record, showing when the ledger was looked up, the result of the check, and the exceptions list that was not checked.
A&A perspective
Limit readers to the people who answer questions. Records include client names and amounts, so the more readers you add, the more people see the information you hold.
A&A perspective
Also decide before ordering where records still inside the retention period will go when the contract ends.
Put a cap in the quote for reading and answering
A&A perspective
Have the vendor write the work of reading records and drafting answers into the quote as a paid line, with a monthly cap on how many cases the running fee covers. Even with a complete record, turning it into an answer in plain language is still work for a person.
From the sources
The post on evaluation lists the weaknesses of people reading transcripts by hand: it is time-intensive and does not scale. It then names ongoing practices that fill the gaps left by other methods: triaging user feedback constantly and sampling transcripts to read weekly.
From the sources
The post on writing tools recommends collecting, as well as top-level accuracy, the number of tool calls, the runtime, the token consumption (how much the AI read and wrote), and tool errors.
A&A perspective
A&A's view is that once you have the record used for explanations, you also have what you need to check the monthly cost. Call counts, runtime and the number of failures can all be counted from the same record.
A&A perspective
Ask for three things in the quote: what one case's investigation covers, the monthly cap, and what happens past the cap. The cap expresses, as a number of cases, the most time the vendor can spend reading records. Work that is not in the quote depends on the vendor's goodwill, and no one promises how fast answers come back.
Hypothetical example
In the invoice example, the scope of one case could read “read that invoice's record and answer which tool's result decided the assignee”. For cases past the cap, decide whether they roll into the next month or are billed one by one.
A&A perspective
How high to set the cap depends on how many why-questions your company gets a month. So this article does not recommend a number.
Summary
A&A perspective
Log the tools the AI called (actions such as a ledger lookup), plus the tools it could have called but did not.
A&A perspective
Keep records for the longest time from a case to a question about it, and no longer.
A&A perspective
Decide before ordering who in your company can read the record, and check one case's record with the vendor at acceptance.
A&A perspective
Have the vendor put the reading and answering work in the quote as a paid line with a monthly cap.
Whether you can answer why your AI agent did something depends on what the record holds, how long it is kept, who in your company can read it, and the cap on reading work, all decided before ordering. What to decide next is to count how many days after a case, and how many times a month, your company is asked why.
FAQ
If the vendor says “we keep logs”, is that enough?
You cannot tell until you see what is in them. Ask to see one sample case, and check whether the tools it could have called but did not are there, how many days are kept, and whether your company can read it.
Does the record also fix output that differs every time?
No. The record answers how a particular case was handled. Whether the same request gives the same result is a matter of acceptance criteria, covered in “AI deliverable acceptance criteria: 4 terms for output that varies”.
Can client names and amounts go into the record?
Yes, within the range of customer data you decided to give the AI. How to set that range is covered in “How much customer data to give an AI contractor: decide 3 tiers first”.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Writing effective tools for agents — with agents
Anthropic · 2025-09-11 (page metadata records a 2026-10-13 update)
Accessed 2026-10-24 - Demystifying evals for AI agents
Anthropic · 2026-01-09 (page metadata records a 2026-10-18 update)
Accessed 2026-10-24