When AI Stops: Count Completion and Human Handoffs Separately
Count human handoffs separately, keep stopped jobs in the completion-rate denominator, and check each deadline.
Key points
- Separate AI completion from handoffs
- Keep stopped jobs in the denominator
- Use reasons and effort to choose fixes
Overview diagram in preparation
Contents
Separate AI completion from human takeover
A&A perspective
A cutoff stops an AI process when it reaches a limit or another agreed condition. Count jobs that stop and pass to a person separately, while keeping them in the total number assigned to AI. Also check whether the business job finished by its deadline.
Hypothetical example
Imagine a production company using AI to prepare estimates. While comparing a customer's request with previous price lists, the process reaches its iteration limit. An employee checks the remaining details and completes the draft estimate. This is a hypothetical example.
From the sources
Anthropic describes setting stopping conditions, such as a maximum number of iterations, to maintain control of an agent. It also explains that an agent can pause for human feedback at checkpoints or when it encounters a blocker.
A&A perspective
Automate & Augment (A&A) proposes reporting the stopping behavior separately from job completion. In this example, the stop mechanism worked, but the AI did not complete the draft estimate on its own.
A&A perspective
Classify the result of the AI's part as a pass, a failure, or a human handoff. A pass means the AI completed the agreed deliverable on its own. A planned stop that a person accepts goes into the human-handoff category. Incorrect output and missed handoffs remain failures.
A&A perspective
Sending a stop notification is not enough to count a handoff. There must be a record that someone accepted the remaining work. Keep the job pending if acceptance is unconfirmed, then judge it against the agreed conditions when the deadline arrives.
A&A perspective
Moving every unexpected stop into the handoff category after the event would hide failures. When accepting a system from a supplier, check that it behaves as agreed. This acceptance check should cover the stopping conditions and the person who receives the work.
A&A perspective
These categories are A&A's operating proposal. Anthropic has not prescribed three categories for monthly reporting. We are applying its description of a process that can stop to a small company's reports from a supplier or internal operator.
Keep stopped jobs in the denominator
A&A perspective
Calculate the AI-only completion rate at the monthly cutoff for jobs accepted that month and assigned to AI. Divide the jobs completed by AI alone by the total in that cohort. Keep jobs taken over by people in the denominator so the report shows how much work finishes automatically.
Hypothetical example
Suppose there are ten jobs: six AI passes, three human handoffs, and one failure. The AI-only completion rate is 60%. Removing the three handoffs from the denominator makes the same workload look more successful.
A&A perspective
Record the outcome after a person takes over in another field for the same job: the estimate was completed on time, completed late, or is still in progress. Do not count an AI handoff and subsequent human completion as two separate jobs.
Hypothetical example
For example, an estimate is due that afternoon. The AI stops as planned and an employee accepts the work, but the estimate is not ready until the next day. The handoff procedure was followed and the business deadline was missed; both judgments remain in the report.
A&A perspective
A comparison table lets the owner distinguish how far the AI got from whether the job was on time. Allowing a handoff does not excuse a missed deadline. For urgent work, the times when a person can accept a handoff also affect whether the system is suitable.
A&A perspective
State the reporting period and cutoff. Retain jobs awaiting acceptance or still in progress as incomplete. Distinguish incomplete work whose deadline has not arrived from work that missed its deadline. Counting only completed jobs would make the longest-running work disappear from the report.
A&A perspective
Exclude only work covered by a rule agreed in advance, such as tasks never assigned to AI. Show the exclusion rules and counts. Removing jobs because they turned out to be difficult makes comparisons between months inconsistent.
A&A perspective
Agree on the unit of counting. If the unit is an estimate request, rerunning it does not increase the job count. Carry unfinished work forward under the same ID without adding it to next month's new requests. Record extra attempts and costs separately, and avoid double-counting employees' work.
| What happened | AI-stage result | Business deadline |
|---|---|---|
| AI completes the agreed estimate on time | Pass: AI-only completion | Met |
| AI stops as planned; a person accepts and finishes on time | Human handoff | Met |
| AI stops as planned; a person accepts but finishes the next day | Human handoff | Missed if due that afternoon |
| No confirmed recipient yet; the deadline has not arrived | Pending: not a completed handoff | Not yet determined |
| No recipient accepts the work before the deadline | Failure: missed handoff | Deadline missed |
Choose improvements from reasons and effort
A&A perspective
Choose the next improvement by looking at why work was handed over and how much human effort it took to finish. A high stop count alone does not show whether raising the limit addresses the cause.
From the sources
Anthropic's article on tool design recommends measuring runtime, the number of tool calls, usage, and tool errors alongside accuracy. A tool is a function the AI calls to perform an action, such as searching or updating a record.
A&A perspective
The article gives examples of errors: choosing the wrong function, supplying incorrect parameters to the right one, making too few calls, or mishandling the returned result. These are examples of possible errors, not a fixed classification of reasons for stopping.
Hypothetical example
In the hypothetical production company, distinguish a job stopped by lack of access to the price list from one where the list was readable but the pricing conditions were unclear. The former points to access permissions; the latter points to the criteria used to prepare an estimate.
A&A perspective
For each row in the report, add the reason for stopping, the recipient, the time they accepted the work, and their working time. Measure the time from when the person starts checking until the remaining work is finished, and record the wait for acceptance separately. An unknown cause can remain unknown while it is investigated.
A&A perspective
If the same cause keeps recurring, the request to a supplier or internal operator becomes more specific than increasing the limit. Test possible changes on the stopped jobs: make the price list accessible, have a person settle ambiguous conditions, or change who receives the handoff.
A&A perspective
For work with a strict deadline and no available recipient, a stop mechanism alone is insufficient. Keep the option of returning that job to a person. Even if working time falls, the saved time does not automatically become profit.
A&A perspective
Both sources explain designs from one developer; they do not evaluate this reporting table. A&A has not measured time savings or an improvement rate either. The question this table can help examine is whether the report makes unfinished work and human effort visible.
Report the stop and the business outcome separately
A&A perspective
Separate AI completion, failure, and human handoffs, and retain evidence that a person accepted each handoff.
A&A perspective
Keep stopped jobs in the total assigned workload. Check separately whether the business job finished by its deadline.
A&A perspective
The reasons for handoffs and the time people spend on them make the next improvement more specific before you consider raising the limit.
Keep AI completion, human handoffs, and failures distinct. Retain stopped jobs in the assigned total, check business deadlines separately, and use the reasons and human effort to choose what to change.
FAQ
Can a handed-over job pass acceptance testing?
If the handoff happened as agreed in advance, that behavior can pass its check. Do not add it to the number of jobs AI completed alone. Record the handoff test result separately from whether the business job was completed by its deadline.
How should a rerun count toward the success rate?
Count the request once. Keep a record of human intervention; an AI rerun after a person fixes the inputs is not an AI-only completion. An automatic retry without human help can qualify. If attempt-level results are also useful, report them separately from job counts, along with extra charges and human checking time.
Should I raise the limit when AI stops often?
First check why it stops. More iterations may not resolve inaccessible documents or undecided business conditions. Test proposed changes on the same stopped examples and compare costs and remaining errors.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Building effective agents
Anthropic · 2024-12-19
Accessed 2026-11-01 - Writing effective tools for agents — with agents
Anthropic · 2025-09-11
Accessed 2026-11-01