From AI news to business decisions: 4 steps to redraw the human-AI line
When AI news breaks, check the terms, run a test set of past work, then decide whether to redraw the human-AI line.
Key points
- Read the terms, like plan and language
- Run a test set of 20 past invoice checks
- Judge the actual output, not "done"
- Redraw, keep or hold the human-AI line
Contents
Read the terms apart from the features
A&A perspective
Turning AI news into a business decision means deciding again, after you read an announcement, which work stays with people and which work you hand to AI. This article calls that division the human-AI line.
A&A perspective
For example, whether to have AI do the first pass of the invoice matching you now pay a bookkeeper for. Shifting work like this from people to AI is what this article calls moving the line.
A&A perspective
Start by reading each announcement in two parts: what the AI can now do, and the terms of use, meaning whether the conditions for using it are in place at your company.
A&A perspective
A headline usually tells you only the first part. The terms sit in the notes and tables of the vendor's own documentation.
From the sources
Take the official documentation from Microsoft, the company behind Word and Outlook, as an example. Microsoft 365 Copilot, the AI built into its office apps, has a feature that builds an agent from a plain-language description of what you want. The documentation says you can try the agent you built right on that screen.
From the sources
The same page also lists the conditions that limit where it works. Which internal knowledge sources an agent can draw on depends on the user's license. Building an agent by describing it in plain language works only when Microsoft 365 is set to one of the supported languages.
From the sources
Some capabilities are still in preview and are available only to organizations enrolled in Microsoft's early-access program. The page also says that actions connecting to external services cannot be built with this feature.
A&A perspective
Automate & Augment (A&A), which writes this article, draws the line between people and AI for companies and takes the work from design through implementation. Our reading is that, for a small company, many announcements drop out at this stage of checking the terms.
A&A perspective
For example, the feature may not be included in your current plan, may not work in your language setting, or may not connect to the tools you already use. If so, you can close the question before testing anything. If you do not use Microsoft, look at the same places in announcements for the tools you do use.
A&A perspective
Keep the date you read the page as well. When A&A checked this page in September, its update date was August 19. Opened again on October 21, it showed September 30. Official documentation also changes after an announcement, and a date lets you compare next time whether the conditions have moved.
Run a test set built from your past work
A&A perspective
When an announcement passes the terms, test it with your own test set. A test set here means a collection of tasks taken from your own past work, each with a known correct answer. Work without a single correct answer, such as judging whether a piece of writing is good, needs a person to grade it and takes longer. Start with work where the answer is fixed.
From the sources
Anthropic has published a guide for development teams that build AI agents on how to design evaluations. It defines an evaluation as testing an AI on tasks with set inputs and success criteria.
A&A perspective
The test set in this article is that kind of evaluation.
From the sources
The guide says this is where teams split when a more capable model comes out. Teams without evaluations face weeks of testing, while teams with them can find the new model's strengths and switch over in days.
From the sources
It also says how to start. Twenty to fifty simple tasks drawn from real failures make a good starting point: begin with the checks you already do by hand, then use bug reports and support requests as material.
A&A perspective
A&A thinks the same idea applies to companies that use AI rather than build it. For an owner, past examples of the work you outsource are enough for a test set. You need no new system: one spreadsheet listing inputs and correct answers is enough to start.
A&A perspective
When you test, give the new model or feature the 20 tasks one at a time and paste each result next to its correct answer.
Hypothetical example
Suppose, for example, that you pay an outside bookkeeper to match each month's invoices against purchase orders. Pick 20 past cases and make sure several of them are cases where the amount or quantity did not match. Which cases did not match, and where, is the correct answer for this test.
A&A perspective
Before you give real cases to an AI, check your contract and confidentiality terms with the provider and how the AI service handles your data. Work you are not allowed to give to an AI does not go into the test set.
A&A perspective
The benchmarks listed in announcement headlines are shared tests for comparing AI performance. Their scores are results on tests someone else wrote. Only your own test set tells you how the AI will do on your work.
From the sources
Anthropic itself says it does not take evaluation scores at face value until someone has dug into the details of the evaluation and read the transcripts of what the AI actually did.
A&A perspective
The feel you get from using Claude Code or ChatGPT yourself is also a result only for the work you tried. Whether the work you outsource feels the same is something you check with a test set for that work.
| Point | Passes when | Invoice example |
|---|---|---|
| Cases that did not match | It listed them all | Every case where the amount or quantity did not match is in the AI's matching table |
| Cases that matched | It did not list them by mistake | The AI did not list cases where both amount and quantity match |
| Two runs | Both runs gave the same result | Running the same 20 cases twice listed the same cases both times |
Judge the actual output, not the "done" reply
A&A perspective
Grade what was actually produced, not what the AI says.
From the sources
Anthropic treats the AI's reply separately from the result left behind afterwards. Its example is a flight-booking agent: even if it answers "Your flight has been booked," what counts is whether the reservation actually exists. Because model outputs vary between runs, the guide also says to run the same task several times.
From the sources
It also recommends testing both the cases where you want the AI to act and the cases where you do not. Anthropic gives web search as its own example: testing only questions that need a search risks an AI that searches for everything, so it added questions that need no search to its tests.
Hypothetical example
In the invoice example, the cases where both amount and quantity match are the cases where you do not want the AI to act. That gives three points to grade. Did it list every case that did not match? Did it avoid listing cases that did match? Did running the same 20 cases twice give the same result?
Hypothetical example
A reply saying "Matching complete" answers none of the three. Put the matching table the AI produced next to the correct answers and look at that.
Hypothetical example
A case whose result changed between the two runs is one where, once handed over, the result will swing from month to month. Keep that case on the side people check, even after you move the line.
From the sources
Decide who grades, too. Anthropic says the people closest to the requirements and the users are best placed to define what success means.
Hypothetical example
In the invoice example, that is the bookkeeper who does the matching today. If you ask the bookkeeper to write the correct answers and do the grading, you only have to read the results and decide. It is extra work, so budget for the fee. When you ask, say up front that they will keep a checking role after the line moves.
A&A perspective
Checking the reply and the result separately is also covered in "Check AI's “done” and what arrived separately: 4 client-side steps".
Decide from the test results whether to redraw the human-AI line
A&A perspective
Once the test results are in, decide whether to move the line for that work, keep it where it is, or put the decision on hold. Record the decision in one line with the date and the reason.
A&A perspective
If the AI lists the cases that do not match, leaves the matching ones alone and gives the same result both times, move the line. The AI does the matching first; the bookkeeper checks the cases the AI listed and also picks a few it did not list to check.
A&A perspective
If it misses on any of the three points, keep the line where it is. Keep the missed cases in your test set and run the same tasks on the next announcement.
A&A perspective
If the terms do not fit and you cannot test, put it on hold. Write down, with the date, which condition would have to change before you test, and close the question for now.
A&A perspective
Even when you move the line, keep a checking role for the outside provider, so that if the AI misses you can return to the old split at once. How to draw the line itself is covered in "What to hand to AI and what people decide: three places for approval".
From the sources
Anthropic writes that it often builds features that work only well enough today, as bets on what models will be able to do in a few months. When a new model comes out, running the evaluations quickly shows which of those bets paid off.
A&A perspective
The test set for work you kept on the people side can be used the same way. Even for work you cannot hand to AI today, running the same tasks on the next announcement tells you whether the day to move the line has come.
From the sources
Anthropic also gives an example of evaluations missing progress. Qodo, a startup that does code review, was at first unimpressed by a new model because it evaluated only with tasks finished in a single exchange. The gains were on longer, more complex tasks.
A&A perspective
If your test set holds only short tasks finished in one pass, add one longer piece of work. That makes this kind of miss easier to notice.
Summary
A&A perspective
Read each announcement as what it can do and the terms for using it, such as plan and language, and close the question when the terms do not fit.
A&A perspective
If the terms fit, test it with a test set of past examples of the work you outsource, such as invoice matching, each with its correct answer.
A&A perspective
Grade what was produced, not the reply, and include the cases where you do not want the AI to act.
A&A perspective
From the test results, decide whether to move the line, keep it or put it on hold, and record the date and the reason in one line.
A&A perspective
A&A learns how far AI can go today from running it in its own daily operations. From there it draws the line between people and AI, and the same person carries the work from design through implementation.
Use AI news as input for deciding whether to move the line between people and AI, based on the terms of use and the results of your own test set. With a test set at hand, the next piece of news becomes something you check by running a test, not something you chase by reading more.
FAQ
How many tasks does a test set need?
About 20 is enough to start. Anthropic tells development teams that 20 to 50 simple tasks is a good place to begin. Before the count, choose cases whose correct answer is clear.
If an announcement shows higher benchmark scores, can I hand the work over on that basis?
Not on the score alone. Anthropic itself does not take evaluation scores at face value until someone has dug into the evaluation and read the transcripts of what the AI actually did. Check how it does on your work with your own test set.
There are too many announcements to test them all.
You do not need to test them all. Close the question on any announcement whose terms do not fit. Test only the ones whose terms fit and that touch work you already have a test set for.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Build agents in Agent Builder
Microsoft · Updated 2026-09-30
Accessed 2026-10-21 - Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-10-21