Your first hire on AI review: two conditions that make it training
Put your first hire on AI review only when the pass/fail standard is written and caught errors become their knowledge.
Key points
- Decide by two conditions, not headcount
- Write the standard from 3 rejections
- Once a month, they explain logged errors
- Until it's written, they gather material
Contents
Decide by two conditions whether your first hire does AI review
A&A perspective
Whether your first hire should do AI review is decided by two conditions, not by how short-handed you are. AI review here means reading a draft the AI produced, before it goes to a client, and deciding whether it passes or goes back.
A&A perspective
The first condition is that the pass/fail standard is in writing before you hand review over. The second is a routine in which the new person logs the errors they find and, once a month, writes down from that log why they happen.
Hypothetical example
The setting is a company that has built AI into its own service. Say a new line of business drafts client reports with AI, and the person running it checks each one before it goes out. You are now adding the first person to that business, whether they move over from elsewhere in the company or are hired from outside.
A&A perspective
Without the two conditions, handing over review still cuts the time the current owner spends checking. But every draft that gives the new person pause comes straight back. What they learn is to notice that something is off, not why it is off, which is the substance of the work. Review works as training only when that substance stays with them.
From the sources
There is a company that placed review on new hires as training. Anthropic, which makes the AI model Claude, ran workshops for small business owners in several cities and published a write-up of what it heard on September 10, 2026.
From the sources
That write-up describes KBSO Consulting, a 40-person engineering firm in Indianapolis. New hires there review every summary Claude generates. The write-up says this is by design, as on-the-job training.
From the sources
According to the same write-up, the idea that resonated most with owners was treating Claude like a new employee at first and building trust over time. Learning to check AI output rather than take it at face value is described as something owners were newly learning.
A&A perspective
Automate & Augment (A&A), which writes this article, separates two things here. That a company placed review on new hires as training is one thing. Knowing the conditions under which review actually trains someone is another.
A&A perspective
The Anthropic write-up does not say whether the new hires learned the work, or whether KBSO keeps its pass/fail standard in writing. The two conditions in this article are A&A's proposed design.
Write the standard before you hand over review
A&A perspective
The first condition: put the pass/fail standard in writing before you hand over review. Without it, the first hire comes back every time to ask whether a draft can go. The number of calls you make stays the same; only the back-and-forth grows.
A&A perspective
You can check whether the standard is written before anyone joins. Pick the last three drafts you sent back and write down why. If someone who does not know those jobs would make the same call from your sentences, the standard is written.
A&A perspective
If all three work, the first condition is met. If even one is something you can explain out loud but cannot put in writing, it is too early to hand over review.
Hypothetical example
Say a business drafts monthly client reports with AI, and the person running it checks 40 a month before sending them. The number is only for illustration. Written out, the reasons for sending drafts back look like "the source of a figure does not match the text" or "compared against the wrong month of the prior year."
From the sources
Others have set the point where work goes to a person. Gumloop makes a tool for building AI automations without code. In a post on February 17, 2026, it described its own support operation, run by two people.
From the sources
According to the post, the system that watches for service problems alerts a person only when it finds something that needs action. By the time a person opens a ticket, the user's details, usage history and error counts are already gathered.
A&A perspective
A&A reads this as drawing the line for handing work to a person in advance. Handing review to a person needs the same line: what state of a draft means send it back, and what state means let it through. With that written down, the first hire can pass drafts on their own judgment.
A&A perspective
The table below shows how the two conditions line up and what to give the first hire in each case.
| Written pass/fail standard | Error log and monthly read-back | What to give the first hire |
|---|---|---|
| Yes | Yes | AI review |
| Yes | Not yet | Set up the logging and read-back rules first, and hand over AI review the same day |
| No | Either way | Material gathering (listing past drafts and client comments) |
Once a month, have them explain logged errors
A&A perspective
The second condition is a routine in which the first hire, once a month, writes down from their log of found errors why those errors happen. A send-back is settled on the spot, so without a routine, nothing of what they found remains.
From the sources
The same Anthropic write-up also tells the story of Tony Severino, who has run his own painting firm near Chicago since 1984.
From the sources
According to the write-up, Severino once caught Claude using a floor-area multiplier instead of the real wall measurements on an early bid. He now asks Claude to “show me your work.” The write-up presents this as a result of Anthropic teaching owners, in its course and workshops, how to judge AI output.
A&A perspective
A&A reads this as a found error leading to a change in how he asks for work. The write-up itself does not draw that link between the error and how he asks now.
From the sources
Gumloop's support runs a routine every month that reads back the past 30 daily summaries and pulls out the most common error patterns. It checks the results against the list of work not yet started and suggests new work to fix the common problems. The post says the operation grew one workflow at a time over months.
A&A perspective
A&A borrows two shapes from these. In one, a found error becomes a change to the next request or to the standard. In the other, errors collected over a set period are read back and named as a pattern. Gumloop's system, though, turns what machines find into work on the product; it is not designed to develop people.
A&A perspective
Moved to a person, the routine becomes two rules. One: record each send-back on a single sheet. Two: at month end, the first hire reads the record back, names the most common pattern, and writes why it happens in terms of how the client uses the report.
Hypothetical example
In the report example, the first hire comes to be able to write: "This client looks at year-on-year figures, so mixing up the comparison month hurts most." That is not speed at checking. It is knowledge of the work itself.
A&A perspective
Set up this record on the day you hand over review. Sending back is the work in front of you; reading back is not. In A&A's view, a routine you try to add later is less likely to get built.
If the standard can't be written yet, they gather material
A&A perspective
While the pass/fail standard cannot be written, give the first hire material gathering for the standard instead of review. Material gathering means listing past drafts side by side with the comments clients sent about them.
A&A perspective
This work puts the first hire in touch with the substance of the business. The finished list becomes the material for writing the standard in the first condition. Hand over review only after the standard has been written from it.
Hypothetical example
In the report example, that means lining up past monthly report drafts with client comments such as "that's the wrong comparison month." When the same kind of comment shows up again and again, that pattern is a candidate sentence for the standard.
A&A perspective
Check whether the standard is written the same way as in the first condition. Write send-back reasons from the patterns in the list; if someone who does not know the jobs makes the same call from them, it is time to hand over review.
A&A perspective
You can explain the assignment to your manager or board with the table above. If you have both the written standard and the rules for logging and reading back errors, give them AI review. If the standard exists but the logging rules do not yet, set up the rules first and hand over review the same day. With no standard, give them material gathering, whatever the state of the log.
A&A perspective
Sometimes this decision is not needed. If a draft passes or fails on format checks alone, hand the checking to a machine before giving it to a person. A format check means looking at things like how dates are written or whether table columns line up. People check only the parts where the call depends on the business.
A&A perspective
One can also argue that where a senior colleague sits nearby to ask, the standard can be built while reviewing. KBSO is a 40-person firm. In a new business with only one person running it, there is no one to ask, so A&A thinks a written standard matters more.
Summary
A&A perspective
Whether your first hire does AI review is decided by two conditions, not by how short-handed you are.
A&A perspective
First, write the pass/fail standard from the last three drafts you sent back.
A&A perspective
Second, set up a routine in which the first hire, once a month, writes down why from their error log.
A&A perspective
While the standard cannot be written, give them material gathering, lining up past drafts and client comments, instead of review.
Whether your first hire does AI review comes down to a written pass/fail standard and a monthly read-back of logged errors. Start by trying to write the reasons for your last three send-backs; if you cannot, make the first hire's job gathering material for the standard.
FAQ
How many cases should the pass/fail standard be written from?
This article's guide is the last three drafts you sent back. Three is A&A's rule of thumb for checking, not a figure from the sources. If you can write all three so that someone who does not know the jobs makes the same call, you are ready to hand over review.
When hiring someone for review, how should I write the job description and evaluation?
The written pass/fail standard becomes the material for both. If you hire someone "for review" before the standard can be written, there is nothing to put in either. Employment terms differ by country and system, so check with a specialist before hiring.
What does treating AI like a new employee mean?
Anthropic's write-up says the idea that resonated most with owners was treating Claude like a new employee at first and building trust over time. A&A reads it this way: early on, both the AI's drafts and the first hire's calls are on the side being checked. Checking either one needs a written standard.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- What 1,000 small business owners taught us about AI
Anthropic · 2026-09-10
Accessed 2026-10-24 - Supporting the world’s most AI-native companies with a 2-person team
Gumloop · 2026-02-17
Accessed 2026-10-24