A&A INSIGHTS
Before taking on more AI delivery work, estimate where human review jams
Faster generation does not raise how much AI service work a small firm can deliver — the founder's review hours do. Separate routine checks, exceptions and redelivery, then use Anthropic's grader taxonomy and Gumloop's own support operation to decide which checks can move and what moving them costs.
日本語で読む
THE STARTING POINT
In a one- or two-person firm delivering AI-generated work to several customers, the schedule is set not by production time but by the hours the founder spends bringing output to the agreed quality. This article argues that capacity is governed by review time, and that the term which matters most is the routine check applied to every item, not the conspicuous exceptions. It reads Anthropic's three grader classes — code, model-based (needing calibration against human graders) and human (gold standard, but slow) — as a sheet for sorting a service firm's pre-delivery checks, alongside Gumloop's account of its two-person support operation. A fictional firm fills in that sheet to show the order.
Generation speed is not what caps the number of jobs you can take
A&A perspective
Asked "the draft now comes out in ten minutes, so can we double our intake?", A&A answers that generation time does not appear in the denominator. In a one- or two-person company delivering AI-generated work to several customers, the schedule is usually set not by the time spent producing but by the time the founder spends bringing the output up to the agreed quality. That is the quantity to estimate before increasing intake — and unless you split it into routine checks, exception handling and redelivery, you cannot see which lever actually raises the number of jobs.
From the sources
Anthropic's "Building effective agents", published on 19 December 2024, states that "agentic systems often trade latency and cost for better task performance" and advises considering when that tradeoff makes sense. On autonomous configurations it adds that "the autonomous nature of agents means higher costs, and the potential for compounding errors."
From the sources
An appendix on the same page names coding as a domain where output quality can be measured objectively, and then says that while automated testing helps verify functionality, "human review remains crucial for ensuring solutions align with broader system requirements." It also lists the conditions under which agents add the most value: tasks that require both conversation and action, have clear success criteria, enable feedback loops, and integrate "meaningful human oversight".
A&A perspective
A&A reads this as the floor that matters for a service estimate. Even in coding, the domain where automatic verdicts should work best, human review does not disappear. So you cannot build a capacity estimate that assumes no person touches any item. If you cannot remove that review, it is more practical to count how many minutes it takes and put the number in the denominator than to keep trying to eliminate it.
A&A perspective
This article is about capacity: how many jobs you can accept per week or month. The separate question of what one job costs and whether the product is worth continuing is covered in "Low-ticket service economics depend on more than generation speed". The denominators differ: economics is denominated in cost, capacity in the founder's hours. Some firms are constrained by both, but the remedies are different, so look at them separately.
Split human time into routine checks, exception handling and redelivery
A&A perspective
A&A suggests three buckets. Routine checking is the inspection and matching that every delivered item requires without exception. Exception handling is the re-investigation and customer clarification that occurs only when an input falls outside what you assumed. Redelivery is the time spent fixing work that has already been delivered and sent back. The three scale against volume in completely different ways: routine checking applies to every item, exception handling only at the rate exceptions occur, redelivery only at the rate work is sent back. As long as you call all of it "checking", that difference stays invisible.
A&A perspective
So if H is the review time available in a month, the number of jobs N you can accept takes roughly this form: N ≈ H ÷ (routine check time + exception rate × exception handling time + rework rate × redelivery time). The formula looks obvious, but what matters is what is missing from it. Generation time is not in that denominator. If generation becomes twice as fast, N does not move. To move it you have to reduce one of the three terms — that is the whole content of the formula.
From the sources
The automation platform Gumloop, in a post published on 17 February 2026 under the name Max Brodeur-Urbas, describes running its support operation on its own product and writes that "our support team has only two people". The same post cites, as the company's own figures, more than 500,000 support-related workflow runs per week, 18 unique MCP tools integrated, and a ticket response time under five minutes.
A&A perspective
This is the company's own account, not an audit verified by a third party. The 500,000 figure counts support-related workflow runs, not tickets, and should not be read as a volume of inquiries. Supporting your own product also carries a different scope of responsibility from service work delivered to a customer and subject to acceptance. A&A still finds it worth reading for service firms, because what is designed there is not what the two people stopped doing but what reaches the two people in the first place. Only that design idea is carried forward below; none of the numbers are transferred into your own estimate.
| Tier of check | What belongs here | What it costs a service firm |
|---|---|---|
| Decidable by matching against the input material | Part numbers, dimensions, quantities, character counts, presence of required fields — anything where the pass/fail reason fits in one sentence and another person reading that sentence reaches the same verdict | Built once. You still need something that notices when the format of the input material changes |
| Decidable once you write the standard | Tone, prohibited expressions, statements not grounded in the input material — the reason can be written down, but verdicts still vary | A calibration period spent comparing model verdicts against human ones. Review time goes up while the check is being moved |
| Only a person can decide | Customer-specific policy, judgments that carry liability, requests with no precedent | Stays proportional to volume. This is the denominator you set your intake against |
| Must not be handed over | Checks where an error surfaces as damage on the customer side, or where you carry contractual responsibility | Excluded from automatic verdicts and budgeted up front as exception handling |
| Not yet sortable | Checks where you cannot yet put your own reason for sending work back into words | Counted as human time for now; moved up a tier once the reason can be written |
Do not make the review faster — reduce what reaches the person
From the sources
According to Gumloop's post, every time a support ticket opens, a workflow queries internal databases and pushes a dossier of the user's profile, activity history and error counts into the ticketing tool. That data is synced every 30 minutes, so that "by the time a human support agent opens the ticket" the most up-to-date information is already there.
From the sources
The post also describes an agent that monitors platform health: "if (and only if) a finding is actionable, it alerts a human." An agent that diagnoses inquiries "first determines what kind of issue it’s dealing with" — a workflow failure, a how-to question, or an agent issue — and chooses which tools to use next accordingly.
From the sources
Anthropic's article on agent design organises this shape as a standard pattern called routing. "Routing classifies an input and directs it to a specialized followup task", and it is described as a good fit where there are distinct categories that are better handled separately and where that classification can be performed accurately.
A&A perspective
A&A's transfer is this. In most service firms where review time is swelling, every generated item is queued in one undifferentiated line and the founder works down it from the top. Insert one classification step and the content of the routine check itself changes. Items decidable mechanically by matching against the input material get cleared first, and only two kinds of item remain in the human queue: those where the verdicts disagree, and those where the input was insufficient to begin with. The point is not that the person reads faster, but that both the number of items in the human queue and the number of judgments left in each one go down. Both sources describe a vendor's own product or a general design pattern; applying them to service delivery is A&A's hypothesis.
Hypothetical example
The following is a fictional example constructed by A&A, not a real customer or result. Consider a two-person firm delivering product description copy to five e-commerce businesses. Today the founder reads all 600 items a month on screen, checking part number spelling, dimensions, materials, prohibited expressions and character count all at once. Insert one classification step and part number, dimensions, materials and character count can be decided mechanically by matching against the input material; what stays in the human queue is only the items where a mismatch was flagged, the items containing statements with no basis in the input material, and the items that need a judgment about wording. The number of things to look at in each remaining item also drops from five to one or two.
Sort the checks into three tiers: matching, standard, and person
From the sources
Anthropic's guide to evaluation design, published on 9 January 2026, organises grading mechanisms into three kinds and lists the strengths and weaknesses of each. Code graders are objective, reproducible and easy to debug, but brittle to valid variations that do not match the expected pattern. Model-based graders are flexible, scalable and able to handle freeform output, but are non-deterministic, "more expensive than code", and "requires calibration with human graders for accuracy". Human graders offer "gold standard quality" matching expert user judgment, but are described as expensive and slow.
From the sources
The same guide recommends that rather than showing every dimension to a single grader at once, you should create clear structured rubrics per dimension and then "grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions".
A&A perspective
A&A proposes using these three classes as a sheet for sorting the checks a service firm performs. The source classifies graders for building an evaluation system, but the question — what do you decide, at what accuracy, at what cost — is identical to the question facing pre-delivery review. The procedure is to write out the checks you currently perform one by one and apply two questions to each: can you state the pass/fail reason in one sentence, and would another person reading only that sentence reach the same verdict? Both yes puts the check in the matching tier; only the first puts it in the standard tier; both no leaves it with a person. The advice to separate dimensions transfers directly as well. Ask a single grader to look at the whole item and say whether it is good, and you learn nothing about which dimension failed, so you have nothing to fix.
Hypothetical example
Sorted for the fictional firm above, it comes out like this. Matching tier: do part number, dimensions and material agree with the input material, are the required fields filled, is the character count in range. Standard tier: does it follow the per-brand tone rules, does it avoid prohibited expressions, does it avoid stating specifications absent from the input material. What stays with a person: the customer-specific policy judgment about what to emphasise for this product this season, and requests with no precedent. A healthy sort has the smallest population in the third tier; if that tier is crowded, suspect that the work has not yet been carved into a product.
The cost of moving a check to a machine arrives first, as calibration
From the sources
After describing per-dimension grading, the evaluation guide states that once the system is robust, it is "sufficient to use human review only occasionally". At the same time it says model grading often takes careful iteration to validate accuracy, and that LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between human grading and model grading.
From the sources
Anthropic's agent design article gives two signs that a generator-plus-evaluator configuration is a good fit. The first is that "LLM responses can be demonstrably improved when a human articulates their feedback"; the second is that the LLM can itself provide such feedback.
A&A perspective
A&A suggests using those two as the questions to ask before moving a check to a machine. First: when you send work back on this check, do you actually put the reason into words? If you skip the words and just show a corrected version, the check cannot move yet. Second: can you write that reason looking only at the deliverable and the input material? If it requires past correspondence with the customer or an unstated industry assumption, the check cannot move until that assumption is written down somewhere. The two questions decide whether a check can move and, at the same time, tell you what you would have to write in order to move it. Both sources describe the design of agent configurations; applying them to a service firm's review process is A&A's reading.
A&A perspective
This is also the step most often mis-estimated in practice. The moment the decision is made to move a check, people book the freed hours starting that month. What actually comes first is the calibration period — running both the person and the model over the same deliverables and closing the gaps between their verdicts. A&A recommends keeping that period inside the estimate and writing down explicitly that review time goes up temporarily while the check is being moved. That is the reason not to schedule a plan to increase intake and a plan to move a check in the same month.
Cap redelivery, and keep a step that confirms the fix worked
From the sources
On autonomous configurations, the agent design article notes that it is common to include "stopping conditions (such as a maximum number of iterations)" in order to maintain control. It also describes agents gaining ground truth from the environment to assess progress, and pausing for human feedback at checkpoints or when encountering blockers.
A&A perspective
A&A recommends carrying this into the contract. For the same reason you set a maximum iteration count on the technical side, the commercial side needs a ceiling: this price includes up to two redeliveries, and anything beyond that is quoted separately. Without a ceiling, one item's correction loop can consume a week of review time and move the deadlines of other customers with it. In terms of the formula above, even a low rework rate leaves capacity uncomputable if the redelivery time per item has no cap.
From the sources
Gumloop's post describes a workflow that runs daily to identify users to follow up with, in order to "ensure that the solutions the support team provided actually work". It also describes an agent that runs whenever a fix ships, informing the users whose issues it relates to.
A&A perspective
If you want to use a rework rate in an estimate, you need this step. Send the correction and consider the matter closed, and any item that was not actually fixed reappears inside the next request and gets counted as new work rather than redelivery. The denominator is then wrong and you overestimate your capacity. At the scale of a small service firm none of this needs automating: a single round trip a fixed number of days after delivery, asking whether the correction resolved the issue, is enough to turn the rework rate into a measurable number.
Hypothetical example
The contract clause for the fictional firm would read: rework requests accepted within 7 days of delivery, up to twice per item; a third request, or a rewrite caused by a change to the input material, is quoted separately; a single confirmation message is sent 10 days after delivery, and the item is treated as complete if there is no reply. The wording itself is hypothetical and the real terms vary by product and customer. What matters here is that the ceiling and the one confirmation round trip are not a matter of contractual courtesy — they are the conditions that make intake volume computable at all.
Fill in the capacity estimate, with the assumptions written down
A&A perspective
There are six cells: hours available for review per month, routine check time per item, exception rate, exception handling time per case, rework rate, and redelivery time per item. Do not fill them from memory the first time — measure for two weeks. No stopwatch is needed; recording, for each item, which of the three categories it fell into and the time you finished is enough. What comes out is your own current value, not an industry standard and not a number that transfers to another firm.
Hypothetical example
Everything below is a hypothetical assumption, not a measured result from A&A or a customer. Suppose the firm above handles 600 items a month with 60 hours available for review. At 3 minutes of routine checking per item, 600 items is 30 hours. At an 8% exception rate and 25 minutes per case, 48 cases is 20 hours. At a 5% rework rate and 15 minutes per item, 30 items is 7.5 hours. The total is 57.5 hours. Against 60 available, that is effectively at the ceiling, and increasing intake in that state increases lateness rather than revenue. Now: halving exception handling frees 10 hours, while cutting routine checking from 3 minutes to 1 minute frees 20 hours. Exceptions are what you notice, but the term that applies to every item is the larger one — and that ordering is what the sheet makes visible.
A&A perspective
This reading carries conditions. First, if no additional work actually arrives to fill the freed hours, no cash appears; do not convert spare time directly into gross profit. Second, getting routine checking down to one minute requires the sorting and calibration from the previous sections first. The sheet is a tool for choosing what to do first, not a promise of a reduction rate. Third, if the assumption of 600 items a month does not hold, the ordering changes too: in work with few items and a heavy unit of delivery, the exception handling term can be the larger one, and then the place to start is different.
Where this estimate does not hold, and the next step
A&A perspective
Start with where it does not apply. In work where the specification differs for every item and the content of the routine check changes each time, routine check time per item is not stable. Estimating volume with this formula then means nothing, and what needs deciding first is not intake but a narrower scope of work. The formula also does not apply where review can be shared across several people, because its denominator is a single founder. If you can hire, you will be drawing the same sheet per reviewer afterwards.
From the sources
Gumloop's post says its support operation did not start out fully formed: over months of continuous iteration, "it grew one workflow at a time".
A&A perspective
A&A reads this as the brake on everything above. The order — build the sorting sheet, finish calibration, put the redelivery ceiling into the contract, then increase intake — can be right without requiring that all of it be built up front. The evaluation guide likewise conditions the reduction of human review to occasional use on the system having become robust; it does not say the order can be skipped. Touching the single line item that consumes the most time right now is enough to start. And neither Gumloop's figures nor Anthropic's design advice predict an outcome at A&A or at the reader's company.
A&A perspective
The next step is simple. For everything you deliver this week, record which of the three categories each item fell into, and accumulate two weeks of that. Then pick the single largest term and apply the two questions from the earlier section: do you put the reason for sending work back into words, and can you write that reason from the deliverable and the input material alone? If you can answer yes to both, that check can move. If you cannot, the definition of the product may still be incomplete. If you cannot make that distinction in-house, A&A's initial consultation can work through which check is setting your capacity. To see the whole picture from customer acquisition through delivery and retention first, "AI-native GTM: a practical guide for solo founders and small teams" places this decision in context.
The delivery capacity of an AI service is set not by generation speed but by the hours a person spends bringing output to the agreed quality. Split those hours into routine checks, exception handling and redelivery, measure them for two weeks, and start with the term that applies to every item. When you move a check to a machine, write the calibration cost into the estimate, because it arrives before the saving does. The decision to take on more work comes after that.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Building effective agents
Anthropic · 2024-12-19 (page notes tooling has changed since)
Accessed 2026-09-23 - Demystifying evals for AI agents
Anthropic · 2026-01-09
Accessed 2026-09-23 - Supporting the world's most AI-native companies with a 2-person team
Gumloop · 2026-02-17
Accessed 2026-09-23
AI-assisted editorial production
A&A uses AI for research, writing, translation and editorial checks. Source facts, our analysis and hypothetical examples are labeled separately.
Editorial check: 2026-09-23