A&A INSIGHTS
When to switch models in a small AI SaaS: customer failures, not benchmarks
Decide a model change on representative customer tasks, not public benchmarks. Build the set from past failures, then write conditions for four exits before you upgrade.
日本語で読む
THE STARTING POINT
A model change in a small AI SaaS is decided not on public benchmark rankings but on whether the work you sell finishes more reliably on a set of representative tasks copied from real customer requests. Collect those tasks from past failures and the checks you already run by hand, write pass and fail as the state the customer receives, and put the observable conditions for four exits — adopt, adopt in part, hold, revert — in writing before the upgrade work begins.
Decide model changes on how many customer tasks still pass
A&A perspective
Whether you may switch to a new model is decided by one thing only: whether the work you sell finishes more reliably than it does now, measured on a set of representative tasks copied from the requests your customers actually send. Not by a public benchmark ranking, and not by how the new model feels when you try it yourself. There are three steps. First, collect representative tasks from past failures and from the behaviours you already check by hand before every release. Second, write pass and fail as a state the customer receives, not as a judgement about output quality. Third, write down the observable conditions for four exits — adopt, adopt in part, hold, revert — before you begin the upgrade work. That is this article's answer; the rest explains the basis for it and the four-row decision table you fill in before the upgrade. The position taken here is that these decisions stall not because information is missing, but because no definition of pass and fail is at hand.
From the sources
The Lovable customer record on Anthropic's site states that the company puts every new Claude release through the same evaluation it has run since the beginning. The wording is "Every new Claude release goes through the same evaluation Lovable has run from the start", and what it measures is "how often the system hits a wall and produces an app that’s broken or isn’t what the user asked for" — the rate at which the system gets stuck, and the rate at which the generated app is broken or differs from what was requested. The same page explains why that gate matters: "Lovable’s users often can’t read the code themselves, so they’re trusting the output to work." The users cannot read the code, so they use the output on trust.
A&A perspective
What a solo or two-person developer can take from this is not a task count or an organisational structure, but only the shape: fix the evaluation as a gate that every release passes through unchanged. Lovable's scale and operating conditions belong to Lovable and are not a forecast for anyone else. The condition that users trust output they cannot verify themselves, however, applies regardless of scale to any small company selling generated work. If the customer cannot check the quality, the only party who can is the seller, through a fixed evaluation. The page does not state the number of tasks, the pass criteria or how often the evaluation runs, so read the task counts and conditions in this article as A&A design proposals rather than as anything Lovable reported.
A better benchmark score does not mean the work you sell finishes
From the sources
Anthropic's guide to evaluations, published on 9 January 2026 by Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe, defines the unit of measurement plainly: "An evaluation suite is a collection of tasks designed to measure specific capabilities or behaviors", and adds that "Tasks in a suite typically share a broad goal". The example given is that a customer support eval suite might test "refunds, cancellations, and escalations". The unit of evaluation, in other words, is not the general merit of a model but a collection of tasks chosen to measure specific capabilities or behaviours.
A&A perspective
Carry that definition into a model-change decision and the comparison becomes clear. A public benchmark is a collection of tasks somebody else selected, and the broad goal they share is not the work you sell. Your own representative task set is a collection built from the requests your customers actually send, and its broad goal is that the work you sell finishes. These are two different instruments, so a rise in the first guarantees nothing about the second. The converse also holds: if the second has not fallen, you may switch even when the first looks unimpressive. This is A&A's reading; the source does not discuss model-change decisions as such.
From the sources
What Lovable measured was likewise not a public benchmark but mismatch on its own tasks. The same page records specific releases as turning points in what users could build, quoting product lead Alexandre Pesant: "Claude Sonnet 3.5 was the first model that made agents work", and Claude Opus 4.5 was "the next big step change in reliability on long-horizon tasks". This is an account of what the company observed in its own product, however, and is not presented as the result of measuring general model performance.
| Exit | What you observe on the representative tasks | What you then do |
|---|---|---|
| Adopt | Not one previously passing item fails, and some previously failing items now pass | Switch all customers. Keep the same representative set fixed until the next upgrade |
| Adopt in part | Some steps improve while items that used to pass now fail on another step | Split models by processing step. Leave the failing step on the old model |
| Hold | The pass count is unchanged and only cost or elapsed time has moved | Do not switch. Write down whether you review at your next release or the next model release |
| Revert | Previously passing items fail, or output no longer matches the customer's own records | Return to the old model. Add the failing items to the representative set and fix it again |
Build the representative task set out of past failures
From the sources
The same guide addresses how large an initial set needs to be. "We see teams delay building evals because they think they need hundreds of tasks", it observes, when in fact "20-50 simple tasks drawn from real failures is a great start". The reason given is that in early agent development each change to the system has a clearly visible impact, and "this large effect size means small sample sizes suffice". It also qualifies this directly: "More mature agents may need larger, more difficult evals to detect smaller effects", so a more mature system needs a larger and harder evaluation.
From the sources
The guide also specifies where the material comes from. "Begin with the manual checks you run during development" — the behaviours you verify before each release and the common tasks end users try. If you are already in production, "look at your bug tracker and support queue". Then "Converting user-reported failures into test cases" is what makes the suite reflect actual usage. On the cost of delay it is equally direct: "Evals get harder to build the longer you wait", and wait too long and you are "reverse-engineering success criteria from a live system".
A&A perspective
A one-person company frequently has no tidy bug tracker or support queue. The equivalents in that case are the text of inbound enquiries, the record of requests you redid, the corrections raised at acceptance, and the discounts or unbilled work you used to settle a complaint. These are only untidy in form; as records of real failures they are sufficient material. A&A's proposal is to draw representative tasks from those four sources, include at least one per customer, and always include any pattern that previously produced a complaint or rework. Do not import the source's 20-50 as a target; decide the count from how many distinct patterns your requests fall into. If there are only five patterns, the total follows from how many cases you want per pattern.
Hypothetical example
A hypothetical example shows the shape. Suppose a three-person SaaS turns meeting recordings into a summary of decisions and a list of action items. The representative set holds 30 tasks. Twelve are patterns that previously drew complaints: recordings where two people talk at once, recordings dense with in-house abbreviations, recordings where the agenda changes partway through. The remaining 18 are the ordinary patterns already checked by hand before each release. Those 30 are fixed, and every model change runs the same 30. When a new complaint arrives, that recording becomes the 31st and the set is fixed at 31 from then on. This is an illustration of the shape, not a record of any real customer.
Write pass and fail as the state the customer receives
A&A perspective
Collecting representative tasks does not help if pass and fail are written as "is the output good". Good and bad shift from person to person, so every upgrade returns the argument to the start. A&A's proposal is to write pass and fail as the state the customer receives. For a meeting summary, not "nothing decided was missed" but "of the decisions that appear in the recording, none is absent from the list". For an action list, not "an owner is written down" but "the owner's name matches the speaker who accepted that work in the recording". Both are forms in which a different person, listening to the same recording, reaches the same verdict. Items whose verdict varies by person belong in a separate reference column, not in pass and fail. The adjacent question of how to confirm a single completion in a workflow you have already delivered is covered in the existing article "How to verify the business result after AI says it is done".
From the sources
The source points the same way. On what makes a good task, it says "A good task is one where two domain experts would independently reach the same pass/fail verdict", and notes that ambiguity in task specifications becomes noise in metrics. Where a model does the grading, it recommends avoiding invented verdicts by giving the grader an escape: "give the LLM a way out", that is, an instruction to return Unknown when there is not enough information.
Hypothetical example
For the hypothetical SaaS above, three lines are enough to show how pass and fail are written. Line one, coverage of decisions: every point in the recording where agreement was reached — "let us go with that", and similar — corresponds to at least one entry in the decisions column of the summary. Line two, owner match: each owner name in the action list matches the speaker who accepted that work in the recording. Line three, nothing invented: the summary and the list contain no proper noun, date or figure that has no basis in the recording. All three can be judged identically by a different person who listens to the recording. This is an illustration, not any real customer's criteria.
Write the four exits — adopt, adopt in part, hold, revert — before the upgrade
A&A perspective
Before the upgrade work begins, write down four exits: adopt, adopt in part, hold, and revert. Of these, the one small teams most often lack is hold, and without a written condition for it the state of "not decided yet" continues indefinitely. Hold is the exit for the case where the number of passing representative tasks is unchanged and only cost or elapsed time has moved. Decide not to switch and to carry the question to the next upgrade, and you get that week back for other work. A hold should always name not a deadline but the next occasion for review: either your next release or the next model release.
From the sources
Judging on cost and elapsed time is possible only because the evaluation already exists. The source states that "you get baselines and regression tests for free", and lists what can be tracked on a static bank of tasks: "latency, token usage, cost per task, and error rates". The same guide argues that evaluations are useful even in early development for encoding expected behaviour explicitly, noting that "Two engineers reading the same initial spec could come away with different interpretations" on how the AI should handle edge cases, and that "An eval suite resolves this ambiguity".
A&A perspective
There is only one rule for reading the table: moving left to right, the action for each observation must be uniquely determined. If the pass count has not fallen and the discussion nonetheless continues, the table was written too loosely. If an observation matches two or more of the four exits, the representative tasks are divided too coarsely; split them by processing step and run them again. The table and its four categories are an A&A design proposal, not something the sources state.
Make reverting possible before you try the new model
A&A perspective
The revert exit is unusable if you only write its condition. Put yourself in a state where reverting actually works, then try the new model. Three things are the minimum: keep the path that calls the old model rather than deleting it, make the switch a single configuration change, and record what was switched and when. Without the third, a complaint from a customer cannot be separated into "the model changed" and "something else changed". Keep the switch log in a form you can line up against per-customer enquiry records on the same time axis.
From the sources
The Lovable record also describes choosing models per step. In Pesant's words, "We have a harness around a main agent that can use subagents to orchestrate tasks effectively, with the right models at each step": a main agent reasons about the build, hands smaller pieces of work to subagents, and matches each task to the model best suited to it.
A&A perspective
Because that structure exists, adopt-in-part is a coherent exit. Applying it to a small business, though, is A&A's hypothesis. The fewer steps you have, the easier they are to separate; but once different steps run different models, the combinations you must verify grow with the number of steps. If your representative tasks are not already divided by processing step, we think there is less rework in choosing adopt or hold rather than adopt-in-part. Whether to split by step is also a question of how much human checking time you have. That estimate is the subject of the existing article "Before taking on more AI delivery work, estimate where human review jams".
With evaluations, the source says, upgrading takes days rather than weeks
From the sources
The source is explicit that having evaluations determines how fast you can adopt a model. When more powerful models arrive, "teams without evals face weeks of testing", while teams that have them can quickly determine the model's strengths and "tune their prompts, and upgrade in days". Evaluations, it argues, are useful not only for tracking regressions but for accelerating development.
A&A perspective
In a company of one or three people, those weeks are the founder's own weeks, during which both selling and delivery stop. So we treat the effort of building a representative task set not as a quality-control cost but as the cost of taking the upgrade decision back into your own hands. To the opposing position — new models are generally better, so switching quickly beats spending effort on evaluations — we agree conditionally. Before existing customers use the product daily, that is right. Once they do use it daily, we think the explanation and rework that follow a complaint after switching tend to exceed what the faster switch gained.
From the sources
The cost of postponing accumulates. The source writes that "Evals get harder to build the longer you wait": early on, product requirements translate naturally into test cases, whereas wait too long and you are "reverse-engineering success criteria from a live system" — working the success criteria backwards out of a system that is already running.
Where this way of deciding does not hold
A&A perspective
Three limits. First, the Lovable record is a customer story on Anthropic's own site, not an independent audit. Figures for revenue and cumulative projects appear at the top of that page, but they are the company's published statements about itself and are not a forecast for any other team, so this article does not use them. The connection that a single person can also make one fixed evaluation the gate is A&A's reading, separate from how that company is organised.
A&A perspective
Second, Anthropic's guide to evaluations is a technical document about assessing the quality of AI agents, not about how to make the commercial decision to change models. The figure of 20-50 is written as a starting point for early agent evaluations, not as a standard for every scale and domain. The four categories — adopt, adopt in part, hold, revert — and the decision table are A&A design proposals rather than statements from the sources.
A&A perspective
Third, passing the representative tasks does not substitute for production. Paths that fail in production remain after the evaluation passes, and an item that fails may turn out not to matter. This article states nothing about A&A's win rates, closing rates, customer results or search rankings. The next step is to count the patterns of output you currently ship and write down how many of them previously drew a complaint. If you want to see where this decision sits between winning customers and keeping them, "AI-native GTM: a practical guide for solo founders and small teams" is the overall map. If the definition of pass and fail shifts from customer to customer and you cannot settle it alone, advisory conversation suits that; if the open question is how to implement the representative task set and the switching path, scoped development will finish sooner.
Whether to change models is settled not by novelty and not by how the new model feels, but by whether the work you sell finishes more reliably on a set of representative tasks copied from the requests your customers actually send. Build that set from past failures and the checks you already run by hand, write pass and fail so that a different person reaches the same verdict, and put the observable conditions for adopt, adopt in part, hold and revert in writing before the upgrade. Prepare the path back to the old model, and the log of what you switched, before you try anything. The four exits and the decision table here are A&A design proposals; the sources do not discuss model-change decisions as such.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Lovable helps anyone create software 20x faster with Claude
Anthropic · n.d.
Accessed 2026-09-25 - Demystifying Evals for AI Agents
Anthropic · 2026-01-09
Accessed 2026-09-25
AI-assisted editorial production
A&A uses AI for research, writing, translation and editorial checks. Source facts, our analysis and hypothetical examples are labeled separately.
Editorial check: 2026-09-25