AI tool or custom build? Compare three routes on the same done criteria
Choose an AI tool or a custom build by holding both to the same done criteria and counting review and upkeep.
Key points
- Set done criteria for inquiry summaries
- Compare 3 routes on the same criteria
- Count review and upkeep beyond AI fees
- Try a tool: keep what works, fill gaps
Contents
Set done criteria for the work you give AI first
A&A perspective
Before comparing an existing tool with custom development, pick one job you want AI to do and write its done criteria. In this article, done criteria are a few lines on who does the job, with what information, and what must be finished for it to count as done.
Hypothetical example
Take customer inquiries. Whether AI only writes a summary, or must also route each inquiry to the right person and log the response, changes which tools you compare. For summaries alone, the AI chat you already use may be enough. Logging means connecting to your customer records.
From the sources
The U.S. National Institute of Standards and Technology (NIST) lists, in its AI risk management framework, clearly defining the business value or context of use. It also lists specifying and documenting the intended scope, and testing before deployment and regularly during operation.
From the sources
Anthropic, which develops AI, names clear success criteria as one condition of tasks where systems that let AI carry out steps add the most value. Meaningful human oversight is among the same conditions.
A&A perspective
Automate & Augment (A&A), the company writing this article, thinks that comparing without done criteria tends to become a count of features. Whether a tool has AI in it does not tell you whether your job gets done.
A&A perspective
A&A cites the NIST framework because writing down the use and scope first is part of a public framework too. The same few lines also let you ask a tool vendor or contractor whether they can test against those conditions.
Compare 3 routes on the same done criteria
A&A perspective
Once the done criteria are written, hold three routes to the same conditions: use an existing tool as is, configure and connect it, or build the parts you need. These are not tiers but different ways to finish the same job. The table below lists where each route may meet the criteria and what to check first.
From the sources
Microsoft's suite of business software includes the AI assistant Copilot. Inside it, a feature called Agent Builder lets you create an AI for a set task by describing it in plain language. According to Microsoft's documentation, you set instructions and the material it draws on, and test it as you build.
From the sources
The same page says Agent Builder does not support authoring actions that connect to external services. For those, it directs you to copy the agent to Copilot Studio, a separate tool for assembling AI behavior from building blocks. If you need your own code, it points to a guide on choosing development tools.
From the sources
It also says the knowledge sources available depend on the user's license.
A&A perspective
A&A reads this as all three routes appearing inside a single product: configure, connect, build. Which one meets your done criteria has to be checked against your own contract and the material you actually use.
A&A perspective
In A&A's view, a company that does not use Microsoft can check the same way. In the documentation of the AI tools you use, find where configuration ends, where connecting outside services begins, and what must be built separately, then set that beside your done criteria.
| Route | Where it may meet the done criteria | Check first |
|---|---|---|
| Use an existing tool as is | Current features look able to finish the job | Usable material, features in your contract, manual work left over |
| Configure and connect | You need to connect internal documents or tools you already use | What can connect, permissions, steps for a day it stops |
| Build the needed parts | Your own approval order or a customer-facing screen is essential | Who maintains it, how to request changes, how to exit |
Beyond AI fees, count review and upkeep
A&A perspective
Do not compare cost on monthly fees or development price alone. Include the time people spend checking what AI produces, and who fixes things each time something changes.
From the sources
Anthropic writes that systems letting AI carry out steps often trade latency and cost for better task performance. It also writes that for many applications, a single AI call with retrieval and in-context examples is usually enough.
From the sources
The NIST framework lists taking viable non-AI alternatives into account when considering the resources required to manage AI risks.
A&A perspective
As that non-AI alternative, A&A recommends putting your current manual process in the comparison table too. Once you know the time it takes today, you can set it beside fees and development cost in the same table.
A&A perspective
A&A recommends adding to the table the time people spend checking, who updates things when internal rules change, and price changes. Even a cheap monthly tool costs the copying and checking that remain every day. For inquiries, note who changes the settings each time reply rules or staff assignments change.
A&A perspective
Changes and exit go in the same table: can you export your records, can a new person read the settings, and can you fall back to manual work on a day the connection stops? Even with custom development, some parts still rely on outside AI and connected services.
From the sources
The NIST framework lists contingency processes for failures in third-party data or AI systems deemed high-risk. It also lists including decommissioning and change management in post-deployment monitoring plans.
Try a tool first, then fill only the gaps
A&A perspective
If the comparison does not settle it, try an existing tool and send only the done criteria it failed to meet to integration or development. Keep using the tool for the parts it covered.
From the sources
Anthropic recommends finding the simplest solution possible and only increasing complexity when needed. It writes that this might mean not building agentic systems at all. It also says to add complexity only when it demonstrably improves outcomes.
From the sources
This Anthropic piece was published in December 2024, and the page itself notes that much of the tooling it describes has changed since.
A&A perspective
A&A reads it not as a product comparison but as a way to order your trials. Tools change, yet the order of starting simple and adding complexity only where it proves better still works when you test in your own company.
Hypothetical example
Say a company takes consultation requests through a web form and wants AI to summarize each request and draft a reply. First, test with real requests whether an existing tool meets the done criteria for the summary and the draft.
Hypothetical example
Suppose the test shows you need approval by each staff member and a way to avoid handling the same request twice. Only those parts become candidates for integration or development. Whether to rebuild the intake screen is a separate decision, made at that point.
A&A perspective
When you send work to development, whether to a contractor or built yourself, put the done criteria and the exit plan in the first request. A&A recommends using those same lines as your check when you accept what is delivered.
Summary
A&A perspective
Before comparing, pick one job for AI and write its done criteria.
A&A perspective
Compare an existing tool, configuration and integration, and custom development against those same done criteria.
A&A perspective
Count review effort, upkeep, changes and exit in the cost, not just fees.
A&A perspective
Try an existing tool, and send only the criteria it could not meet to development.
Whether to use an existing tool or build your own depends less on feature counts than on whether your job reaches its done criteria, and on the checking and upkeep that follow. What to decide next is which job, such as handling inquiries, you will write a few lines of done criteria for and test in the tool you already use.
FAQ
Where do no-code integration tools fit among the three routes?
In this article they belong under configuration and integration. Microsoft's documentation likewise directs you to a separate building-block tool when you need to connect services or add steps. What matters is not whether you write code but which part of the done criteria it fills.
If I can build it myself with Claude Code, is building from the start faster?
The faster you can build, the cheaper it is to test. Even so, the upkeep after building and someone to update it when internal rules change remain. A&A recommends testing the done criteria once in an existing tool, seeing what falls short, and then deciding what to build.
How can I check before signing whether an existing tool is enough?
In the trial, run the few lines of done criteria against your real material. Microsoft's documentation says the knowledge sources available depend on the user's license. At the same time, check whether the trial and the paid plan offer the same features.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Building effective agents
Anthropic · 2024-12-19 (date shown on page)
Accessed 2026-10-27 - Build agents in Agent Builder
Microsoft · Updated 2026-09-30
Accessed 2026-10-27 - AI RMF Core
NIST · 2023 (AI RMF 1.0)
Accessed 2026-10-27