AI specification changes: choose which workflows to review
Use a map of models, services and review questions to select workflows to check after AI specification changes.
Key points
- Link workflows and review questions
- Keep unknown workflows in the review set
- Separate testing from business judgment
Overview diagram in preparation
Contents
Link review questions to workflows and dependencies
A&A perspective
A review question asks whether a workflow delegated to AI still behaves as intended. To prepare for specification changes, record the workflow, its model and connected services alongside the question, rather than keeping a question list alone.
Hypothetical example
Suppose a small creative studio uses AI to draft inquiry replies, summarize meetings and extract invoice details. A notice that a model will be retired arrives. A list containing only workflow names cannot tell the owner which ones need attention.
From the sources
Anthropic has published a customer case study about Spellbook, a service for drafting and reviewing contracts. It describes a feature that indexes contracts together with the recurring questions asked about them, allowing teams to find contracts to reassess when the law changes.
A&A perspective
Automate & Augment (A&A), the organization writing this article, thinks linking review targets to questions can also inform checks on AI used in internal operations. However, a product feature for finding contracts is not the same task as investigating effects on business workflows.
A&A perspective
In your own table, record the workflow, model or connected service, its version and the review question. An API is an interface for exchanging data with an external service. For workflows using APIs, naming the service and function provides a starting point for matching change notices.
Hypothetical example
For a hypothetical reply workflow, a review question might be: “Does the reply promise a delivery date when none has been agreed?” Place an example input and a pass condition beside it, such as: “If the supplied material has no delivery date, the draft says that confirmation is needed.”
A&A perspective
Saving only the previous answer can lose the purpose of the check. Keeping the question and pass condition lets the team check the same business requirement under the same conditions even when the wording of the answer changes.
Keep matching and unknown workflows in the review set
A&A perspective
When a change notice arrives, match the affected model or connected service against the list and put matching workflows in the review set. Keep workflows with unknown dependencies in that set for investigation as well.
Hypothetical example
In the studio example, suppose reply drafting and meeting summaries use the same model, while invoice extraction uses another. If the shared model is being retired, both replies and summaries need review even though their questions differ.
A&A perspective
Searching only the wording of questions can miss this shared dependency. A reply workflow cannot be excluded simply because its delivery-date question contains none of the terms in the notice. First check how the affected dependency maps to the workflow.
A&A perspective
Nor should invoice extraction be declared unaffected just because it uses a different model. Inspect its settings or ask its builder whether it uses the retiring service somewhere along the way. We recommend explicitly marking unknown entries as unconfirmed rather than leaving blanks.
Hypothetical example
The table below is a fictional classification. Inclusion in the review set means deciding what to check, not discovering a fault. If a workflow is excluded, retain the evidence establishing that it runs on a different dependency.
A&A perspective
This list alone cannot detect changes absent from the notice or connected services absent from the table. Even passing every recorded question does not prove that no effects exist outside those checks.
A&A perspective
Maintaining the list whenever a workflow changes still takes effort. If there is only one workflow and its whole operation is easy to check, reviewing it in full may be simpler than building a table to select candidates.
| Workflow situation | Review treatment | First check |
|---|---|---|
| Reply drafts use the retiring model | Include in review | Does the replacement avoid promising delivery dates? |
| Meeting summaries use the same model | Include in review | Are owners and deadlines kept correct? |
| Invoice extraction has unknown dependencies | Keep and investigate | Does any intermediate step use the retiring service? |
Separate running checks from judging business outcomes
A&A perspective
When commissioning a review, distinguish the person who reruns the questions from the person who decides whether the answers are fit for business use. One person may fill both roles, but record execution and acceptance as separate results.
From the sources
In its case study of insurance broker Newfront, Anthropic describes a set of 20 important customer questions that previous AI models had struggled to answer. Newfront used them in model selection: an example of preparing questions before making a comparison.
A&A perspective
This case does not demonstrate selecting workflows affected by specification changes through questions. A&A proposes reusing saved questions and example inputs in checks after a change. The source is an account published by the provider, and the same results are not assured in your business.
Hypothetical example
At the fictional studio, the contractor generates new reply drafts with a replacement model, while an internal sales colleague checks how delivery dates are handled. The contractor returns the results and questions that could not be tested. The internal colleague checks the answers against the pass conditions.
A&A perspective
A review request includes the subject and deadline of the notice, candidate workflows, questions and example inputs, and the person receiving the results. Where a dependency is unconfirmed, identify the settings investigation as work that must happen before rerunning questions.
A&A perspective
A change originating outside the company is not necessarily included in the contractor’s maintenance fee. Check the existing agreement to establish what it covers for retirement handling, review and repairs. Who caused a change does not, by itself, determine who pays.
A&A perspective
The number of questions helps describe the workload, but does not represent the full cost. Account separately for preparing inputs, investigating dependencies, human judgment and repairs after failed checks. A&A has not measured time or cost savings from this proposed approach.
A&A perspective
Once review candidates have been selected, “Re-acceptance after AI updates: what your vendor tests, what you check” addresses which changes require acceptance checks inside your company. The list in this article gathers the candidates to feed into that decision.
Summary
A&A perspective
Linking workflows, models or connected services, and review questions provides a starting point for matching change notices to your own operations.
A&A perspective
Keep workflows with unconfirmed dependencies in the review set alongside those that match the notice.
A&A perspective
Separate rerunning questions from judging business acceptance, and check both the maintenance scope you can commission and the effort that remains.
Linking workflows, models or connected services, and review questions provides a starting point for matching change notices to your own operations. Keep workflows with unconfirmed dependencies in the review set alongside those that match the notice.
FAQ
Can a question list replace an inventory of the models we use?
You need both. Workflows with different questions may be affected by a change to a shared model or service. Keep rows with an unknown workflow-to-dependency mapping in the review set until they are checked.
Is the review complete once all previous questions pass?
That establishes a pass only within the scope of those questions. New changes can introduce problems that earlier questions do not cover. The responsible person compares the notice with the questions to identify gaps, and records any scope that could not be tested.
Can AI read change notices and select workflows for review?
There is room to use it for a draft selection. However, missing dependencies and incorrect mappings can remain. A person should check exclusion reasons and unknown workflows; completing candidate selection should not be treated as completing the impact investigation.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Spellbook runs 530,000 contract reviews a month with Claude
Anthropic · Publication date not stated
Accessed 2026-10-29 - Newfront modernizes insurance experiences with Claude
Anthropic · Publication date not stated
Accessed 2026-10-29