When research reports disagree: have AI keep the grounds, not an average
When research reports disagree, have AI keep each report's grounds and open conditions instead of averaging them.
Key points
- Feed each source to AI separately
- Have AI table each split and its cause
- List open conditions; add your numbers
- Have a person check the cited passages
Contents
Feed each source to AI separately
A&A perspective
Give AI one source per request, and each source's claims and the passages that support them stay separate in your hands. By an averaged answer we mean one that erases the differences between sources and folds them into a single line such as "effective depending on the use case."
Hypothetical example
Say you want to bring AI into customer inquiries, and you have gathered a comparison guide, an industry survey and a proposal from an outside developer. Hand all three to AI at once and ask for a summary, and you may get an answer that fits each of them a little.
A&A perspective
Even if that answer reads well, you cannot tell which part of which document it rests on, so it cannot settle the decision.
From the sources
In December 2024 Anthropic published an article on designing systems that put AI to work, titled "Building effective agents." It says that for complex tasks with several considerations, AI generally does better when each consideration is handled by a separate call.
From the sources
Quillit, a North American tool for analyzing market-research interviews, has a case study. It says that in interview research, critical insights risk getting buried in transcripts and notes.
From the sources
One of the jobs Claude takes on in this tool is creating precise references to the relevant parts of transcripts and video recordings.
A&A perspective
Automate & Augment (A&A), the company writing this article, uses both ideas when comparing documents: splitting the work into separate requests, and having AI produce references that point to the exact passage. We give AI one document per request and have it write down three things: the claims, the page or section behind each claim, and the premises.
A&A perspective
The premises record who the document is about, when it was written and who wrote it. With one document per request, when the conclusions split you can trace the difference back to the exact place in each document.
Have AI table each split and its cause
A&A perspective
Once you have the notes for all three documents, use a separate request to have AI line them up side by side and sort the places where the conclusions agree from the places where they split. For each split, have it state whether the difference lies in the subject, the date or the author's position.
From the sources
In June 2025 Anthropic published how it built a feature that has AI do research, in an article titled "How we built our multi-agent research system." The article lists the criteria used to grade the finished answers.
From the sources
The criteria include whether the claims match the sources, whether the cited sources match the claims, and whether primary sources were used over lower-quality secondary ones.
From the sources
The article also describes a bias that people found while testing the system. The early system kept choosing content built to rank high in search over authoritative but lower-ranked sources such as academic papers and personal blogs. Adding source-quality heuristics to the prompts is described as having fixed this.
A&A perspective
A primary source is one published by the people who did the research or the party involved. A&A reads this as a reason to look at who wrote a document, and for what, before looking at its conclusion. A comparison guide from a seller can be built on the premise that its own product wins.
A&A perspective
Give the author's position its own column, and you can see whether a split comes from a difference in substance or a difference in who wrote the document.
Hypothetical example
As a hypothetical, say a company of six people is deciding, from three documents, whether to start handling inquiries with an off-the-shelf AI service or to have an outside developer build one for them. The documents and numbers are invented. The table of the three documents appears at the end of this section.
A&A perspective
The table shows that A is about companies with many inquiries, and B is a proposal from the side that would build the system. C surveyed 300 small and mid-sized companies in 2024, is the oldest of the three, and draws its line by inquiry volume. Rather than one of them being wrong, the conclusions split because the premises differ.
| Document | Conclusion | Subject / date / author's position |
|---|---|---|
| A: comparison guide | An off-the-shelf service is enough | Companies with 500+ inquiries a month / 2025 / seller of an off-the-shelf service |
| B: proposal | Build a custom system | This company / 2026 / outside developer that would build it |
| C: industry survey | Off-the-shelf is enough under 200 a month | 300 small and mid-sized companies / 2024 / industry association |
List open conditions and fill them with your own numbers
A&A perspective
Once the table is ready, use one more separate request to have AI list the conditions that would settle the conclusion. The one who answers those conditions is not AI but the owner, who knows the company's numbers.
From the sources
The December 2024 Anthropic article also describes another pattern. One call generates an answer, and another evaluates it and returns what to fix. For complex search tasks, it says the evaluator decides whether further searches are warranted.
A&A perspective
A&A uses that evaluating request to list what is still unknown. AI knows only what the documents say. Your inquiry volume and how many wrong answers you can tolerate are not in any of them.
Hypothetical example
The request can read, for example: "List the conditions that could change the conclusions in this table and that none of the documents answer. For each condition, note which row of which document it affects." With the affected rows noted, you know where to return in the table once the conditions are filled in.
Hypothetical example
In the example, AI might list three conditions: the monthly inquiry volume, whether wrong answers about prices or delivery dates are acceptable, and who updates the information the answers draw on.
Hypothetical example
Say the owner fills them in as "80 a month," "no wrong prices" and "I update it myself."
Hypothetical example
Back in the table, A's subject does not match the company. The company falls under C's line of fewer than 200 inquiries a month, so it lands on the side where an off-the-shelf service is enough.
Hypothetical example
What remains is one point: whether the off-the-shelf service can keep it from quoting wrong prices. That can be checked by asking the seller before signing. The third answer, "I update it myself," stays the owner's work whichever option is chosen, so it does not decide anything.
Before deciding, have a person check the cited passages
A&A perspective
Even when the table and the filled-in conditions point to an answer, before deciding, open the originals at the passages AI cited, only for the rows that decide the conclusion. There is no need to reread everything.
From the sources
Quillit's product manager put numbers on citation accuracy: around 60 to 70 percent before using Claude 3.5 Sonnet, and between 89 and 98 percent after. The same quote calls that level of precision crucial for clients' confidence in the insights they receive.
A&A perspective
These are Quillit's own figures, from analyzing interview transcripts. A&A thinks the fact that the figure stays below 100 percent applies to comparing documents too. A passage AI cites may not match the claim.
From the sources
Anthropic's research feature adds a dedicated role that, after the research is done, identifies the specific locations for citations. The article says this ensures that all claims are properly attributed to their sources.
A&A perspective
The owner only needs to check the rows where the conclusions split and the rows that decided the matter once the conditions were filled in. In the example, that is two places: A's "500 or more a month" and C's "fewer than 200 a month." If the originals say so, you can decide from that table.
A&A perspective
If you cannot find the passage in the original, or the number differs, drop that row from the table. Then check whether the conclusions still split among the remaining documents.
A&A perspective
Checking can only confirm that a document says what AI reports. If the content of the document is itself wrong, opening the original does not remove the error.
Summary
A&A perspective
Give AI one document per request and have it write down the claims, the passages behind them and the premises.
A&A perspective
Where conclusions split, have AI table them and state whether the subject, the date or the author's position differs.
A&A perspective
Have AI list the open conditions that could change the conclusion, and answer them with your own numbers.
A&A perspective
Before deciding, have a person check the passages AI cited in the originals, only for the rows that decide the conclusion.
When documents disagree, having AI lay out each source's grounds and premises leaves you something to decide on, where an average does not. What you decide next is which of your own numbers go into the open conditions AI lists, and which rows you check in the originals.
FAQ
Isn't it enough to ask ChatGPT or Claude to compare all three documents at once?
It is when there are few documents and the decision is light. When the decision is heavy and the conclusions split, having AI write up each source separately lets you trace a difference back to the exact place in each document.
Should I leave out documents written by an AI service's seller or an outside developer?
Keep them, and record who wrote them in the author's-position column of your table. Documents from sellers can still contain facts that apply to your company.
I don't have time to check the originals.
Check only the rows that decide the conclusion: the rows where the conclusions split and the rows that settled the matter once the conditions were filled in. In the hypothetical example with three documents, that was two places, which takes far less time than rereading everything.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Building effective agents
Anthropic · 2024-12-19
Accessed 2026-10-24 - Quillit eliminates 80% of the time-consuming tasks of qualitative research using Claude
Anthropic · Publication date not stated
Accessed 2026-10-24 - How we built our multi-agent research system
Anthropic · 2025-06-13
Accessed 2026-10-24