Test AI PDF data extraction: 4 steps that keep source pages and blanks
Give AI only the copying and compare it field by field on your own files to decide if PDF extraction is worth it.
Key points
- PDF-to-ledger copying only; no judgment
- Keep each value's page and blank reason
- Compare fields: AI vs staff, safe docs
- Add check time; expand winners or stop
Contents
Give AI only the PDF-to-ledger copying
A&A perspective
If you test one thing first, give AI only the copying from PDFs into your ledger. PDF data extraction means pulling values such as part numbers and amounts out of incoming documents into a form you can copy into a table or ledger.
From the sources
Newfront, which runs an insurance brokerage in North America, is a company that handed this copying to AI. Anthropic, the company that develops AI, presents it as an example of a customer using its AI.
From the sources
According to the story, the company processes documents that summarize past insurance claims automatically with AI. The AI pulls out values even when the PDF layout differs from one document to the next. It also removed the re-entry people used to do between old and new systems.
From the sources
The company's co-founder says the insurance business needs both human expertise and technology. Routine tasks go to AI, and brokers focus on advice and on solving complex problems.
From the sources
The story says the cost of processing these documents fell by 60%, with higher accuracy.
A&A perspective
Automate & Augment (A&A), which writes this article, does not use the 60% figure for its own estimates. It comes from a customer story published by the AI's developer, not from an outside check. What is worth borrowing is not the number but the split between the work given to AI and the work people keep.
A&A perspective
AI's part ends at copying document values into the ledger. Expert judgment, such as whether to accept a discount or which product to recommend, stays outside the test. The more your suppliers' document layouts differ, the more a test is worth.
Keep each value's page and blank reason
A&A perspective
When AI extracts values, have it return, for each value, the page it came from and, if the field is blank, the reason. With the page, the person checking does not have to reread the document from the start.
From the sources
Anthropic offers developers who build its AI into their systems a feature that shows which sentences in the original document the AI used for an answer. It describes the mechanism as splitting PDFs into sentences, passing them to the AI, and attaching the location of the original sentence to answers taken from the document.
A&A perspective
When you test it yourself, first check whether your AI tool can show the source page for each value. If it cannot, instruct it to write the page number next to each value, and check that number too when you compare.
Hypothetical example
Say a parts trading company with 20 employees copies part numbers, quantities, unit prices and delivery dates from suppliers' quote PDFs into its sales ledger. This is a hypothetical example. If the unit price field is empty, do not let AI fill in zero; have it leave the field blank and give a reason.
Hypothetical example
If the unit price sits in a table on a later page of the quote, the AI returns that page number along with the price. The person checking opens only that page and compares the figures.
A&A perspective
Sort blank reasons into not written, unreadable and conflicting. If it is not written, ask the supplier; if it is unreadable, get the original document; if it conflicts, check which version is newer.
A&A perspective
The reasons are split because each one leads to different next work. Lumping them all together as "needs checking" makes the checker hunt for the cause again. If AI is not allowed to guess at blanks, numbers without a basis stay out of the ledger.
| Blank reason | Example in a quote | Next work |
|---|---|---|
| Not written | The unit price field is empty | Ask the supplier |
| Unreadable | The scan has blurred the numbers | Get the original document |
| Conflicting | One part number has two unit prices | Check which version is newer |
Compare fields one by one: AI vs staff on your own files
A&A perspective
Decide whether it is worth it by comparing AI extraction and staff copying field by field on your own non-confidential documents. An overall accuracy rate alone does not tell you which fields you can hand over.
From the sources
Newfront also tested on its own material when it chose an AI. It collected 20 customer questions about the details of insurance plans that earlier AI models had struggled to answer. Anthropic's AI answered 12 of the 20 correctly, the story says.
A&A perspective
What Newfront compared was answers to questions, not copying from PDFs. Even so, A&A thinks the same way of testing on your own material works for copying. The difference is that you compare by ledger field, not by question.
A&A perspective
Use only non-confidential documents for the test. A copy with supplier names and other sensitive parts masked keeps the real layout. Mix in revised versions, scanned paper and documents where the delivery date appears only in a note. Clean documents alone hide the places that take a long time to check.
A&A perspective
Put the results on one sheet, one row per ledger field. The columns are the field name, the AI's value and the staff member's value, the page the AI took the value from, the blank reason, and the minutes it took to check the AI's value.
A&A perspective
For rows where the AI's value and the staff value differ, open the original and also note which one was right.
A&A perspective
Side by side, the difference by field becomes visible. Part numbers and quantities may be quick to check, while a delivery date that appears only in a note may mean rereading the page every time.
Add check time, then expand only faster fields
A&A perspective
Decide whether to continue by whether any field is still faster than copying by hand once you add the time to check the AI's values. Expand only those fields, and keep copying the rest by hand.
From the sources
Anthropic recommends that systems using AI start with the simplest approach possible. It writes that complexity should be added only when it is shown to improve results.
Hypothetical example
Put hypothetical numbers on the parts trader from earlier. With 120 quotes a month and 8 minutes to copy each, copying takes 16 hours a month. If checking after the AI takes 3 minutes per quote, checking takes 6 hours a month.
Hypothetical example
With these hypothetical numbers, the difference is 10 hours a month. But looking field by field on the sheet, you may find that delivery dates written only in notes require rereading the page every time, so copying them by hand is faster. In that case, drop delivery dates and hand over only part numbers, quantities and unit prices.
A&A perspective
Freed-up time does not turn into money by itself. It becomes money only when faster quote replies let you take on more work, or when you cut outsourced data entry or overtime.
A&A perspective
Work out how many months it takes to pay back using your own numbers. Put in the upfront cost including the effort of testing, the AI's monthly fee, and the outsourcing or overtime cost you can cut each month. Divide the upfront cost by the monthly cut minus the monthly fee to get the months to pay back. If the cut is smaller than the fee, it does not pay back.
A&A perspective
If every field takes longer than copying by hand, hold off for now. Expand the scope only after you find faster fields.
Summary
A&A perspective
Give AI only the copying from PDFs into the ledger, and keep expert judgment out.
A&A perspective
Have it keep the source page and the blank reason for each extracted value.
A&A perspective
Compare AI and staff copying field by field on your own non-confidential documents.
A&A perspective
Expand only the fields that are still faster than hand copying once check time is added, and hold off if there are none.
Whether to keep using AI for PDF data extraction depends not on overall accuracy but on per-field time once checking is included. What you decide next is which non-confidential documents to test with and which fields to compare.
FAQ
If our documents all share one layout, is AI still worth testing?
If the layout is fixed, reading software that picks up text at set positions (OCR) may be enough. AI is most worth testing on documents whose layout differs by supplier.
Can we let AI read documents we receive from suppliers?
At the testing stage, use non-confidential documents or copies with sensitive parts masked. Before real use, check what your confidentiality agreements with suppliers say.
How many documents should we test with?
There is no set number. Newfront, which runs an insurance brokerage in North America, compared AI models on 20 questions that earlier AI models had struggled to answer when it chose one. More than the count, make sure the set covers each supplier's layout, revised versions and hard-to-read scans.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Newfront modernizes insurance experiences with Claude
Anthropic · Publication date not stated
Accessed 2026-10-24 - Introducing Citations on the Anthropic API
Anthropic · 2025-06-23
Accessed 2026-10-24 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-10-24