Latest / AI Builds It: Easy Coding Tools / 7 OCR Quality Tests to Run Before Deploying a Document-Reading AI Agent

Transcript

7 OCR Quality Tests to Run Before Deploying a Document-Reading AI Agent · AI Builds It: Easy Coding Tools

• 0:00
  SPEAKER_00: Judge the reader by its worst page.
• 0:02
  SPEAKER_00: A document reading agent should be judged by the worst page it will ever be handed.
• 0:07
  SPEAKER_00: Most get judged by an average instead.
• 0:10
  SPEAKER_00: More often than not, that average comes from a tiny sample the vendor picked, which looks very little like the crumpled invoices and claims the agent will meet after launch.
• 0:20
  SPEAKER_00: Do the sum for a dense page of 2,000 characters at 99% character accuracy.
• 0:26
  SPEAKER_00: About 20 of those characters will be wrong.
• 0:30
  SPEAKER_00: And nothing in the figure says which ones.
• 0:33
  SPEAKER_00: For an agent, that's the whole question.
• 0:35
  SPEAKER_00: Even where character accuracy looks respectable, the words that matter usually score lower.
• 0:41
  SPEAKER_00: An assessment of the British Library's 19th century newspaper archive found characters 83.6% right and whole words 78%, while the meaningful words people actually search for fell to 68.4%.
• 0:57
  SPEAKER_00: OCR errors don't land evenly either.
• 1:00
  SPEAKER_00: When researchers measured how OCR noise in digitized newspapers hurt the software that later analyzed the text, they found a one-character slip could split a sentence in two, even in text that was mostly correct.
• 1:14
  SPEAKER_00: Placenames were missed more often than people's names, with dates somewhere in between.
• 1:20
  SPEAKER_00: Newspapers aren't invoices.
• 1:22
  SPEAKER_00: Still, dates faring worse than names should worry anyone automating accounts payable, where a date and an amount are exactly what an agent acts on without a second look.
• 1:35
  SPEAKER_00: Seven tests can show where a given reader's errors fall before the agent goes live, ranked here by what skipping each one is likely to cost.
• 1:45
  SPEAKER_00: Score only the fields an agent will act on.
• 1:48
  SPEAKER_00: Most of the money rides on this first test, which scores the reader only on the fields the agent will use.
• 1:56
  SPEAKER_00: Start with 200 documents from the inbox the agent will actually work.
• 2:01
  SPEAKER_00: Each one gets its totals, dates, account numbers, and names marked by hand.
• 2:06
  SPEAKER_00: Then count exact matches per field.
• 2:09
  SPEAKER_00: A total that's off by one decimal place gets no partial credit, since the agent will pay it in full.
• 2:16
  SPEAKER_00: Agree the pass mark for each field before the results come in.
• 2:21
  SPEAKER_00: Hasmarks chosen after the fact have a suspicious habit of landing just under whatever the reader scored.
• 2:28
  SPEAKER_00: Somewhere in the results, there's often a field nobody asked about, a purchase order number or a tax ID, scoring far below the headline figure, the vendor quoted.
• 2:40
  SPEAKER_00: An agent paying suppliers inherits the bank details score, which might be 91% on a reader sold as 99%.
• 2:49
  SPEAKER_00: Business cards show the problem in miniature.
• 2:52
  SPEAKER_00: A card might hold a name, a title, a company, three phone numbers and an email address in about 40 words, and nearly every one of those words belongs in a field.
• 3:04
  SPEAKER_00: Plenty of designers print the mobile number in pale gray, too.
• 3:09
  SPEAKER_00: Even when a reader transcribes the whole card without a single error.
• 3:14
  SPEAKER_00: It can still file the fax number as the mobile.
• 3:17
  SPEAKER_00: On a character score, that's a perfect read of a useless contact.
• 3:22
  SPEAKER_00: Shop for an OCR SDK for business card scanning, and Apris will walk you through which documents arrive and in what volume, before it asks what the reading is supposed to produce.
• 3:34
  SPEAKER_00: That last question seems to be the one agent projects skip most often.
• 3:40
  SPEAKER_00: In agent work, the answer fits on a sticky note, usually total, due date, payee, and bank details.
• 3:47
  SPEAKER_00: The first test scores those fields and nothing else.
• 3:51
  SPEAKER_00: Two people should mark each document without seeing each other's work.
• 3:55
  SPEAKER_00: Ideally, one of them comes from accounts payable, and the other has never seen these invoices before.
• 4:02
  SPEAKER_00: Before any scoring starts, they settle every disagreement.
• 4:06
  SPEAKER_00: Y2.
• 4:08
  SPEAKER_00: One tired marker who misreads a smudged 7 turns the answer key into one more unreliable reading.
• 4:15
  SPEAKER_00: If one supplier sends 40% of the invoices, roughly 40% of the test set should come from that supplier.
• 4:23
  SPEAKER_00: After that, freeze the set and leave it alone.
• 4:27
  SPEAKER_00: So next year's release from the same vendor faces this year's pages.
• 4:33
  SPEAKER_00: Find out whether the reader can admit doubt.
• 4:36
  SPEAKER_00: Recognition engines usually know when they're unsure of a word.
• 4:41
  SPEAKER_00: And many attach a confidence value to each word they read.
• 4:45
  SPEAKER_00: That number tends to get lost on the way to the agent, which ends up with a clean string of text and every hesitation sanded off.
• 4:53
  SPEAKER_00: Ask the vendor whether the agent ever sees it.
• 4:56
  SPEAKER_00: Next, damage a few documents on purpose.
• 4:59
  SPEAKER_00: Put a coffee ring over one total and fold another page straight through an account number.
• 5:06
  SPEAKER_00: Then watch what the agent does with each.
• 5:08
  SPEAKER_00: If the agent stops or asks somebody, it passes.
• 5:13
  SPEAKER_00: One that carries on with a confident guess at the coffee stain total has just shown you what production is likely to look like.
• 5:20
  SPEAKER_00: Set the review threshold field by field rather than page by page.
• 5:25
  SPEAKER_00: Nobody gets hurt if a smudged delivery address waits for tomorrow's batch.
• 5:30
  SPEAKER_00: Before an account number in the same state reaches a payment run, a person should look at it.
• 5:36
  SPEAKER_00: This test comes with a catch.
• 5:38
  SPEAKER_00: Flag a third of the pages for review and watch what the reviewers are doing by the end of the month.
• 5:44
  SPEAKER_00: Many will be clicking approve after a glance.
• 5:47
  SPEAKER_00: In hospitals, clinicians see so many electronic warnings that they end up overriding the vast majority, critical ones included.
• 5:57
  SPEAKER_00: The patient safety research on alert fatigue has been tracking that habit for years.
• 6:04
  SPEAKER_00: When scoring this test, put the good pages flag by mistake right next to the damaged pages the agent caught.
• 6:12
  SPEAKER_00: A reviewer who hasn't corrected a single field in a month has probably stopped reading.
• 6:18
  SPEAKER_00: Hidden text aimed at the agent.
• 6:20
  SPEAKER_00: Nikkei went looking last year and found hidden prompts in 17 research papers, set in white text or in type too small to read.
• 6:30
  SPEAKER_00: The prompts told any AI reviewer to praise the work.
• 6:33
  SPEAKER_00: Anyone reading those pages normally saw nothing unusual, which was the point.
• 6:39
  SPEAKER_00: On an invoice, the same trick could move money.
• 6:42
  SPEAKER_00: Security guidance files, the trick under indirect prompt injection, and counts any file an AI system reads.
• 6:51
  SPEAKER_00: A resume as much as an invoice, as a possible way in for instructions nobody would notice.
• 6:58
  SPEAKER_00: Those lines only reach the agent if the reader picks them up, which makes hidden text an OCR problem as much as a security one.
• 7:07
  SPEAKER_00: As tests go, this one is cheap.
• 7:10
  SPEAKER_00: Slip a line of near-white text onto a sample invoice telling the agent to update the payee's bank details.
• 7:18
  SPEAKER_00: If the agent obeys, the pilot stops there.
• 7:21
  SPEAKER_00: Bank details are the obvious target.
• 7:24
  SPEAKER_00: In the FBI's 2025 complaint data, Business Email Compromise, the scam where a hijacked you faked email gets money wired to the wrong account, accounted for about$3 billion in reported losses.
• 7:38
  SPEAKER_00: Any change to a supplier's bank account, however official the document looks, deserves a phone call to the number already on file.
• 7:47
  SPEAKER_00: PDFs add a second trap.
• 7:49
  SPEAKER_00: Underneath the printed page, there can be a text layer that says something else entirely.
• 7:55
  SPEAKER_00: Some readers trust that layer and never look at the picture at all.
• 8:00
  SPEAKER_00: It takes about an afternoon to compare each test file's hidden layer with a fresh read of the page image.
• 8:06
  SPEAKER_00: With the files open anyway, invisible text, white-on-white or two-point type, can be stripped or flagged before the agent ever sees it.
• 8:17
  SPEAKER_00: Words in the wrong order.
• 8:19
  SPEAKER_00: Every word on a two-column contract can come out right while the columns get spliced together line by line into clauses no lawyer ever wrote.
• 8:29
  SPEAKER_00: Tables are worse.
• 8:30
  SPEAKER_00: On a long invoice, the line items often wrap onto a second row.
• 8:35
  SPEAKER_00: And the grand total may not turn up until page 3.
• 8:39
  SPEAKER_00: The agent has to work out which number is which from text that may not have arrived in reading order.
• 8:46
  SPEAKER_00: Margins cause trouble of their own.
• 8:48
  SPEAKER_00: Say someone strikes out clause 7 and writes a new figure beside it in pen.
• 8:53
  SPEAKER_00: A reader that drops the handwriting hands the agent the old clause as if nothing had changed.
• 8:59
  SPEAKER_00: Worse still, is a paid stamp the reader misses, which can get the same invoice paid twice.
• 9:05
  SPEAKER_00: Very little of this shows up on the leaderboards that rank AI models.
• 9:10
  SPEAKER_00: Ten of your own multi-page invoices, read back in order, will tell you more in a few minutes.
• 9:16
  SPEAKER_00: With page numbers and positions attached to each value, a reviewer checking a flag total can go straight to the right spot in a 40-page scan.
• 9:26
  SPEAKER_00: Ormats that change at the border.
• 9:29
  SPEAKER_00: A London supplier writing 0304 means the 3rd of April, as it would across most of Europe.
• 9:37
  SPEAKER_00: And an agent used to American paperwork might schedule the payment for March.
• 9:43
  SPEAKER_00: Amounts are no safer.
• 9:45
  SPEAKER_00: A German invoice for€1,250 is asking for 1,250 of them.
• 9:52
  SPEAKER_00: Read with American habits, the same figure is a euro and a quarter.
• 9:57
  SPEAKER_00: The body that maintains the metric system settled the argument in 2003 by accepting either a point or a comma as the decimal marker.
• 10:07
  SPEAKER_00: Expect both in the same day's invoices.
• 10:10
  SPEAKER_00: Then there are accents.
• 10:12
  SPEAKER_00: Nunez with the accents stripped off can be, as far as a supplier database is concerned, a different company.
• 10:19
  SPEAKER_00: The N itself can be stored in two different forms, as a single character or as an N plus a separate accent mark.
• 10:27
  SPEAKER_00: On screen the two look identical.
• 10:30
  SPEAKER_00: Yet they can still fail an exact match.
• 10:33
  SPEAKER_00: Before any comparison, the names should be normalized, which takes a single line of code.
• 10:39
  SPEAKER_00: Bilingual pages, the kind cross-border shipping produces by the box, need a few samples of their own in the test set.
• 10:47
  SPEAKER_00: Run the same invoice past the agent in three regional formats.
• 10:52
  SPEAKER_00: If the dates and totals it reports don't match across the three versions, the reader is probably guessing, and an agent will pay on the guess.
• 11:01
  SPEAKER_00: Recording the expected date and number format for each supplier gives the agent a rule to check its reading against.
• 11:09
  SPEAKER_00: Photograph test documents the way customers do.
• 11:12
  SPEAKER_00: Ranked by how often it happens, the bad photograph might deserve a higher place than sixth.
• 11:19
  SPEAKER_00: Many of the agents that collect invoices and receipts for finance teams are likely to meet more phone snapshots than clean PDFs, most of them taken at an angle under a desk lamp.
• 11:31
  SPEAKER_00: Clean scans flatter almost every reader.
• 11:34
  SPEAKER_00: In the industries where fax still turns up, a fax of a photocopy is close to the hardest page a reader will ever meet.
• 11:43
  SPEAKER_00: Thermal paper receipts also fade as they age, so a receipt pulled out for an audit can be harder to read than the day it was printed.
• 11:52
  SPEAKER_00: Recapture a few dozen of the first test documents the way customers really do, crooked and badly lit, then score the fields again.
• 12:01
  SPEAKER_00: The gap between the two scores is the figure to compare across vendors.
• 12:06
  SPEAKER_00: A prompt in the upload screen to flatten the page or move out of the glare costs almost nothing, and on phone photos, it can matter as much as the engine.
• 12:17
  SPEAKER_00: Keep measuring after launch.
• 12:19
  SPEAKER_00: This column has spent most of its length arguing against averages.
• 12:24
  SPEAKER_00: The seventh test is the exception.
• 12:27
  SPEAKER_00: Unlike the others, it can't be finished before launch.
• 12:31
  SPEAKER_00: Suppose a supplier redesigns its invoice in March and a branch office swaps its scanner in June.
• 12:38
  SPEAKER_00: Field accuracy can then slide a point at a time without anyone deciding it should.
• 12:44
  SPEAKER_00: A running field accuracy average can sit in the agent observability tools, many teams already use for cost and speed.
• 12:53
  SPEAKER_00: When something does change, that average is often the first place it shows.
• 12:58
  SPEAKER_00: Split it by supplier.
• 13:00
  SPEAKER_00: In a blended figure, one supplier's collapse can vanish inside everyone else's good month.
• 13:07
  SPEAKER_00: Even that average depends on someone checking a sample of live documents by hand.
• 13:12
  SPEAKER_00: Every week, long after the project team has moved on, a few dozen documents a week from the largest suppliers is a sensible place to start, with more added whenever the numbers wobble.
• 13:25
  SPEAKER_00: Plenty of project plans leave that checker out.
• 13:29
  SPEAKER_00: The six tests before this one can be run once and signed off.
• 13:34
  SPEAKER_00: But this one carries a salary for as long as the agent keeps reading.
• 13:39
  SPEAKER_00: Before anyone signs, every vendor on the shortlist should get the same 200 of your own documents photographed the way your customers send them and report back field by field.
• 13:53
  SPEAKER_00: Get the results in writing.
• 13:55
  SPEAKER_00: Then put a named person and a budget line beside the seventh test before launch rather than after.
• 14:03
  SPEAKER_00: A vendor that declines to test on your documents probably knows how its reader handles them.
• 14:09
  SPEAKER_00: The expectation of a named human checker now appears in at least one government's guidance.
• 14:15
  SPEAKER_00: In Singapore, the updated framework for agentic AI, revised in May, added guidance on automation bias and still holds that humans are accountable for what agents do.
• 14:28
  SPEAKER_00: Handing that checking job to a second agent is the obvious next move, and plenty of teams will make it.
• 14:35
  SPEAKER_00: Few of those teams have decided who will check the second agent's reading.
• 14:40
  SPEAKER_00: All links to sources are available in the text version of this article.
• 14:45
  SPEAKER_00: You can find the full article at easycoding.tools slash blog.
• 14:51
  SPEAKER_00: Thanks for listening.
• 14:52
  SPEAKER_00: For more practical AI coding guides, tool comparisons, and resources for builders, visit easycoding.tools.
• 15:00
  SPEAKER_00: And if it's easier to remember, just go to aibuilds.it.
• 15:05
  SPEAKER_00: It redirects to easycoding.tools.