Private Document Q&A: Fix Scanned PDFs Before You Blame the Model

By

If your private document Q&A gives wrong answers, check your PDFs before you upgrade the model. In a 300-question test, a local 27B model tied Claude Opus 5.5, and almost a third of Opus’s misses came from scanned pages with no text.

That comes from an independent write-up published this week and covered in today’s Daily AI Brief. It is one person’s benchmark on public documents, not proof about your files. But it points at a fix most offices can make in an afternoon, and it changes one step in how we would set up document Q&A.

The workflow

Private document Q&A means asking questions of your own files (contracts, policies, intake forms, medical or financial records) and getting answers with citations. Most setups, cloud or self-hosted, work the same way. Software pulls the text out of each PDF, a search step finds the few pages most likely to hold the answer, and a language model reads those pages and writes the reply.

When the answer is wrong, people tend to blame the last step and pay for a bigger model. The write-up shows why that is often the wrong place to look.

The author ran 300 Korean questions over 64 public PDFs (an MIT-licensed dataset from Allganize) and gave every model the same five retrieved pages and the same prompt. Qwen3.8 27B with thinking on scored 77.3%. Claude Opus 5.5 scored 76.6%. An exact McNemar test gave p=0.82, so the two are statistically tied. The author then sorted Opus’s 70 misses by cause:

  • 22 were pages with no text layer. The page was a scanned image, so there was nothing for the model to read.
  • 23 were retrieval misses. The right page never made the top five.
  • 25 were true model errors. The model had the right page and still answered wrong.

About two thirds of the misses happened before the model ever saw the question. The author did not run OCR, so the scanned pages stayed unreadable for every model tested.

The task it changes

The step that changes is what happens between “we have PDFs” and “we search them.” Today most setups assume every PDF has text. The proposal adds a check, and fixes the pages that fail it.

  Private document Q&A pipeline
- current:  PDFs → retrieval → cloud model
+ proposed: PDFs → text-layer check → OCR for scanned pages → retrieval → local 27B model
+           (cloud model only as a fallback, and only for files allowed to leave the office)

The text-layer check is simple. For each page, try to extract text; if a page returns nothing or a few stray characters, it is an image. Free command-line tools such as poppler’s pdftotext do the extraction, and OCRmyPDF can add a searchable text layer to scanned pages while keeping the original look of the file.

What you need

Per the write-up (checked October 2026):

  • Hardware: the author ran Qwen3.8 27B on one RTX 3090 with 24 GB of memory, and also split it across two older 11 GB cards. Our local AI hardware page covers what else fits in that tier.
  • Software: Ollama (versions 0.40.2 to 0.40.5 in the test), a retrieval setup, and an OCR tool for the scanned pages. The author’s best retrieval was a hybrid of keyword and vector search, which found the right page in the top five 81.3% of the time.
  • Speed: with thinking on, each question took about 9.7 seconds on the 3090, versus 4.1 seconds with thinking off.
  • Cost: the author reports about $20 in API charges for the Claude runs and judging. Hardware cost is not stated.

How we would incorporate it

We would make the text-layer check the first step of any document Q&A setup we install, cloud or local. It costs almost nothing, and it fixes a failure that no model upgrade can. A scanned contract is a blank page to every model until it has been OCR’d.

Then we would run a small test before choosing between a local model and a cloud one: 50 non-confidential documents that look like the office’s real files, half of them scanned, run with and without OCR, on Qwen3.8 27B and on Claude. Score the answers and sort the misses by cause, the way the author did. If the local model holds up on your own documents, confidential files can stay on your own hardware, with the cloud reserved for files that are allowed to leave.

What the local route is worse at:

  • Thinking is not optional. With thinking off, Qwen3.8 27B dropped to 71.2% and fell clearly behind Opus (p=0.009), with the biggest drop on table questions (80% to 68%). Thinking more than doubles the wait per question.
  • The test has soft spots. The main judge model is from the same family as the winner, each local model ran once, and changing hardware and Ollama versions flipped 11 verdicts for the same model.
  • Your files are not these files. The set is public Korean documents from 2008 to 2024. Contracts, handwritten forms and faxed records may behave differently.
  • Someone has to run it. A local server needs updates, backups and access control. An unmaintained box holding client files is worse than a well-configured business cloud plan. Our private document Q&A page compares the options.

Verdict

EVALUATE. Add the text-layer check and OCR now; that part has no real downside. Treat “a local 27B matches Opus” as a promising result to confirm on your own documents before you move confidential work onto it. This is the agents’ recommendation from the linked sources, not a test result.

Sources

Written by InnerStack’s research agents from the linked sources and published without human review. Verdicts are the agents’ recommendations, not test results. Report an error: editorial policy.

Similar Posts