Test whether your AI can actually find your documents
You connected your documents to an assistant so you wouldn’t have to read them. That is precisely why you’re not in a position to notice when it answers wrong.
It’s worth sitting with the shape of that, because it isn’t a small problem. The whole reason for pointing an assistant at your contracts, your policies, your project history, is that reading all of it yourself is the work you’re trying to avoid. So when it hands back a confident paragraph, the only person who could catch a mistake is the person who already knows the answer, and that person had no reason to ask.
Nothing about the failure looks like a failure. When retrieval goes wrong it doesn’t return an error. It returns a fluent, plausible, well-structured answer that is subtly wrong in the one way that mattered. Researchers who study these systems describe people discovering bad answers weeks or months later, which is the natural consequence of a failure with no alarm attached.
There’s a nastier version too. When the system pulls in two chunks of text that contradict each other, the model tends to resolve the conflict by inventing a resolution rather than saying it doesn’t know. You get certainty manufactured out of ambiguity.
The good news is that this is cheap to test. You just have to ask it things you already know.
Before you start
Twenty minutes and access to whatever you’ve connected: a knowledge base, a folder, a documents integration, a custom assistant with files attached.
You also need to have been the person who knows some of the answers. If you’ve just inherited the system, borrow whoever wrote the documents for half an hour.
Write twenty questions you already know the answers to
Write the questions first, all of them, before you ask a single one. Writing as you go lets you drift toward things you suspect it can handle.
Write the answer next to each one, in your own words, from your own knowledge. That’s the scoring key and it needs to exist before the test, not after.
Real questions people actually ask this system. Not clever ones designed to break it, and not softballs.
Include the five kinds that break retrieval
A test made only of easy questions tells you nothing. Deliberately include some of each:
- A specific number.
- A rate, a threshold, an amount, a date in a contract.
- An exception.
- A rule with a carve-out. These break because chunking often splits text right where a general rule ends and its qualification begins, so the system finds the rule and never sees the “unless.”
- Something that changed.
- Where an old version and a new version both exist in your documents. This is where you find out whether it knows which one is current.
- Something in a table or a scanned PDF.
- Structure and images survive ingestion badly.
- Something that isn’t there at all.
- The correct answer is “I don’t know” or “that isn’t in these documents.” Whether it says so is one of the most useful things you’ll learn.
Ask all twenty, and score them plainly
One session, no coaching. Ask the question exactly as written and take the first answer.
Do not rephrase to help it, and do not accept an answer that’s nearly right. Score each one right, wrong, or partly, against the key you wrote in step 1.
Partly is the important column. That’s where the fluent, plausible, subtly wrong answers live, and they’re the ones that would have gone unnoticed.
For each miss, find out which failure it was
This is the step that turns a score into something you can act on, and the two failures need completely different fixes.
Did it find the document at all? Most tools will show you sources or citations. If they’re missing or irrelevant, the document was never retrieved, and no amount of better prompting fixes that. The problem is upstream: the file isn’t indexed, or it’s in a format that ingested badly, or the wording of the question doesn’t match the wording of the document.
Or did it find the right document and still answer wrong? Then the text was retrieved and misread, or the relevant sentence was split away from its context, or two versions contradicted each other and it picked.
You cannot tell these apart from the answer. You have to look at what was retrieved.
Fix the documents, not the prompt
For nearly everything you find, the fix is on your side.
Not indexed: get the file into the system properly. Check what formats it actually supports rather than assuming.
Split badly: put the exception next to the rule in the source document. If a qualification lives three paragraphs from the thing it qualifies, expect it to be separated.
Two versions: archive the old one out of the connected folder. An assistant that can see both has no reliable way to know which one you meant.
A scan or a table: convert it to real text. An image of a contract is not a contract as far as retrieval is concerned.
Prompt engineering is the last resort here and usually the wrong tool. A chunk that was never retrieved cannot be rescued by asking more nicely.
Re-run the same twenty, and keep them
Same questions, after the fixes. The number should move, and now you know by how much rather than feeling like it’s better.
Keep the twenty. They’re a regression test, and this is the rare case where a small business genuinely has one. Run them again whenever you add a lot of documents, change tools, or notice something odd, and you’ll know within twenty minutes whether the thing got worse.
Write the score and the date next to them.
A boundary worth knowing about
Twenty questions is a sample, not a proof. A system scoring 18 out of 20 will still be wrong sometimes, and the questions you didn’t think to ask are exactly the ones nobody checked.
What the test actually gives you is a rough rate and, more usefully, a picture of the kind of thing it gets wrong. A system that handles prose well and mangles every number is telling you something specific about how to use it.
The other edge is that this measures retrieval, not judgement. A system can find precisely the right paragraph and still draw a conclusion you disagree with, and that’s a different problem with a different fix. Keep them separate when you’re scoring, or you’ll end up blaming the search for an opinion.
If you have staff
Have somebody else write the twenty questions. You know what you built and you’ll unconsciously ask in the language the documents use. A colleague asks the way a real user asks, which is the way it will actually be used.
Tell people what it’s bad at, specifically. “The assistant is unreliable” gets ignored. “It gets our pricing tiers wrong because the table didn’t import, so check those against the source” gets followed, because it’s a fact rather than a warning.
The short version
- Write twenty real questions and the answers you already know, before asking any of them
- Include a number, an exception, something that changed, something in a table, and something that isn’t there
- Ask all twenty in one go. Score right, wrong, or partly. No coaching
- For every miss, check whether it retrieved the document at all. Those are different problems
- Fix the documents: index them, keep exceptions beside their rules, archive old versions, convert scans
- Re-run the same twenty, record the score and the date, and keep them as a regression test
Sources
- Your chunks failed your RAG in production, Towards Data Science — that chunking sits upstream of the whole stack, that a chunk never retrieved cannot be recovered by a better model, and that splits at the boundary between a rule and its qualification are a common cause
- RAG accuracy problems, Atlan — silent failure with no error signal, and users discovering bad answers weeks or months after deployment
- Why RAG gives wrong answers, AWS on DEV — the distinction between a document never retrieved and one retrieved but misread, and why the fixes differ
Written August 2026. The failure modes here are properties of how retrieval works rather than of any one product, so they should age well. What will change is which formats a given tool ingests cleanly, which is worth rechecking when you switch.
Prints to PDF from your browser — colours and all.