Why a language model's fluency is never evidence, and what to do about it.
I was testing a system that automates official fee forms for law firms. The flow was simple: the model reads the case documents, extracts the data and the system fills in the form. In one test, the model returned an impeccable tax-office code: the right number of digits, the right format, consistent with the rest of the case.
It didn't exist.
The error had none of the signals we usually associate with failure. No empty field, no strange text, no obvious inconsistency. It was exactly what a person would expect to see. A tired reviewer with twenty case files in front of them would have approved it. The filing would have gone in and been rejected days later.
This is the most important thing to understand about language models in serious work: they fail with the same confidence they succeed with. An invented value doesn't sound different from a real one. That's why human review alone isn't enough: people catch what looks odd, and these errors don't.
A boring rule: every office code is checked against the official list before moving on. If it isn't there, the case stops and gets flagged. No AI in that step, just an exact comparison against the source of truth.
The whole system follows three layers:
If a value has a source of truth (an official list, a database, a calculation), don't ask the model to get it right: check it. Keep the model for what can't be checked with a rule, like reading a messy document, and keep people for judgment.
That system cut manual time per filing by 75-90%, with zero incorrect submissions since launch. Not because the model is perfect, but because it was never asked to be.
If you run an AI system in production and don't know which mistakes it makes with full confidence, a technical audit is the first step to find out.