What hallucination is: a complete guide to document AI inventing values that were never there

Hallucination is what happens when a model, unable to confirm its grounding, produces the most plausible value instead of saying it does not know. In document processing it surfaces as an amount or a date appearing in the result that was never in the source.

Why hallucination is especially dangerous in document work

A plausible wrong value is more dangerous than an obviously wrong one.

When a conversational AI gets a fact wrong, the user senses something is off and checks. Document processing does not work that way. If a supplier name is extracted from an invoice and a company name that does not appear in the source is filled in convincingly, the reviewer has no reason to doubt it. The format fits, the length fits, and it sits comfortably in context.

So in document work a one percent difference in accuracy does not end as one percent of cases going wrong. A review operation handling 15,000 cases a month sees one percent as 150 cases, and those 150 pass unfiltered into core systems. A blank field gets noticed; a filled field does not. That is why the impulse to fill an empty box is the hardest problem in document AI.

What hallucination actually is: how it differs from a recognition error

Recognition error and hallucination across three axes

First, the cause differs. A recognition error misreads a character that exists in the source, where strokes have smudged or overlapped into something else. Hallucination produces a value with no basis in the source at all. It is not a failure to read; it is excess generation.

Second, how it surfaces differs. Recognition errors usually look wrong, appearing as non-existent words or malformed numbers, so post-processing rules or human review catch them. Hallucination looks right. It passes format validation and reads naturally, and it is not found without comparison against the source.

Third, the remedy differs. Recognition errors are reduced by better image quality and higher model accuracy. Hallucination is not solved by raising accuracy. It requires a design that controls generation so no value is produced when confidence is low. In an evaluation drawing 1,000 pages at random from 900,000 public administrative documents, key recognition accuracy separated at 97.3 percent against 80.4 percent while value recognition accuracy separated far more widely, at 96.0 percent against 48.7 percent. What creates that gap is how the model behaves when it cannot find the value.

The two are not alternatives. Lower recognition accuracy means more situations where grounding cannot be confirmed, which enlarges the room for hallucination. Control of generation goes on top of recognition quality.

Five design requirements for controlling hallucination

Five requirements that hold in real operations

First, do not confirm a value when confidence is low. Designing every field to be filled without fail means the model produces something even without grounding. Leaving the field empty and routing it for review has to exist as an option inside the model's output.

Second, return the source position with every value. When a value arrives with the coordinates it came from, anything ungrounded cannot show a position and is exposed immediately. It also lets a reviewer check only the suspect field.

Third, provide confidence at field level. One score for the whole document tells you nothing about where to look. Per-field confidence is what makes it possible to surface only sub-threshold items on the review screen.

Fourth, run validation rules alongside. Date ordering, sum reconciliation, and name agreement across documents are business conditions and belong in rules rather than the model. Cross-validating the same field across several documents is particularly effective at catching generated values.

Fifth, distinguish the states in which a value is absent. Real intake documents are full of blanks that require a judgment. A cell with two dots, a cell covered by a white bar, a row where only the printed unit label survives, a table with fourteen empty cells in a row. Whether that means not applicable, an omission, or a scanning loss is sometimes undecidable from the document itself, so those fields have to be passed on with their state marked rather than filled in.

How hallucination control is applied in practice

Write the exceptions into the extraction criteria

The field definition document produced early in a project has to record not only what to extract but when not to extract. Printed instructions such as signature or seal are placeholders rather than values and belong outside the extraction set, as do lines stating that submitted applications will not be returned. Without that distinction, the model mistakes printed text for entered values.

In one card issuer's automation, distinguishing instructional text from actual entries was treated as a core task. What not to extract decides output quality more often than what to extract.

Validate twice with technologies of different character

Judging by one method alone lets whatever that method misses pass straight through. Practice therefore layers two technologies of different character.

One side recognises text and structure, confirming that a character really sat in that position and which cell of the table it came from. The other side understands context and business meaning, asking whether the value makes sense as that field and whether it contradicts the document as a whole.

Results where the two disagree, or which fall outside the defined format, are separated automatically. It is a mechanism for stopping insufficiently grounded values from flowing into downstream systems, and the 3 Zero AI Worker strategy calls this structure Zero Hallucination.

Use the review screen as a control mechanism

Control of generation does not live only inside the model. Presenting extraction results on a review screen that highlights only low-confidence fields, and jumping to the corresponding position in the source when a value is clicked, cuts review time substantially.

Adding a screen that manages extraction fields and schemas per document type lets a new form be handled by editing the extraction criteria. Versioning those criteria and comparing against the previous version on the same documents shows whether a change lowered performance.

Hallucination control in the Korean environment

Korean documents contain several conditions that make invented values likely.

Numeric notation comes first. Read a document that uses a period as the thousands separator literally and twenty million won becomes twenty won. Some forms place two amounts that share a leading sequence and differ by a single middle digit side by side as the original filing and the amended claim. Digit-count validation and field-path confirmation have to run as rules on documents like these.

Then line numbering. Corporate registry forms exist in which eleven business purposes are all numbered starting from one. Whether there are eleven items or one repeated has to be judged, and renumbering without grounding fabricates a structure that was not in the source.

Finally, document condition. On a fax retransmitted several times, different date formats overlap on a single line and the document contains no basis for settling the receipt date. Passing such a field on as undecidable, without producing a value, is the correct handling.

Frequently asked questions