Data and quality
Hallucination Explained: When Document AI Produces Values That Are Not There, Complete Guide
Korea Deep LearningPublished
Hallucination means that when a model cannot confirm the evidence, it produces the most plausible-looking value instead of saying it does not know. In document processing, it shows up as amounts or dates that do not exist in the original appearing in the results.
Why hallucination is especially dangerous in document work
A plausible value is more dangerous than a wrong one.
In conversational AI, when a fact is off, the user notices something strange and checks again. In document processing, that does not happen. If a supplier name extracted from a tax invoice is filled in with a company name that is not in the original, in a plausible form, the reviewer finds no reason to doubt it. The format is right, the digits are right, and it fits the context.
So in document work, a difference of 1% in accuracy does not end as a 1% error in processed cases. In a review operation handling 15,000 cases a month, 1% means 150 cases, and those 150 cases get registered in the core system without anyone catching them. An empty value is noticed by staff, but a filled-in value passes through. This tendency to fill in blanks is why hallucination is the hardest problem to handle in document AI.
The exact definition of hallucination: how it differs from recognition error
Three axes of difference between recognition error and hallucination
First, the cause differs. A recognition error is misreading a character that is in the original, such as where strokes are smeared or overlapping and read as another character. Hallucination is a value generated without any basis in the original. It is not a reading failure but excessive generation.
Second, the way it appears differs. A recognition error usually produces an odd result. It shows up as a nonexistent word or a broken number, so it is caught by post-processing rules or human review. Hallucination produces a plausible result. It passes format validation and looks natural to people, so it is not discovered unless it is compared against the original.
Third, the response differs. Recognition errors are reduced by improving image quality and model accuracy. Hallucination is not solved by raising accuracy. It requires designing to control generation itself, so that no value is created when confidence is low. In a public administrative document, when we randomly sampled and measured, the difference in value recognition accuracy was much larger than in key recognition accuracy. How the system behaves when it cannot find a value creates this gap.
The two problems are not substitutes. When recognition accuracy is low, situations where the evidence cannot be confirmed increase, and the room for hallucination grows too. The right order is to place generation control on top of recognition quality.
Five design requirements for controlling hallucination
Five requirements for controlling hallucination in real operations
First, if confidence is low, do not confirm the value. If every item is designed to be filled in, the model creates something even without evidence. The option to leave the value empty and pass it to review must exist within the model's output.
Second, return the original location with each value. When the location in the original where an extracted value came from is returned with it, a value without evidence cannot show a location and is exposed immediately. It also lets reviewers check only the suspicious items.
Third, provide confidence scores per item. A single score for the whole document does not reveal which item to look at. Only with confidence per item can operations show only the items below the threshold on the review screen.
Fourth, run validation rules together. Conditions the business requires, such as date order, agreement of amount totals, and name matching across documents, are set as rules, not left to the model. Cross-validation, which matches the same item across several documents, is especially effective at filtering out generated values.
Fifth, distinguish the state of an empty value. Real intake documents contain many blanks that require judgment. There are boxes with only two periods, boxes covered by a white bar, rows where only the printed unit remains with no value, and tables where fourteen cells are entirely empty. Sometimes the document contains no evidence for whether a blank is not applicable, an omitted entry, or scan loss. Such items should not be filled in arbitrarily; the design should mark their state and pass them on.
Practical application of hallucination control
Use exception handling in the extraction criteria
The item definition sheet created at the start of a project should specify not only what to extract but also when not to extract. Printed guidance text, such as a signature or seal placeholder, is not a value and must be excluded from extraction. The same goes for guidance sentences such as "submitted applications are not returned." If this distinction is not defined, the model mistakes printed text for input values.
In one card issuer's automation, understanding the document well enough to separate guidance text from actual input values was treated as the key challenge. In many cases, what not to extract matters more to the quality of results than what to extract.
Use the review screen as a control device
Generation control does not happen only inside the model. If extraction results are shown on a review screen that highlights only low-confidence items, and clicking each value shows the corresponding location in the original, review time drops significantly.
Adding a screen that manages extraction items and schemas per document type lets you respond to new forms by revising the extraction criteria when they arrive. Keeping criteria as versions, and having a procedure to compare against the previous version using the same document, lets you confirm whether a change lowered performance.
Double-verify with two technologies of different nature
If only one method is used for judgment, whatever it misses passes through. So in practice two technologies of different nature are layered.
One recognizes text and structure. It confirms whether characters were actually at that position and which table cell they came from. The other understands context and business meaning. It checks whether the value makes sense as that item and whether it is consistent with the context of the whole document.
When the two judgments disagree, or results fall outside the defined format, they are automatically separated. This is a device to keep values lacking evidence from being reflected in downstream systems. In the 3 Zero AI Worker strategy, this structure is handled under the name Zero Hallucination.
Hallucination control in a domestic environment
Domestic documents contain several conditions that easily lead to values being wrongly generated.
First, number notation. If a document that uses periods as thousands separators is read as is, 20 million won becomes 20 won. Some forms place two amounts that share the same leading digits but differ in one middle digit side by side, as the initial declaration and the correction claim items. For such documents, digit-count validation and item path confirmation must run together as rules.
Next, item numbering. Registration forms exist where all eleven business purpose items begin with number 1. You must judge whether there are eleven items or one item repeated, and re-numbering without basis creates a structure that is not in the original.
Finally, document condition. In faxes that have been resent many times, different date notations overlap on one line, and sometimes the document contains no evidence for deciding the receipt date. For such items, the correct handling is to pass them on as unjudgeable instead of creating a value.
Frequently asked questions
If accuracy is high, does hallucination also decrease?
Only partly. High recognition accuracy reduces situations where the evidence cannot be confirmed, but how the system behaves when it fails to find a value is a separate design problem.
How far can the occurrence rate be lowered?
In a case validated on PDF documents based on real data, it was confirmed at a very low level. It depends on the document type and item definitions, so measuring with your own documents is more accurate.
How can you tell whether a value was generated?
Judge it by the original location and confidence returned with the value. A common practice is to separate items that cannot show a location, or that have low confidence, as review targets.
How are blanks handled?
They are not filled in arbitrarily. Their state is marked and they are passed on. Since there are cases where the document contains no basis for judging whether a blank is not applicable or an omitted entry, it is better to set this judgment as a business rule.
How far should validation rules go?
As far as the conditions the business requires. Once checks that staff used to perform by hand, such as date order, amount totals, and name matching across documents, are moved into rules, most generation errors are filtered out.
Does matching several documents help?
It is very effective. A value generated in one document will not match the same item in another document, so it shows up as a mismatch in the cross-validation stage.
Send just one real business document, and we will reply with the extraction result together with evidence locations and confidence.
Contact us






