Ground Truth Data Explained: The Reference Data That Scores AI Results, Complete Guide

Ground truth data is a collection of correct values confirmed by people for each item or question in a document. It is the standard used to score AI results and compare performance across versions.

Why ground truth data is needed

No one could confirm how accurate it was.

The first question asked when introducing document AI or changing a model is how accurate it is. But unless what counts as the correct value is decided in advance, this question cannot be answered. Figures in a proposal or public benchmark scores do not show performance on your own documents.

The same problem arises in operations. To judge whether results improved after revising extraction criteria or prompts, you must score before and after on the same basis. Without a standard, staff scan a few cases and judge, and those judgments differ from person to person.

Ground truth data fixes this standard as data. Once built, it can be reused repeatedly for adoption validation, change validation during operation, and model replacement comparisons. Even when a model or vendor changes, scoring with the same ground truth data allows a fair comparison.

The exact definition of ground truth data: how it differs from training data

Three axes of difference between training data and ground truth data

First, the purpose differs. Training data is examples that show the model the rules to learn. Ground truth data is the standard for scoring how many the model got right.

Second, management differs. Training data is continually increased and changed to raise performance. Ground truth data has its version fixed so comparison is possible, and when it is changed, the reason and timing are recorded.

Third, the required precision differs. Training data can still teach the overall trend even if some errors are mixed in. If there are errors in ground truth data, the whole scoring result is distorted, so each item must be confirmed. One wrong answer causes a correctly extracted result to be recorded as wrong.

If the same document is used for both purposes, the model is scored on documents it has already seen, and performance comes out higher than reality. So the two datasets are managed separately from the start.

Four principles for building ground truth data

Four principles for building reliable scoring criteria

First, reflect the real business distribution. If only well-performing documents are collected, scores drop in operation; if only difficult documents are collected, the gaps do not show. Mix forms and document conditions in line with the actual intake ratio.

Second, document the judgment criteria. Decide the scoring rules in advance, such as whether a date notation difference counts as correct or whether spacing differences are allowed. Without criteria, the same result is scored differently by different people.

Third, have two or more people confirm. An answer made by one person keeps that person's mistakes. An efficient method is to have two people enter the values separately and re-check only the items where the results diverge.

Fourth, fix the versions. If you change the ground truth, previous scores can no longer be compared, so keep a change history and compare only within the same version.

Building this data according to these four principles is also the most time-consuming part of BMT. If the process is rushed, every later comparison is shaken, so reflect it fully in the schedule.

Practical application of ground truth data

Build it by splitting into evaluation elements

Judging a whole document as correct or wrong does not reveal where it is weak. Ground truth needs to be built per evaluation element, such as items, tables, and text blocks, so areas to improve become visible. Each element uses a different metric: text is judged by similarity, table by structural similarity, and input items by exact match.

In a public sector case, the elements to detect in the documents under evaluation were first listed, and element detection, text, and table structure were measured separately.

Mark unjudgeable items separately

Sometimes from the document alone you cannot tell whether an empty cell means not applicable, an omitted entry, or scan damage. If such items are scored as wrong, you end up evaluating document quality instead of the model, so classify them as unjudgeable and count them separately.

Unjudgeable items are not discarded. Decide how to handle them as operating rules. If you decide whether to request supplementation or hand the item to a staff member for confirmation, that decision carries directly into the exception handling criteria for operations.

Build search and generation work per question

When evaluating document search or generated answers, rather than items, create a bundle of question, expected answer, and evidence location. In a tax-field case, the search evaluation was also built by first setting the target documents and then composing the question set.

Questions should be written in the wording actual staff would use. Include questions that require reading several documents together, so the real performance shows.

Ground truth in a domestic environment

Ground truth data often contains the original values from real business documents, so it frequently includes personal information. Items such as names and resident registration numbers should be de-identified before the ground truth is made. Where the original is needed, work is done only inside the internal network with access restricted.

In finance and the public sector, outsourcing evaluation is often difficult. It is better to prepare judgment criteria documents and an entry tool together, so that the responsible department can confirm the ground truth directly.