What ground truth data is: a complete guide to the reference data used to score AI results

Ground truth data is data that collects the correct values people have confirmed for each field or question in documents, and it is the standard for scoring AI results and comparing performance across versions.

Why ground truth data became necessary

Nobody could confirm how accurate it really was.

The first question when adopting document AI or changing a model is how accurate it is. But without deciding in advance what the correct value is, that question cannot be answered. Figures in a proposal or public benchmark scores do not show performance on the organisation's own documents.

The same problem arises in operation. Judging whether results improved after revising extraction criteria or a prompt requires scoring before and after against the same standard. Without one, staff glance at a few cases and decide, and that judgment varies from person to person.

Ground truth data fixes that standard as data. Once built, it can be used again and again for adoption validation, validation of changes during operation, and comparison when models are replaced. Even when the model or vendor changes, scoring against the same ground truth data allows a fair comparison.

What ground truth data actually is: how it differs from training data

Training data and ground truth data across three axes

First, the purpose differs. Training data consists of examples shown to a model so it learns patterns. Ground truth data is the standard for scoring how much the model got right.

Second, the way it is managed differs. Training data keeps growing and changing to raise performance. Ground truth data has its version fixed so comparison is possible, and the reason and timing are recorded when it changes.

Third, the precision required differs. Training data can still be used to learn overall tendencies even with some errors mixed in. Errors in ground truth data distort the whole scoring result, so every field has to be confirmed. A single wrong answer records a correctly extracted result as a mistake.

Using the same documents for both purposes means scoring on documents the model has already seen, which makes performance look higher than it is. The two datasets are therefore kept separate from the start.

The four principles that make up ground truth data

Four principles for a scoring standard that can be trusted

First, reflect the real distribution of work. Collecting only documents that come out well makes scores fall in operation, while collecting only difficult ones hides the differences. Mix forms and document conditions in line with the actual intake mix.

Second, write down the judgment criteria. Record the scoring rules first, such as whether differences in date notation count as correct or whether spacing differences are allowed. Without criteria, the same result is scored differently by different people.

Third, have at least two people check. Ground truth made by one person keeps that person's mistakes. An efficient method is for two people to enter values separately and re-check only the fields where their results diverge.

Fourth, fix the version. Revising the ground truth makes comparison with earlier scores impossible, so record the change history and compare only within the same version.

Following these four principles is also the part of a BMT that takes the most time. Rushing it unsettles every comparison that follows, so allow enough for it in the schedule.

How ground truth data is applied in practice

Break it down into evaluation elements

Scoring a whole page as right or wrong does not show which part is weak. Building ground truth by evaluation element, such as fields, tables and text blocks, shows where to improve. The metric also differs by element, for example similarity for text, structural similarity for tables, and exact match for input fields.

In a public-sector case, the elements to be detected in the documents under evaluation were listed first, and element detection, text and table structure were then measured separately.

Mark undecidable fields separately

Sometimes the document alone cannot show whether an empty box means not applicable, an omission or scan damage. Scoring such fields as wrong evaluates document quality rather than the model, so they are classified as undecidable and counted separately.

Fields classified as undecidable are not discarded; operating rules set how they are handled. Deciding whether to request supplementary material or route them to staff for confirmation carries straight over into the exception handling criteria in operation.

Build question units for search and generation work

When evaluating document search or generated answers, the unit is a set of question, expected answer and source location rather than a field. The search evaluation in a tax-sector case was likewise built by fixing the documents in scope first and then writing a question set.

Questions should be written in the words staff actually use, and questions that require reading several documents together should be mixed in for real performance to show.

Ground truth data in the Korean environment

Ground truth data contains source values from real business documents, so it often includes personal information. Fields such as names and resident registration numbers are de-identified before building the ground truth, or, where the original is needed, work is carried out only inside the internal network with restricted access.

In finance and the public sector, evaluation is often hard to outsource, so it is better to prepare the judgment criteria document and an input tool together so the responsible department can confirm the ground truth itself.

Frequently asked questions