What Is Data Drift? When the Data Coming into Operation Changes, Explained Completely

Data drift is the phenomenon in which the characteristics and distribution of data entering an operating environment come to differ from those at the time the model was built and evaluated. It is a cause that gradually lowers the performance of an AI that still appears to be working normally.

Why Data Drift Matters

The same model, but the results have changed.

When a document AI is introduced, its performance is verified against the documents of that moment. After passing validation, the model stays the same, but months later the number of cases split out as exceptions grows, or errors keep recurring on particular items. The model has not gotten worse. The documents coming in have changed.

Forms are revised, intake channels change, and documents from new trading partners get mixed in. Replacing the scanners or opening mobile capture for intake also changes image quality. Each change looks small on its own, but as they pile up, the performance measured at validation can no longer be trusted.

Data drift is hard to deal with because it happens quietly. The system keeps running without errors and the results come out in the right format, so unless someone watches for it, the drop in performance is noticed only long after it started.

The Precise Definition of Data Drift: How It Differs from Concept Drift

Three Axes Along Which Data Drift and Concept Drift Differ

First, what changes differs. Data drift is a change in the look of the data being input. Concept drift is a change in the relationship between input and correct answer, that is, a change in what is considered a correct result.

Second, the typical causes differ. Data drift comes from form revisions, changes in intake channels and changes in image quality. Concept drift arises when the criteria for judgment in the work change, such as revisions to laws or changes in review standards.

Third, the response differs. For data drift, you respond by adding the new-looking data to the evaluation set and supplementing the extraction criteria or preprocessing. For concept drift, the judgment rules and the correct-answer data themselves must be revised to the new criteria.

In real operation the two often arrive together. For example, a form is revised and the items to be checked change at the same time. It is necessary to record the causes separately, because that is the only way to decide which one to fix first. If the records are mixed together, a form problem gets mistaken for a criteria problem and the rules are fixed wrongly.

Four Types of Data Drift

Four Types That Commonly Appear in Document Work

First, form change. Even a form with the same name changes its item positions, table structure and terminology when it is revised.

Second, quality change. The state of the image changes, such as scan resolution, fax noise, and shooting angle and shadows. This type shows up strongly as mobile intake grows.

Third, composition change. The combination of documents, or the order of pages, arriving in a single file changes. An early rise in classification errors is the signal.

Fourth, change in value distribution. The distribution of the values themselves shifts, such as the scale of amounts, the range of dates, or the share of cases where a particular item is blank. As the reference values in validation rules no longer fit, exceptions increase.

These four types show up first at different stages. Form and composition changes appear first in classification and extraction, quality changes in recognition, and value distribution changes in validation. Looking at which stage's indicator moved helps narrow down the type.

How to Respond to Data Drift in Practice

Keep a Record of the Distribution at the Baseline Point

Record the document composition, quality and item value distribution at the point when the validation for adoption was finished. Values such as the share of intake by form, the distribution of image resolution and the share of blank items serve as the baseline for comparing changes later. Tying the baseline distribution to the same version number as the correct-answer data keeps it clear which point in time you are comparing against.

Without a baseline, when performance drops during operation, it is hard to tell whether the documents have changed or whether some other cause is at work.

Detect Early Through Operational Indicators

Correct answers are not available in operational data, so accuracy cannot be measured directly. Instead, look weekly at the average confidence score, the share of low-scoring cases, the share of cases split out as exceptions, and the share of values a reviewer corrected. When these values move together, they are a signal to suspect drift.

Splitting the view by form and by intake channel helps you find quickly where the change started.

Update the Evaluation Set First When Drift Is Confirmed

When drift is confirmed, sample documents that show the new appearance, add them to the correct-answer data, and re-score the current version. If performance falls short of the baseline, prepare a new version with supplemented extraction criteria or preprocessing, validate it on the same evaluation set, and only then put it into operation.

If you change the criteria without updating the evaluation set, you cannot confirm that the change actually improved results on the new documents. Keep the updated evaluation set as a new version and retain the earlier one as well, so you can explain the state before and after the change.

Data Drift in the Domestic Setting

In Korea, government forms often change in the same period as the revision of laws and notices, and some work, such as year-end tax settlement, sees intake volume and document mix change at certain times of the year. Knowing these schedules in advance means drift can be handled as a planned check rather than discovered after the fact.

In finance and the public sector, operational data is often hard to take outside, so monitoring and re-evaluation must be possible inside the internal environment.