What data drift is: a complete guide to the change in data that arrives during operation

Data drift is a phenomenon in which the characteristics and distribution of data arriving in the operating environment diverge from what they were when the model was built and evaluated, and it gradually erodes the performance of AI that appears to run normally.

Why data drift is a problem

It is the same model as yesterday, but the results have changed.

When document AI is adopted, performance is validated on the documents of that time. After validation, the model stays the same, yet a few months later more cases are separated out as exceptions or errors recur on particular fields. The model has not got worse; the documents coming in have changed.

Forms are revised, intake channels change, and documents from new business partners mix in. Replacing scanning equipment or opening mobile photo submission changes image quality too. Each change looks small on its own, but together they make performance at validation time impossible to rely on.

What makes data drift difficult is that it progresses quietly. The system runs without errors and results come out in the right format, so unless someone watches for it, the drop in performance is noticed only much later.

What data drift actually is: how it differs from concept drift

Concept drift and data drift across three axes

First, what changes differs. Data drift is a change in what the incoming data looks like. Concept drift is a change in the relationship between input and correct answer, in other words in what counts as a correct result.

Second, the typical causes differ. Data drift comes from form revisions, changes in intake channels and changes in image quality. Concept drift arises when the criteria for business judgment change, such as a legal revision or a change in review criteria.

Third, the response differs. Data drift is handled by adding data with the new appearance to the evaluation set and supplementing extraction criteria or preprocessing. Concept drift requires revising the judgment rules and the ground truth data themselves to the new criteria.

In real operations the two often arrive together, as when a form revision also changes the fields that have to be checked. Only by recording the causes separately can it be decided which to address first. If the records are mixed, a form problem can be mistaken for a criteria problem and the rules revised wrongly.

The four types of data drift

Four types that often appear in document work

First, form changes. Even a form with the same name changes field positions, table structure and terminology when it is revised.

Second, quality changes. The condition of images changes, such as scan resolution, fax noise, and camera angle and shadows. As mobile submission grows, this type becomes prominent.

Third, composition changes. The combination of documents merged into one file, or their page order, changes. A rise in errors at the classification stage is the first sign.

Fourth, changes in value distribution. The distribution of the values themselves changes, such as the size of amounts, the range of dates, or the share of cases where a particular field is empty. Exceptions rise as the thresholds in validation rules stop fitting.

Each type shows up first at a different stage. Form and composition changes raise errors first in classification and extraction, quality changes in recognition, and changes in value distribution in validation, so seeing which stage's metrics moved narrows down the type.

How to respond to data drift in practice

Keep a record of the baseline distribution

Record the composition, quality and field value distribution of documents at the point adoption validation was completed. Values such as the intake share by form, the distribution of image resolution and the share of empty fields become the reference for comparing change later. Tying the baseline distribution to the same version number as the ground truth data avoids confusion later about which point is being compared.

Without a baseline, even when performance falls in operation it is hard to tell whether the documents changed or something else is the cause.

Detect it early through operating metrics

Accuracy cannot be measured directly on operating data that has no ground truth. Instead, watch weekly the average confidence score, the share of low-score cases, the share of cases separated out as exceptions and the share of values reviewers corrected. When these move together, it is a signal to suspect drift.

Breaking the figures down by form and by intake channel quickly shows where the change started.

Once confirmed, update the evaluation set first

When drift is confirmed, sample documents with the new appearance, add them to the ground truth data, and re-score the current version. If performance falls short of the standard, validate a new version with supplemented extraction criteria or preprocessing on the same evaluation set before applying it to operations.

Revising the criteria without updating the evaluation set makes it impossible to confirm whether the revision actually improved results on the new documents. Keep the updated evaluation set as a new version and retain the previous one so the change can be explained.

Data drift in the Korean environment

In Korea, government forms often change when laws and notices are revised, and some work, such as year-end tax settlement, sees intake volume and document composition shift at particular times of year. Knowing these schedules in advance lets drift be handled as a planned check instead of being discovered after the fact.

In finance and the public sector, operating data is hard to move outside the organisation, so monitoring and re-evaluation have to be possible inside the internal environment.

Frequently asked questions