View all glossary terms

Data and quality

What Is Personal Information De-identification? The Prerequisite for Using Documents as Data, Explained Completely

Korea Deep LearningPublished

Personal information de-identification is the process of finding the items in a document that can identify an individual, then masking them or replacing them with other values, so that the document can be moved to another purpose such as training or search.

Why De-identification Is a Prerequisite

It has to be removed before the data can be used.

Organizations are increasingly trying to connect their document assets to search and generative AI. But when you open the actual documents, names, resident registration numbers, account numbers, addresses and contact details appear everywhere. Applications and supporting documents are mostly bundles of personal information.

If a document in this state is loaded into an index as it is, personal information is exposed in search results. If it is used for training, the model may end up memorizing those values. For this reason, when public data is moved to another purpose, de-identification is in practice a prerequisite rather than an option. The order also matters. If processing happens after the index is built, the original values are already in the search targets.

The Precise Definition of De-identification: How It Differs from Masking

Three Axes Along Which Screen Masking and De-identification Differ

First, the point of application differs. Screen masking hides values at the moment they are displayed, and the original value remains in the data. De-identification changes the data itself.

Second, the goal differs. The goal of screen masking is to control viewing. The goal of de-identification is to make the data usable for another purpose.

Third, whether it can be reversed differs. Screen masking shows the original to anyone with permission. De-identification can be made irreversible depending on the method, or it can be set up so that the original is restored only when needed through a separate mapping table.

The two methods are not alternatives to each other. A common setup in practice keeps the originals in a controlled area and limits viewing with screen masking, while using de-identified copies for search and analysis.

Five Criteria for Designing De-identification

Five Criteria Applied in Practice

First, define the target items as a list. Include not only the obvious items such as names and resident registration numbers, but also items that can identify someone when combined. Affiliation, job title, date of birth and detailed address together make identification possible.

Second, decide the treatment for each item. Whether to delete entirely, partially mask, or replace with another value changes how useful the data remains afterward. Deleting dates outright makes time-series analysis impossible.

Third, handle images and text together. In scanned documents the values are inside the image, so processing only the text leaves the original image untouched.

Fourth, keep the order. It must be processed before the chunks are created and before the index is loaded.

Fifth, set up a verification step. After automatic processing, a person needs to sample documents and confirm that nothing identifying remains.

How to Apply De-identification in Practice

Separate Originals from Copies

Deleting the original is risky, because there will inevitably be situations where the basis must be checked later.

In practice, the original is kept in a controlled area, and a de-identified copy is made separately and used for search and analysis. Giving the two areas different access permissions makes it possible to satisfy both usability and control.

Match the Strength to the Purpose

The stronger the de-identification, the safer the data, but the less useful it becomes. If the purpose is statistical analysis, de-identify strongly; if the purpose is document search, process only the specific items. The right level has to be set according to the purpose.

Processing everything in bulk without a defined purpose leaves data that cannot be used later.

De-identification in the Domestic Setting

In Korea, personal information protection laws and institution-specific guidelines apply together. Public institutions often have their own processing standards, so check the guidelines of the institution concerned before introduction.

Document conditions matter too. Resident registration numbers and account numbers have fixed formats and are relatively easy to catch, but names written by hand, names inside seals, and ID cards in photographs are easy to miss. Include the image areas in the processing scope and add sample verification.

The Order of Processing Determines the Result

When de-identification is performed affects its effectiveness. If it is done after the search index is built, the original values remain in the index and can be exposed in search results.

For that reason, the safe order is to perform it right after the document is structured and before the chunks are created. At this point, the type of each item has already been determined, so items such as names, resident registration numbers, account numbers and addresses can be pinpointed precisely.

The processing method also varies with the purpose. For screen display, partial masking is enough, but when the data is moved for training or search material, replacing or deleting is the safer choice. In one card-financing screening automation, a configuration that automatically handled masking and system linkage for normal cases was used.

Frequently asked questions

What happens to the originals?

It is safer to keep them as they are in a controlled area. The usual setup is to create a de-identified copy separately and use it for search and analysis.

When should processing happen?

Before the chunks are created and before the search index is built. If it is done after the index is built, the original values remain in the search targets.

Are images in scanned documents also processed?

Yes, it is possible. Because the values are inside the image, processing only the text leaves them in the original image. Confirm that image area processing is included.

Is verification needed after processing?

Yes. After automatic processing, sample documents and set up a procedure in which a person confirms that no items remain.

Can it be reversed?

It depends on the method. If a mapping table is kept separately, the values can be restored when needed. Without one, the process cannot be reversed. Decide according to the purpose.

When should it be done?

Right after the document is structured and before the search chunks are created. At that point the type of each item is already known, so the items to de-identify can be pinpointed. If it is done after indexing, the original values remain in the search targets.

Should I use masking or replacement?

It depends on the purpose. For screen display, masking is enough. When moving data for training or as search material, replacing or deleting is safer.

Tell us the documents you plan to process and their purpose, and we will outline the scope of de-identification together with you.

Contact us