What de-identification is: a complete guide to making documents usable for another purpose

De-identification locates the fields in a document that could identify an individual and masks or replaces them, so the document can be moved into another use such as training or retrieval.

Strip it out before using it as data

It has to come out before the document can be used as data.

More organizations are connecting their document estates to retrieval and generative AI. Open the actual documents, though, and names, resident registration numbers, account numbers, addresses, and contact details sit throughout. Applications and supporting certificates are largely made of personal data.

Index those documents as they are and personal data surfaces in search results. Train on them and the model risks memorising those values. Moving public data into another use therefore makes de-identification close to a precondition rather than an option. Order matters too: process it after the index exists and the original values are already within scope of retrieval.

What de-identification actually is: how it differs from screen masking

Screen masking and de-identification across three axes

First, the point of application differs. Screen masking hides at the moment of display; the original value remains in the data. De-identification changes the data itself.

Second, the objective differs. Screen masking controls who can view. De-identification makes the data usable for a different purpose.

Third, reversibility differs. Screen masking reveals the original to anyone with permission. De-identification can be made irreversible, or paired with a separate mapping table so restoration happens only when needed.

The two are not alternatives. Keeping originals in a controlled area with screen masking limiting access, while retrieval and analysis run on a de-identified copy, is the common arrangement in practice.

Five criteria for applying it

Five criteria that hold in application

First, define the target fields as a list. Not only the obvious ones such as name and registration number, but fields that identify an individual in combination. Affiliation, position, date of birth, and a detailed address together make identification possible.

Second, decide the treatment per field. Deleting entirely, masking partially, or replacing with another value changes what remains usable afterwards. Deleting dates outright makes time-series analysis impossible.

Third, handle images alongside text. In scanned documents the value sits inside the image, so processing text alone leaves it intact in the original image.

Fourth, keep the order. Process before chunks are built and the index is loaded.

Fifth, add a verification step. After automatic processing, sampling and having a person confirm nothing remains is necessary.

How de-identification is applied in practice

Manage originals and copies separately

Producing a de-identified copy does not mean the original can be discarded. In most cases the original is still needed as the business record.

Keep originals in a controlled area and run retrieval and analysis on the copy. Placing both in the same location blurs which is being referenced and the control loses its meaning.

Retain the verification samples

Even with a human confirmation step after automatic processing, unless the result of that confirmation is recorded there is nothing to improve on next time. Knowing which fields are prone to being missed allows the rules for those fields to be strengthened.

The fields that get missed are often predictable: contact details written by hand, an account number buried inside a table, a name hidden under a stamp.

De-identification in the Korean environment

Korean public and financial documents carry specific conditions. A resident registration number has a fixed format and is straightforward to detect, but information that allows inference about an individual from context — the substance of a case, a diagnosis — cannot be caught by format. When building the field list, separate what can be caught by format from what requires contextual judgment.

On top of this, in a network-separated environment the de-identification processing itself has to run inside the internal network. Sending documents to an external service for processing contradicts the purpose of de-identification.

Frequently asked questions