View all glossary terms

Core concepts

What Is Unstructured Data: Data That Cannot Go Straight Into a System, Explained

Korea Deep LearningPublished

Unstructured data refers to data in the state of documents, images, scans and faxes that are not arranged in rows and columns and therefore cannot be loaded into a database as they are. People can read it, but systems cannot process it directly.

Why Unstructured Data Is a Problem

The data exists, but it cannot be used.

Most of the data that companies and institutions hold is unstructured. Contracts, applications, supporting documents, official letters, manuals and meeting minutes pile up on file servers and in document management systems. By volume, no organization is short of data.

The problem is that this data cannot be queried. To check which conditions a certain contract contains, someone must open the file and read it. Introducing generative AI does not change this much, because the data is not in a form the model can take. Whether it is business automation or search, the starting point is the step of reading this data.

The Exact Definition of Unstructured Data: How It Differs from Structured Data

Three Axes That Separate Structured and Unstructured Data

First, the form differs. In structured data, items and values sit within a structure defined in advance. Unstructured data has no such structure. The same business registration number may appear inside a table in one document, inside a sentence in another, and next to a stamp in a third.

Second, the way it is handled differs. Structured data can simply be queried. Unstructured data must go through a step of reading it and identifying its structure before it becomes something that can be queried. This step is the domain of document AI.

Third, the way quality is measured differs. For structured data, you check for missing and duplicate records. For unstructured data, you must look at both recognition accuracy and structure restoration accuracy. This is because even when characters are read correctly, the item links can be wrong, and then the data is unusable.

The two kinds are not in opposition. The purpose of processing unstructured data is, in the end, to convert it into structured data so it can be loaded into existing systems.

Five Factors That Make Business Documents Difficult

Five Factors Seen in Real Received Documents

First, the formats are inconsistent. Hangul word processor documents, office documents and spreadsheets come into the same work, alongside PDFs and images.

Second, the condition is uneven. Low-resolution fax scans, images taken on the floor that show wrinkles and shadows, and documents resent several times so their headers overlap all arrive together.

Third, there are many obstructions. Stamps invade tables, strikethroughs cross the text, and watermarks overlap. People read through them naturally, but for a machine they cause misreading.

Fourth, the forms keep changing. Even the same application has different layouts depending on the branch and the year, and government forms are revised periodically.

Fifth, one case consists of several documents. A single work case brings in an average of six types of documents, and the processing is not complete until the information across those documents is cross-checked.

Practical Ways to Handle Unstructured Data

Design Reading and Structuring as Separate Steps

Processing unstructured data is divided into two steps. First, recognize characters, tables and figures. Then assign the structure they form and the business items.

Splitting it this way narrows down the cause when a problem occurs. If a character is wrong, it is a recognition problem. If the character is right but the item link is off, it is a structuring problem.

Decide the Output According to Purpose

Even for the same document, the results to produce differ depending on what it will be used for. If the goal is to enter values into a business system, you need item and value pairs. If the goal is search and generation, you need a form that preserves structure, and fragments.

If you analyze once and create several outputs together, you do not need to reprocess the original document when the way it is used changes later.

Unstructured Data in the Korean Environment

In the document assets of Korean public institutions and finance, the share of Hangul word processor format is large. Whether this format can be processed in its original form is the first gate, and if merge information in tables is lost during conversion, there is no way to recover it.

Intake channels are also a condition. Faxes are still in use, and branch scans and mobile phone photos are mixed in. Many documents contain personal information, so de-identification and internal network operation are required together.

Look at the Condition of the Documents Too

When people hear unstructured data, they tend to think only of the fact that the format is not fixed. In real work, the bigger variable is the condition of the documents. Even the same form can differ greatly in processing difficulty depending on the route by which it arrived.

Real intake trays contain documents whose folded top edge was cut off when scanned, documents where the margins and part of the body were lost through repeated copying, documents photographed on the floor showing wrinkles and shadows, and faxes resent several times with overlapping date marks. Stamps, strikethroughs, watermarks and handwriting are mixed in as well.

So when reviewing adoption, it is better to organize the condition of the documents by intake route, not only the list of forms. Values measured on a clean sample and values measured on documents from the actual intake tray come out differently.

Frequently asked questions

Is a PDF that contains text unstructured data?

Generally, yes. You can extract the characters, but if it is not defined which item each value belongs to, it cannot be loaded into a system directly.

What share of the data does it account for?

It is generally understood that most of the data a company holds falls into this category. It varies by organization, so surveying your actual document assets is the more accurate approach.

Does processing it turn it into structured data?

That is the purpose. After recognition and structuring, converting it into item and value pairs allows it to be loaded into existing systems.

Are Hangul word processor documents also processed?

Yes. However, table structure is easily lost when converting to another format, so check whether the original format is supported.

Where should one start?

Pick one high-volume, repetitive work case and organize the documents that come into that work first. Trying to handle all assets at once makes progress stall.

What does unstructured data refer to?

It refers to documents, images, faxes and scans that are not organized into tables. This category covers most of the data a company holds.

What should be prepared for validation?

Organize the document condition by intake route along with the list of forms. Including damaged documents and faxes is what reveals the actual performance.

Send us one real intake document, and we will return the structured results as they are.

Contact us