What is table recognition: a complete guide to turning numbers in a table into business data

Table recognition is the processing that identifies row and column boundaries and the span of merged cells in a table drawn on a document, restores which field each value belongs to, and converts the result into structured data a machine can use directly.

Why table recognition is needed

Every character was read, and the numbers still do not add up.

Extracting text from documents is not new. On printed material, character-level accuracy above 99 percent is unremarkable, and one public-sector evaluation recorded 98.80 percent text similarity. In that same evaluation, table structure similarity came in at 79.18 percent. The characters were nearly all read; restoring which cell they had been sitting in is where a gap of almost twenty points opened up.

What that gap does to the work is the real issue. A VAT return puts major category, minor category, classification, line number, amount, rate, and tax into a single table. Reading the figure 5,000,000 correctly is of no use if the value cannot be tied to the specific line it belonged to. The operator opens the original again to check, which adds a step rather than removing one. Table recognition is not a problem of reading values; it is a problem of restoring where those values lived.

What table recognition actually is: how it differs from OCR

OCR and table recognition across three axes

First, the target differs. OCR works on characters in an image. Table recognition works on the cell boundaries, merge spans, the split between header and data regions, and the coordinate relationships between values. Many tables are drawn without ruling lines at all, so boundaries implied by whitespace and alignment have to be inferred as well.

Second, the objective differs. OCR aims to transcribe every character. Table recognition aims to produce data with the same structure as the original table. Given the same table, OCR lists the strings inside the cells in order; table recognition rebuilds a grid with row and column coordinates and then places the values into it.

Third, the metric differs. OCR is measured by character accuracy or fuzzy similarity. Table recognition is measured by table structure similarity, which converts the extracted table into a tree and compares it against the tree of the original. Every cell value can be correct and the score still falls if a merge span is wrong. That is why a decision made on text accuracy alone produces surprises on table-heavy documents.

The two are not alternatives. Characters OCR failed to read cannot be filled in by table recognition, and when structure restoration fails, correctly read characters lose the place they belong. Structure restoration goes on top of character recognition.

What it takes to put table recognition into production

Six conditions that hold in real operations

First, merge spans have to be identified precisely. A cell merged across both rows and columns creates a structure in which one value represents several rows beneath it. In one municipal document on the return of disbursed funds, a single value in the classification column covers the three rows below it, national and provincial funds each split into balance and interest, and those split again into revenue and expenditure accounts. Misreading the span moves an entrusted project's return account onto a directly executed project.

Second, the field path has to be restored through multi-level headers. Government forms and tax returns commonly stack headers two or three deep. Pulling one value requires the entire path from the top header down, not just a column name.

Third, elements that obscure the table have to be handled. Real intake documents carry stamps over the table, strikethroughs across rows, and overlapping watermarks. A person reads the digit under a stamp without difficulty; to a model identifying structure, the stamp erases a cell boundary. One stamp that shifts the row division throws off the field binding for every value below it.

Fourth, information outside the table has to stay connected. The unit may be given as thousands in the table caption, an exception may be attached by footnote marker, or a reconciliation formula may be written as prose below the table. Cutting out only the table loses that information and can shift a value by a factor of a thousand.

Fifth, performance has to be measured with a structure metric. If the only figure offered during evaluation is character accuracy, table performance is simply unverified. Ask for table structure similarity separately, measured on your own documents. The fact that public-sector evaluations have produced both 86 percent and 79.18 percent on table structure tells you how much this metric moves with document type.

Sixth, each value's position in the source has to be returned. Without it, a reviewer checking one suspect value ends up reading the whole table again.

How table recognition is applied in practice

Decide the output format first

The first decision in the project is not which model to use but which output format to take. Receiving a table as HTML, as Markdown, or as a JSON schema changes every step that follows.

Tables with merged cells are safest in HTML, where row and column spans are expressible as attributes and almost no structure is lost. Markdown reads well and drops straight into a language model, but it has no way to express a merge, so merged cells are either flattened by repeating values or the structure is abandoned. For documents headed into retrieval and generation, producing both formats is the practical choice. When the goal is entering values into a business system, taking JSON keyed by field path from the start is the better fit.

Build it on layout analysis, not on character recognition

Table recognition is not a standalone function; it runs on the output of layout analysis. The table region is detected on the page, row and column structure is identified inside it, and only then are the characters in each cell recognized. The accuracy of that first step therefore caps table performance. In one evaluation, all 302 assessed elements were detected with 100 percent positional accuracy while table structure similarity stood at 79.18 percent, which shows that finding a table and rebuilding its interior are problems of different difficulty.

On method: fixing coordinates by rule works only where the form is fixed. A model that interprets layout and context together makes the difference on tables without ruling lines and on tables with complex merges.

Use the review screen to inspect structure

Table recognition is hard to judge without looking at the result. A screen listing cell values alone gives no way to tell whether merges were captured or header paths are correct.

The review environment should therefore place the source image and the reconstructed table side by side, with the corresponding cell highlighted in the original when a value is selected. Showing the header path alongside makes it immediately clear which field a value belongs to. Once a reviewer can check a suspect cell without rereading the table, exception-based operation becomes workable even on table-heavy documents.

Table recognition in the Korean environment

Korean documents present conditions that overseas solutions have generally not prepared for.

Format comes first. Public institutions and a large share of enterprises use Korean word processor files, where table cell attributes and padding behave differently from other formats. If merge information is lost during conversion, nothing downstream can recover it, so support for the format has to be confirmed at the conversion stage.

Then the structure of the forms themselves. Government forms and tax returns put line numbers, circled figures, and instructions for completion inside the same table. A cell is sometimes guidance rather than data. Extracting such a cell as a value contaminates the business data, so distinguishing instructions from entered values has to be part of the processing.

Finally, the condition of the documents. Low-quality scans received by fax, images photographed on a desk with creases and shadows, and pages where retransmission has stacked headers on top of each other all arrive in the intake queue. Ruling lines break or bleed under those conditions, and following the lines is not enough to rebuild the table.

On top of this, financial and public organizations are frequently subject to network separation, which blocks outbound API calls. Whether the entire document pipeline, table recognition included, can run inside the internal network becomes the first gate in technical review.

Frequently asked questions