View all glossary terms

Adoption and operations

What Is BMT? How to Validate a Solution with Your Own Documents, Explained Completely

Korea Deep LearningPublished

BMT (Benchmark Test) is a validation procedure in which an organization considering adoption has several candidate solutions process its own actual work documents under the same conditions, using evaluation criteria defined in advance, so that their performance can be compared and a decision on adoption can be made.

Why BMT Is Needed

Published scores and the scores on our documents are different.

The accuracy figure is the number that most often comes up when solutions are reviewed. But the same product can score high on one set of documents and fall well short on another. In actual validation cases, table structure recovery on tax-report forms was rated high, while the same item dropped sharply on administrative documents with multi-level headers and overlapping merged cells. Document type classification also had cases where financial document sets scored high and public document sets scored low.

This gap comes from the nature of the documents, not from which product is better. So if an adoption decision is made only on a public benchmark score or the accuracy stated in a proposal, the results after going into operation will differ from expectations. BMT moves that risk to before adoption.

The Precise Definition of BMT: How It Differs from a PoC

Three Axes Along Which BMT and a PoC Differ

First, the target differs. BMT targets several candidate products. The purpose is to run the same documents under the same conditions and compare performance side by side. A PoC targets a single solution and checks whether it works when attached to the actual work flow.

Second, the goal differs. The goal of BMT is selection: it judges which product performs better on our documents. The goal of a PoC is design: it fixes the business rules, exception handling and system integration structure.

Third, the performance indicators differ. BMT uses indicators that allow comparison across products, such as document type classification accuracy, item-level exact match rate and table structure similarity. A PoC uses operational indicators such as processing time per case, the share touched by hand, and the share of exception cases. In one financial case, the BMT stage first set an exact match target, measured document classification and item extraction performance, and the operating criteria were confirmed in a later stage.

The two procedures are not alternatives to each other. The usual order is to narrow the candidates with BMT and then fix the operating design with a PoC. However, when the document set is simple and the candidates have already narrowed to one, the two may be run together.

Six Design Principles for Making BMT Results Reliable

Six Design Principles Used in Real BMT Projects

First, the evaluation documents must be drawn to match the distribution of the work. If only well-scanned documents are evaluated, scores drop after adoption, and if only hard documents are gathered, the differences between products do not show. Mix document types and conditions to reflect the actual intake share. In one financial case, the initial document set was organized first and the evaluation targets were selected from it. In one public case, evaluation elements were extracted from the target documents and measured.

Second, the answer set must be built first. Unless people settle what the correct value of each item is, scores cannot be calculated. This is the most time-consuming part of BMT, and if it is glossed over, every later comparison becomes unreliable.

Third, the metrics must be defined in advance. The word accuracy means different calculations for different products. Decide in writing before starting whether matches must be exact or similarity-based, whether tables are judged by structural similarity, and what unit classification is scored on.

Fourth, documents that are exceptions must be included. Documents with a seal crossing a table, fax headers stacked several layers deep, and documents with blank values that require judgment should be included, or the differences between products will not appear. Normal documents alone will give most products similar scores.

Fifth, conditions must be matched. The same files, the same resolution and the same item definitions must be used for the comparison to hold. If only one side gets carefully tuned extraction rules, the result shows preparation time rather than product performance.

Sixth, the processing environment must be checked too. Even with equal accuracy, a product cannot be adopted if it cannot be installed internally or requires an excessively large accelerator setup. Put performance metrics and deployment requirements in the same table and judge them together.

How to Carry Out BMT in Practice

Finish Document Collection and Item Definition First

The first output of BMT is not an evaluation result but a document list and an item definition sheet. Which forms, how many of each, and which items are extracted from each form, and in what format, must be settled before there is a basis for comparison.

It also helps to organize the classification scheme at this stage. In one financial case, the classification code system was set up while organizing the document sets, and only then did the work move on to extraction validation. Measuring extraction performance without a defined classification leaves out the bottlenecks that will appear in real operation.

Separate Measurement from Interpretation

Measure mechanically and interpret separately. You need to distinguish whether a low score on an item is due to the product's limits, an ambiguous answer-set definition, or the absence of supporting information in the original document.

In practice, a blank cell may be not applicable, an omission in entry, or scan loss, and there is sometimes no basis in the document to tell which. Counting such an item as wrong turns the product evaluation into a document quality evaluation. It is better to classify undecidable items separately and set how to handle them with operating rules.

BMT in the Domestic Setting

When running BMT in domestic finance and the public sector, the first thing that runs into trouble is where the documents will be processed. In a network-separated environment, evaluation documents cannot be taken outside, so you must first confirm that a candidate product can be installed temporarily inside and run the validation there. A product that cannot meet this condition cannot be evaluated at all, regardless of its performance.

Check the document format conditions as well. Confirm whether Hangul word-processor files, government forms and fax receipts are included in the evaluation targets. Values measured on publicly available sample documents differ from values measured on your own intake documents. That difference is exactly why BMT exists.

Frequently asked questions

Can't we judge from public benchmark scores alone?

They are a useful reference. However, public evaluation sets and your own documents differ in composition, so rankings may not reproduce. Base the adoption decision on results measured with your own documents.

How many evaluation documents are enough?

It depends on the variety of the document set. There is a financial case that ran the evaluation across several forms, and a public case that extracted evaluation elements from a small number of documents. Whether the actual intake distribution is reflected matters more than the count.

Do we really need an answer set?

Yes. Without confirmed answers, scores cannot be calculated and products cannot be compared. It is the most labor-intensive part of BMT, but it cannot be skipped.

Which metrics should we look at?

It depends on the work. For work that enters values into a system, the item-level exact match rate is the priority. For documents where tables are the main content, use table structure similarity. For automated intake, use document type classification accuracy as the primary metric.

Can BMT be done in a network-separated environment?

Yes. If the candidate product is temporarily installed internally and the validation is run there, the documents do not leave the environment.

Does operation start right after BMT finishes?

Usually there is one more step in between. The usual flow is to go through a validation stage that fixes business rules, exception handling and system integration before the move to operation.

Send us your BMT design form and item definition sample. We only need the list of documents.

Contact us