View all glossary terms

Technology and architecture

What Is a VLM: A Complete Guide to Vision-Language Models That Understand Documents Without Coordinates

Korea Deep LearningPublished

A VLM (Vision Language Model) is a model that processes document images and language together inside a single model. It interprets not only the characters but also the layout they sit in and their relationship to neighboring items, grasping meaning in a way close to how a person reads a document.

Why VLMs Emerged

When the form changed, recognition stopped.

Document recognition technology long worked on the basis of coordinates. You would specify in advance where the name is on an application and where the amount is, then read the characters at those positions. This worked well for documents with fixed forms. The problem was that real work documents are not fixed. The same registry certificate can have items in a different order and position depending on when it was issued and what it records, and the same application can be laid out differently by branch and year. With the coordinate method, every added form needed more labeling and training. In financial work with dozens of document types, that cost blocked adoption itself.

A VLM takes a different approach. Instead of specifying where to read with coordinates, you tell it in language what to look for, and the model examines the whole document and finds that item. With a single request to extract the application deadline and the support amount, the values can be pulled from the announcement.

The Exact Definition of a VLM: How It Differs from Rule-Based OCR

Three Axes That Separate Rule-Based OCR and VLMs

First, the target differs. Rule-based OCR targets the pixels inside a specified region. A VLM targets the visual information of the whole page together with the language written on it. So even when an item name and its value are apart, it can connect the two based on their relationship, and it judges whether a checkbox in a table is selected by looking at the surrounding context.

Second, the goal differs. The goal of rule-based OCR is to transcribe the characters at a specified position accurately. The goal of a VLM is to understand what the document is saying and return the information requested. The former starts from knowing the position but not the value, while the latter starts from knowing the name of the value but not the position.

Third, the performance metrics differ. For rule-based OCR, character-level accuracy was enough. A VLM must be judged on value accuracy per item and on how well performance holds up when the form changes, in addition to character accuracy. In an evaluation that sampled public administrative documents, the gap between methods was much larger for value recognition accuracy than for key recognition accuracy. The technical difference shows most clearly not in finding the item name, but in accurately retrieving the value attached to that name.

The two approaches are not substitutes. In sections where the forms are completely fixed and the volume is high, the rule-based approach is still fast and inexpensive. For document types that are numerous and where forms change often, placing a VLM is the practical combination.

Five Requirements for Putting a VLM to Work

Five Requirements for Running a VLM in Production

First, it must be specialized for documents. General vision-language models are good at describing photos and charts, but what business documents require is not description but values. Elements not found in ordinary images, such as seals, strikethroughs, overlapping handwriting and the four layers of fax headers, must be learned for performance to hold on real received documents.

Second, it must not invent values that do not exist. The structural weakness of a VLM is that when it is not confident, it tends to fill a blank with the most plausible value. In document work, this property becomes an accident. Control that leaves items with low confidence unconfirmed and passes them on for review must be built into the model.

Third, it must return the position of each value. Understanding without coordinates does not mean not returning coordinates. It means no coordinates need to be specified at the input stage. At the output stage, the position in the original where the value came from must still be shown for review to be possible.

Fourth, it must interpret tables and structure together. If it recognizes characters but loses their arrangement, the connection between items and values breaks. This is why, in international document parsing evaluations, the scores on structure restoration items differ widely even among top models.

Fifth, control requires the model to be self-hosted. In a setup that calls an external model, the original documents leave the organization, and performance can change without notice when the model is updated. For finance and the public sector, both conditions are barriers to adoption.

How to Apply a VLM in Practice

Design the Extraction Criteria as Sentences

In VLM-based document processing, the first deliverable of the project is not training data but an extraction specification. For each item, you write instructions such as: find the claim date at the top of the document and format it as year, month and day; take the total claim amount from the sum in the last row of the table and remove currency symbols and commas; treat a contract number labeled as a policy number or contract No. as the same item.

The advantage of this approach is that the cost of responding to a form change is low. When a new item is added or the notation changes, you revise the specification rather than retrain the model. However, because the specification is effectively the performance, decide early on who manages the document, and how its versions are kept.

Confirm the Inference Environment First

A VLM needs more computation than a rule-based approach. In actual operations, per-page processing speed and batch throughput have been confirmed on GPUs. When reviewing adoption, calculate the GPU scale by combining monthly volume with how concentrated the peak hours are.

Decide the deployment form as well. A container-based installation inside the internal network is the common pattern. Separating model inference, post-processing and result storage into layers makes later integration with business systems easier.

The Domestic Environment for VLMs

The first condition that comes up in domestic adoption is network separation. Because external APIs cannot be called from the work network, whether the model can be installed and operated internally is the starting point of the technical review. Configurations that call large overseas models through an API are often excluded from consideration at this point.

The documents themselves also differ. Hangul word processor documents, government forms and low-resolution fax scans received by fax are the actual targets, and seals and handwriting come with them. Models that have been repeatedly trained and validated on domestic documents have an advantage in this area.

Multilingual expansion is also a consideration. In one case that processed credit review documents for an overseas subsidiary, accuracy was confirmed for both printed and handwritten text in local languages, including tonal characters. Performance varies by language, so validation with real documents in the target language is required.

Divide It Alongside Rule-Based Sections

Not every section needs a vision-language model. For sections where the form is completely fixed and the volume is large, such as a structured report that arrives in the same format every day, rule-based processing is faster and cheaper.

A common arrangement in practice is to split the front end from the back end. The front end, where you do not know the form of a received document, and the sections where the layout shifts with each form are handled by the vision-language model. The back end, which enters the extracted values into a fixed screen and procedure, is handled by an automation tool. Splitting it this way concentrates computation where it is needed and keeps the stability of the rule-fixed sections intact.

The basis for splitting sections is the variability of the forms. Counting how many times the forms changed over the past year and how often new forms are added roughly determines where to place the model.

Set the Model Update Procedure Together

A model is not installed once and left alone. When a new document set arrives or an improved version is released, it must be applied. If this process is not controlled, something that worked yesterday can suddenly change today.

A safe setup has three steps. First, put the changed extraction specification or the new model version into a review environment. Next, compare its performance against the existing version using the same evaluation documents. Finally, once the person responsible checks the results and approves, apply it to operations, and if a problem occurs, roll back to the previous version.

When this procedure is in place, the change history becomes an asset. Which form caused which change, and how performance changed as a result, is recorded, so operational knowledge does not scatter when people change roles.

Frequently asked questions

Does a VLM replace OCR?

No. It is more accurate to view a VLM as document understanding layered on top of character recognition that includes character recognition itself. In high-volume sections where the forms are fixed, the existing approach is still efficient.

Do I need to retrain when the form changes?

No. Unlike the coordinate approach, values are found based on item names and context, so the model still works when the layout changes. When a new item appears, you respond by revising the extraction specification.

How do you solve the hallucination problem?

The approach is to control the generation itself. Items with low confidence are not confirmed as values. Instead, the value is returned together with its source position so that review is possible. A case was confirmed at a very low level on validation with PDF documents built from real data.

Is a GPU absolutely required?

Yes. A GPU inference environment is a prerequisite for practical processing speed. When reviewing adoption, calculate the required specification based on monthly volume.

Can handwriting and low-quality documents be processed?

Yes. There is a financial document validation case where high accuracy was confirmed by similarity for both printed and handwritten text. However, for items with large variation in handwriting, it is safer to also set up validation rules.

Should a general-purpose vision-language model be used as is?

It is not recommended. General models perform well on ordinary images, but most problems in the field occur in exceptional situations. A model trained on elements found only in business documents, such as seals, strikethroughs and overlapping fax headers, is what is needed.

Is it worth adopting if there are only a few document types?

If the forms are fixed and the volume is large, rule-based processing can be more efficient. The difference shows up in work where forms change often or new types keep being added.

Can it be adopted in a network-separated environment?

Yes. With an on-premise deployment installed on a server inside the customer's network, it runs without any external communication.

Send us one real work document, and we will return the recognition and extraction results as they are.

Contact us