What is a VLM: a complete guide to understanding documents without coordinates

A VLM (Vision Language Model) processes a document image and language inside a single model, interpreting not only the characters but the layout they sit in and their relationship to surrounding fields, arriving at meaning in a way closer to how a person reads a document.

Where VLMs came from

Recognition used to stop the moment a form changed.

Document recognition ran on coordinates for a long time. You specified in advance where the name sits on an application and where the amount sits, then read characters from those positions. On fixed forms it worked well. The difficulty was that real business documents are not fixed. The same corporate registry certificate orders and positions its entries differently depending on issue date and recorded items; the same application form is laid out differently by branch and by year. Under a coordinate approach, every additional form meant another round of labelling and training, and in financial operations with dozens of document types that cost blocked adoption outright.

A VLM approaches this differently. Instead of specifying where to read by coordinate, you say in language what to find, and the model looks across the whole document for it. A single request to extract the submission deadline and the funding amount can pull those values out of a public notice.

What a VLM actually is: how it differs from rule-based OCR

Rule-based OCR and VLM across three axes

First, the target differs. Rule-based OCR works on the pixels inside a specified region. A VLM works on the visual information across the whole page together with the language written on it. It can therefore connect a field label to a value even when they sit apart, and can judge whether a checkbox inside a table is selected by reading the surrounding context.

Second, the objective differs. Rule-based OCR aims to transcribe the characters at a given position accurately. A VLM aims to work out what the document is saying and return the information asked for. The first starts knowing the position and not the value; the second starts knowing the name of the value and not its position.

Third, the metric differs. Character-level accuracy was enough to judge rule-based OCR. A VLM has to be judged on field-level value accuracy as well, and on how far performance holds when a form changes. In an evaluation drawing 1,000 pages at random from 900,000 public administrative documents, key recognition accuracy separated at 97.3 percent against 80.4 percent, while value recognition accuracy separated at 96.0 percent against 48.7 percent, nearly a factor of two. The technology gap shows up far more in retrieving the value bound to a field name than in finding the field name itself.

The two are not alternatives. Where a form is entirely fixed and volume is high, the rule-based approach remains faster and cheaper. Placing a VLM on the segments where document types are many and forms change often is the practical arrangement.

Five requirements for putting a VLM into production

Five requirements that hold in real operations

First, it has to be specialised for documents. A general-purpose vision language model is strong at describing photographs and charts, but what business documents demand is not description, it is values. Stamps, strikethroughs, overlapping characters, and fax headers stacked four deep do not appear in general imagery, and a model has to have learned them before it performs on real intake documents.

Second, it must not invent values. The structural weakness of a VLM is the tendency to fill a blank with the most plausible value when it is not confident. In document work that tendency becomes an incident directly. The control that leaves low-confidence fields unconfirmed and routes them for review has to sit inside the model's behaviour.

Third, it has to return the position of each value. Understanding without coordinates does not mean returning no coordinates. Coordinate specification is unnecessary at input; at output, where in the source a value came from still has to be shown, or review is impossible.

Fourth, it has to interpret tables and structure alongside text. Recognising characters while losing the layout breaks the binding between a field and its value. That is why scores on structure restoration diverge sharply even among the leading models in international document parsing evaluations.

Fifth, control requires owning the model. Calling an external model sends the original document outside, and performance shifts without notice whenever that model is updated. In finance and the public sector, either condition alone is enough to block adoption.

How a VLM is applied in practice

Design the extraction criteria as written instructions

In VLM-based document processing, the first deliverable is not training data but an extraction criteria document. Find the invoice date near the top of the document and normalise it to year, month, day. Take the total billed amount from the sum in the final row of the table, stripping currency symbols and separators. Treat contract number, policy number, and contract no. as the same field. Instructions of that kind are written out field by field.

The advantage is a low cost of response when forms change. New fields or altered notation are handled by editing the criteria rather than retraining the model. Because the criteria document effectively is the performance, decide early who owns it and how versions are recorded.

Confirm the inference environment first

A VLM carries more computation than a rule-based approach. Production deployments show roughly 0.5 seconds per page on A100-class accelerators, with batch throughput around 1,800 pages per hour; 100 pages in 3.5 minutes and 500 pages in 17 minutes have both been recorded. When evaluating, size the accelerator against monthly volume together with peak-hour concentration.

Decide the deployment shape at the same time. Container-based installation inside the internal network is the common arrangement, and separating inference, post-processing, and result storage into layers makes the later integration with business systems easier.

Place it alongside, not instead of, rule-based segments

Not every segment needs a vision language model. Where the form is entirely fixed and volume is high, a routine report arriving daily in the same layout for instance, rule-based processing is faster and cheaper.

The arrangement used most often in practice splits front from back. The front, where an arriving document's type is unknown and layouts shift between forms, is handled by the vision language model. The back, where extracted values are entered through fixed screens and procedures, is handled by an automation tool. Computation then concentrates only where it is needed, and the stability of the rule-fixed segment is preserved.

The dividing criterion is volatility of form. Counting how many times the forms changed over the past year, and how often new ones are added, usually settles where the model belongs.

VLMs in the Korean environment

The first condition encountered in Korean deployments is network separation. Outbound API calls from the business network are blocked, so whether the model can be installed and run internally becomes the starting point of technical review. Configurations that call a large overseas model by API are frequently excluded at this stage.

The documents themselves differ too. Korean word processor files, government forms, and low-quality scans received by fax are the actual processing targets, with stamps and handwriting on top. This is an area where a model trained and validated repeatedly on domestic documents holds an advantage.

Multilingual expansion is worth considering as well. In processing credit review documents for an overseas subsidiary, accuracy reached 95 percent on printed local-language text including tonal marks and 92 percent on local-language handwriting. Performance varies by language, so validation on real documents in the target language is necessary.

Frequently asked questions