What VLMOps is: a complete guide to running extraction criteria as an operational asset

VLMOps is the operating framework in which the extraction criteria for a document vision language model are defined in natural language, versioned, evaluated for performance against the same documents, and promoted to production through an approval step.

Why the cost lands in operations rather than adoption

Forms keep changing.

A document automation project performs best on the day the build finishes, because it was made to fit the forms and fields of that moment. The difficulty comes afterwards.

Government forms get revised, business rules change and add fields to extract, and new document types start arriving. If the structure retrains the model each time, every change drags data preparation, training, and validation behind it. After a few rounds the operator starts deferring change requests, and the automation begins to diverge from reality. VLMOps targets exactly that point.

What VLMOps actually is: how it differs from MLOps

MLOps and VLMOps across three axes

First, what is managed differs. A general model operations framework manages training data, model weights, and the training pipeline. VLMOps puts extraction criteria and schemas at the centre of what is managed. What to extract and how is recorded as prose, and that prose becomes the unit of version control.

Second, the response to change differs. The former gathers data and retrains. VLMOps responds by editing the criteria. Adding a field or accommodating a change in notation does not touch model parameters.

Third, who handles it differs. The former is the territory of data scientists and engineers. In VLMOps, the business owner can write in field names, positions, formats, and exceptions directly. An instruction to find the invoice date near the top of the document and normalise it to a set format is itself the configuration.

The two are not alternatives. Additional training is still required where adjusting criteria cannot reach. But as a matter of sequence, trying the criteria first gives the better return.

The four functions that make up VLMOps

Four functions of the operating framework

First, natural language direction. The business owner specifies in prose what to extract, from where, and in what format. An instruction to treat contract number, policy number, and contract no. as the same field belongs here.

Second, version control. A new version is created only when the content of the criteria actually changes, and previous versions are retained rather than overwritten. What was added, which format hint changed, and why it changed are recorded with a note.

Third, evaluation on identical documents. The previous and new versions are compared side by side on the same evaluation set. Overall accuracy, document-level exact match rate, and processing latency are read together.

Fourth, controlled deployment. Results are reviewed, internal approval is obtained, the change is promoted to production, and progress is monitored so a rollback is possible if needed.

How VLMOps is applied in practice

Manage the criteria document as an asset

Extraction criteria are business knowledge, not system configuration. Which field of which form is extracted in which way, and why that was decided, has to be recorded for operations to survive a change of staff.

Record the exception policy alongside the per-field instructions. Rules such as not extracting printed instructional text, or not treating a signature placeholder as a value, belong there.

Use review history as the basis for change

When to change the criteria is a data judgment rather than an instinct. Repeated corrections by reviewers on a particular field mean it is time to revise that field's instruction.

A rising share of cases routed to unclassified or to exceptions signals that new forms are entering the flow. Checking these indicators periodically catches performance quietly degrading before it becomes visible elsewhere.

Decide what is kept as an asset

Building an operating framework comes down to deciding what will be kept in reusable form. Without that definition, configuration scatters every time a project ends.

Four things are treated as assets in practice: which model was used, what instructions set the extraction direction, which fields are extracted in which format, and what threshold counts as passing. Change history is added on top. What changed, when, why, and what effect it had on performance all have to be recorded for a rollback to be possible and for the reasoning to survive a change of staff.

New departments and processes receive only the parts that differ, and feedback from the field flows back into the quality criteria. Accumulated assets also carry into the agent workflow, since the classification scheme, field schemas, and validation rules are used there as they are. The thicker the assets, the shorter the build for each new process.

VLMOps in the Korean environment

Korean public and financial forms are revised frequently, with effective dates fixed by public notice. Because it is known in advance when a new form arrives, criteria can be prepared and evaluated in the review environment before the effective date.

Approval procedure is a requirement as well. Changes promoted to production have to record who reviewed and who approved for audit to be answerable. On top of that comes the condition that the whole process runs inside the internal network.

Frequently asked questions