What LLMOps is: a complete guide to the operating framework for using large language models reliably at work

LLMOps is an operating framework that, after large language models are deployed to business work, manages prompts, model versions, retrieval data, answer quality and cost together to keep results at a consistent level.

Why LLMOps is needed

The same question received a different answer.

Large language models produce impressive results at the demonstration stage. Yet a few weeks after deployment, a question that worked well last week gets an off-target answer, or the answer format changes and downstream systems can no longer read the result. Costs rising faster than expected is also common.

The causes can lie in several places. Someone may have edited the prompt wording, the model provider may have updated the version, or the documents being searched may have changed. Without a record of what changed and when, finding the cause alone takes days.

With conventional software, the same code gives the same result. The results of a large language model vary with text, data and external models that sit outside the code, so a framework that manages those elements separately is needed.

This need grows as large language models are used not only to generate answers but to make and carry out business decisions. A slightly different answer is merely an inconvenience in a reference chatbot, but in work such as review or registration it leads to incorrect processing.

What LLMOps actually is: how it differs from MLOps

MLOps and LLMOps across three axes

First, what is managed differs. MLOps centres on models the organisation has trained itself and their training data. With LLMOps, the model is often not trained in-house, so prompts, retrieval data, model choice and call settings become the main things to manage.

Second, where change arises differs. In MLOps, the party changing the model is inside the organisation. In LLMOps, changes the organisation cannot control occur, such as an external provider updating a model or ending support for an earlier version.

Third, how evaluation works differs. MLOps evaluation centres on metrics that can be compared directly with a correct answer, such as classification accuracy or numerical error. LLMOps deals with answers written as sentences, so several criteria must be considered together, including whether the answer matches the source documents, follows the format and avoids prohibited expressions.

The two are not substitutes. LLMOps applies the MLOps principles of version control, pre-deployment validation and operational monitoring to the characteristics of large language models.

Four things LLMOps manages

Four things to keep as versions during operation

First, prompts. Instructions, examples and output format definitions are versioned like code, with a record of who changed them and why, because editing a single sentence can change results significantly.

Second, model versions and providers. Record which version of which model was called with which settings, and track providers' schedules for retiring versions.

Third, retrieval data. The document sets that ground answers, the chunking criteria and the indexing date must be managed so that, when answers change, you can tell whether the data is the cause.

Fourth, evaluation and cost. Accumulate scores on a fixed question set, cost per call and response time so they can be compared across versions.

If the record for any one of the four is missing, the cause cannot be narrowed down when results change. It is good practice to bundle the combination of versions of all four into a single deployment unit.

How LLMOps is applied in practice

Build an evaluation question set first

Before going into operation, build an evaluation set that bundles questions likely to arise in real work with expected answers and source documents. In a tax-sector case, the RAG evaluation set was built by fixing the documents in scope and gathering the questions staff actually ask.

Define the scoring criteria at the same time. For each question, it must be possible to judge whether the answer matches the source documents, whether any required item is missing and whether the specified format was followed.

Only with this set can before and after be compared on the same criteria each time a prompt or model changes. Failure cases newly found in operation are added to the evaluation set to confirm that the same problem does not recur.

Apply one change at a time

If prompts, models and retrieval data are changed at once, there is no way to know what caused results to improve or worsen. Change one element at a time, validate it on the evaluation set and put it into operation after approval. Keep the previous version so that you can roll back to the last combination immediately if a problem arises.

Log calls and break down costs

During operation, record the input, output, version used, response time and cost of each call. Breaking costs down by task and department shows where costs are growing and identifies work for which a lighter model is sufficient. Call logs also serve as the basis for reproducing the input and version at the time when a user reports a strange answer.

LLMOps in the Korean environment

In Korean finance and the public sector, network separation rules often make external model APIs hard to use, so setups that also run models installed internally are common. In that case, model replacement and performance comparison must be possible in the internal environment.

When using external models, mask personal information at the input stage so that it does not enter prompts, and decide where call logs are stored and how long they are kept. Actual performance can only be judged with an evaluation set built from Korean business documents.

Frequently asked questions