Work and processes
Human-in-the-Loop Explained: Designing to Narrow the Cases People Review, Complete Guide
Korea Deep LearningPublished
Human-in-the-Loop means designing so that, among the results AI processes, only items with low confidence or items that trip a rule are checked by people. It is a way to manage operational risk between reviewing everything and reviewing nothing.
Why review design determines the success of automation
It was either all reviewed or nothing reviewed.
When introducing document automation, how to handle review is often decided later. The result settles into one of two states: out of anxiety, checking everything, or trusting it and checking nothing.
The former erases the effect of automation. The latter means errors stay hidden until they lead to a business incident. What is actually needed is the space between. It is a design that distinguishes which cases are safe and which are risky, and sends only the risky ones to people. This is why review must be treated as a structure to design from the start, not a function to attach later.
The exact definition of Human-in-the-Loop: how it differs from full review
Three axes of difference between full review and selective review
First, the target differs. Full review looks at every case. Selective review looks only at cases that contain an item below the baseline. Picking out items rather than cases is the key.
Second, the goal differs. The goal of full review is not to miss any error. The goal of selective review is to narrow the items to check while not missing errors. The moment this second condition is attached, design becomes necessary.
Third, the performance indicators differ. Full review looks at the number of errors found. Selective review looks at both the proportion sent to review and the proportion of those actually corrected. If most of the cases sent on pass without correction, the baseline is set too high.
The two methods are not substitutes. In the early stage of adoption, it is safer to start with full review to collect data, and then move to selective review based on that data.
Five elements needed for review design
Five elements used in real operations
First, item-level confidence. If there is only one score for the whole document, there is no way to tell which value to look at.
Second, display of original locations. When a value is selected, the corresponding part of the original should appear, so that the reviewer does not read the document from the beginning.
Third, validation rule results. The reason a case was sent to review should be shown on screen. Where to check depends on whether it was low confidence or a total that did not match.
Fourth, correction history. Who corrected what, and how, must be recorded, so the team can respond to audits and use it as a basis for adjusting the baseline.
Fifth, a return path. The corrected value must flow back through the process and be registered in the system. Without this path, reviewers end up entering results separately.
Practical application of Human-in-the-Loop
Set the baseline from data
There is no way to set the baseline exactly right from the start. Set it conservatively at first and adjust it by collecting actual review results.
Items that are almost never corrected can have the baseline lowered and be passed to automatic processing. Items that are corrected often get a raised baseline or added validation rules. Repeating this adjustment gradually narrows the review targets.
Shorten the reviewer's movement
The design of the review screen greatly determines actual processing time. Highlighting only the problem items and showing their location in the original shortens the confirmation time per case.
Placing a screen that manages extraction items and schemas per document type alongside it also lets reviewers adjust the criteria directly when they find items that are repeatedly corrected.
Human-in-the-Loop in a domestic environment
In Korean finance and the public sector, review history is subject to audit. Automatically processed cases and cases corrected by people must be recorded separately, and the values before and after correction must be kept together.
Separation of authority is also a requirement. Some work requires dividing review authority from approval authority, and documents containing personal information often have restricted access. Separating the review environment from the production environment makes it easier to meet these requirements.
The review screen determines the savings
A design in which only low-confidence cases are seen by people holds up only when the review screen supports it. If cases were separated as exceptions but the staff member still reads the whole document from the beginning, the savings are not large.
The screen needs three things. First, it should show why the case became an exception. Which point to check depends on whether confidence was low or whether the cross-check across documents found a mismatch. Second, the value must be linked to the original, so that selecting an item shows its location and reduces the time spent confirming by eye. Third, the corrections made must be kept.
As correction history accumulates, it becomes the basis for improvement. If the same item on the same form is corrected repeatedly, that is a sign it is time to adjust the extraction criteria or validation rules.
Frequently asked questions
What review ratio is appropriate?
It depends on the work. There are cases in payment review and credit review where a review proportion was set and operated. The greater the impact of an error, the more conservatively it should be set.
Can we start with selective review from the beginning?
Starting with full review and collecting data is safer. Without a basis for setting the baseline, adjustments end up depending on intuition.
How are corrections made by reviewers used?
They are used as the basis for adjusting the baseline and improving extraction criteria. If corrections repeat on the same item, treat it as a signal to add a rule at that point.
How much review history should be kept?
It follows the audit requirements of each business. The usual setup separates automatic processing from manual correction and keeps the values before and after correction.
How is the review target decided?
Divide it into cases where value confidence is below the standard, cases where cross-document matching found a mismatch, and cases that business rules require human judgment, and set the target by these categories.
How do we align criteria when there are several reviewers?
Gather the items where results diverge when two people review the same document, and document the judgment criteria. Without these criteria, it is hard to use the review history as a basis for changing the standards or for additional training.
Tell us your current review process, and we will review with you a design that switches to selective review.
Contact us



