What a confidence score is: a complete guide to the value that separates automatic processing from human review
A confidence score is a value that expresses, case by case, how likely a result extracted or judged by AI is to be correct, and it is used as the criterion for dividing cases that can be processed automatically from cases a person should check.
Why a confidence score is needed
No one could re-check every result.
Even when AI processes documents, a chance remains that a result is wrong. So in the early stages staff often re-check every result, and when that happens the review time stays the same even after AI has been adopted. If nothing is checked at all, on the other hand, wrong values flow straight into business systems.
What is needed is a criterion that decides, case by case, which results can be trusted and passed on and which a person needs to look at. Accuracy tells you the overall average, but it does not tell you whether the single case that has just arrived is correct.
A confidence score makes that distinction possible because it is attached to each individual result. Cases scoring above the threshold pass through automatically, and only those below it are sent to the review screen, which reduces the volume of review.
In organisations with limited review staff, this difference translates directly into processing capacity. Instead of the same people skimming everything, they spend their time on uncertain cases, so fewer errors are missed and throughput rises. How far to raise the share of automatic processing can also be decided from the actual error rate in each score band rather than by instinct.
What a confidence score actually is: how it differs from accuracy
Accuracy and confidence scores across three axes
First, when it is calculated differs. Accuracy is compiled after the fact by comparing results with answers prepared by people. A confidence score is calculated at the moment a result is produced, during operation when no correct answer exists, so it can be used immediately before the result is passed to the next step.
Second, the unit differs. Accuracy is an average across a batch of documents or a whole period. A confidence score is attached to a single document page or a single field. Even when the average is high, errors can cluster on a particular form or field, and field-level scores reveal where.
Third, the purpose differs. Accuracy is used to compare models or versions and to decide whether to adopt. A confidence score is used during operation to decide whether an individual case should be processed automatically, sent for review or separated out as an exception.
The two values check each other. A score can be used as a criterion only once ground truth data confirms that cases with high confidence scores were in fact correct at a high rate. A score that has not been checked is closer to a guess dressed up as a number.
Four conditions for using confidence scores in operation
Four conditions for trusting and using the score
First, calculate it at field level. With only a single score for the whole document, there is no way to tell which field is uncertain, and the entire document ends up being reviewed again.
Second, align it with the actual rate of correct answers. Cases shown with a high score must also be correct at a high rate in reality. If scores and actual accuracy diverge, setting a threshold has no meaning.
Third, set a threshold for each field. Fields where an error has a large impact, such as amounts and account numbers, get a high threshold, while fields used only for reference get a lower one.
Fourth, verify afterwards. Draw a sample from the cases processed automatically and check whether they were actually correct. This is the only way to learn how many errors occur in cases no person looked at.
If any one of the four conditions is missing, the score remains reference information. Skipping the second and fourth in particular means that even with a threshold in place, you cannot explain whether that threshold is right.
How confidence scores are applied in practice
Set thresholds from review history
No one can know the right threshold from the start. In the early stages, review every result while recording how many actual errors occur in each score band, and move bands with almost no errors to automatic processing first.
Each time the threshold is raised or lowered, look at how review volume and the number of missed errors change together. Putting the two figures in one table makes it easier to agree the scope of automation with the business department.
Show the score together with its evidence
If the review screen shows only the score, staff have to search the original again. Highlighting the location in the original where the value was read and showing it alongside reduces checking time.
If only the fields that fell below the threshold are marked, rather than the whole document, staff need to check just those cells and move on. Records of values corrected by staff are kept as material for the next threshold adjustment and for improving the model.
Watch for changes in the score distribution
If the share of low-scoring cases rises during operation, it may be a sign that the documents have changed. Looking at the score distribution regularly by form and by intake channel makes it possible to notice data drift early.
After the model or extraction criteria change, the score distribution changes too, so check the thresholds again. Applying a new version's scores to the old thresholds unchanged can shift the share of automatic processing in unintended ways.
Confidence scores in the Korean environment
In Korean finance and the public sector, audit requirements often call for the grounds of a decision to be recorded even for cases processed automatically. The less a case was checked by a person, the more important it is to be able to show the basis for that decision later. Recording the confidence score, the threshold and whether the case was processed automatically in the processing history makes it possible to explain why no person looked at it.
Difficult conditions that often appear in Korean documents, such as handwritten Hangul or fields overlapped by seals, tend to produce low scores, so thresholds for those fields are adjusted separately.
Frequently asked questions
The two terms often refer to the same concept. Both are values that express how likely a result is to be correct.
No. The score only expresses a likelihood, so the actual rate of correct answers must be checked continuously through sample verification.
When document composition or the model changes, the score distribution changes too, so check it again periodically.
It becomes safer, but more cases go to review and the benefit of automation shrinks. Setting different thresholds according to each field's impact strikes a better balance.
They can, but the calculation method differs, so they may be less precise than for extraction results. It is safer to include a step that checks the evidence location as well.
By checking the score distribution and the errors in each band on an evaluation set before adoption, you can estimate roughly what share will be processed automatically at a given threshold. If operational documents differ from the evaluation set, the actual share will differ too, so confirm it again with early operational results.
Use rule-based checks such as cross-document comparison and format validation as the criterion, but switching to a setup that provides scores lets you control review volume more precisely.