Escape vs overkill: measuring a vision system honestly

Every vision proposal quotes an accuracy figure, and the figure is almost always useless. On a line running at 98% yield, a system that passes every single part without looking at it scores 98% accurate. It has caught nothing. The number is true and it means nothing.

Quality engineering solved this vocabulary problem decades before anyone put a neural network on a factory floor. The two words that matter are escape and overkill, and they cost completely different amounts of money.

The two errors, and why they are not symmetric

Both are errors. They are not the same error. An escape on a brake component and an overkill on a plastic clip are separated by several orders of magnitude, and any single accuracy number averages that distinction away completely. This is the whole reason the two words exist.

It is also why the sensitivity setting on an inspection station is a business decision rather than a technical one. Tightening it trades escapes for overkill. Loosening it trades the other way. There is no setting that eliminates both, and the correct position depends on what each costs you, which is a conversation for the quality and finance teams rather than for whoever is holding the mouse.

Why accuracy misleads, with numbers

Take a line producing 10,000 parts with a 2% true defect rate, so 200 bad parts and 9,800 good ones. Two systems:

System A has the better accuracy score and is worthless. System B looks worse on the headline number and is the one you want, because it converted 190 potential customer complaints into 230 scrapped parts. Whether that is a good trade is arithmetic you can actually do: 190 times the cost of an escape against 230 times the cost of a part. Accuracy cannot express that question, let alone answer it.

The four numbers worth asking for

If you are evaluating an inspection system, these are the figures to insist on, and all four should come from your parts rather than from a datasheet.

The third one deserves emphasis because it is the one most often skipped. Gauge repeatability and reproducibility is standard practice for a caliper or a CMM, and there is no reason an automated inspection should be exempt. Asking a vendor for gauge R&R on their vision system is a fair question, and the answer tells you a lot about whether they have deployed into regulated manufacturing before.

Measure it before you go live, not after

The traditional way to find out whether an inspection works is to run it in production and see what happens. That is an expensive experiment, because the failure mode you are testing for is a defect reaching a customer.

The alternative is backtesting: assemble a set of images with known ground truth, run the complete inspection recipe over every one of them offline, and get the escape and overkill counts before the station ever gates a real part. On the OV80i this runs on the camera itself and reports in exactly this vocabulary, escapes and overkill rather than false positives and false negatives, with a confusion matrix you can click into to see which images went wrong.

Two details make the difference between a backtest that helps and one that flatters you. First, the test set has to include marginal parts, because a set of obvious passes and obvious failures will report a number nobody can reproduce in production. Second, the test set has to be kept and re-run after every change to the recipe, otherwise you are tuning for one number while quietly breaking another. A test set that travels between cameras as a file is what makes that practical across a fleet.

What good looks like in a proposal

A useful specification for an inspection station reads something like this, and notice that it contains no accuracy figure at all.

1. Zero escapes on the agreed defect set of N parts, including the marginal examples both parties signed off.
2. Overkill below an agreed percentage of production, because this is the number that determines whether operators keep the station switched on.
3. Repeatability demonstrated across at least two runs of the same parts.
4. The test set retained, version controlled with the recipe, and re-run as an acceptance step after any change.

The second point is the one people underestimate. Overkill is not merely a scrap cost. It is the mechanism by which good inspection systems get bypassed: if the station rejects parts an operator can see are fine, the station loses credibility, and within a month someone is waving parts through. A system with slightly more escapes that operators trust often outperforms a stricter one they have learned to override.

None of this is specific to AI. It applies equally to a rule-based tool, a human inspector, or a go/no-go gauge, and that is rather the point: the arrival of learned models does not require new vocabulary, it requires using the vocabulary quality engineering already had. If you want the wider context on how these stations get built, our guide to machine vision systems covers the whole picture, and the piece on anomaly detection covers the approach where getting the escape and overkill balance right matters most.