Backtesting an inspection: proving it works before it ships

The default way to find out whether an inspection works is to install it and watch. That is an expensive experiment, because the thing you are testing for is a defect reaching a customer, and you only learn the answer once it has.

Backtesting is the alternative. You assemble images whose true condition you already know, run the complete recipe over every one of them offline, and get your escape and overkill counts before the station gates a single real part. It is not a novel idea. It is what every other gauge on the floor goes through before it is trusted, and there is no reason an AI inspection should be exempt.

What a backtest actually does

The mechanics are simple enough to describe in a sentence: take a set of captures, label what each one truly is, run the live recipe across all of them, and compare the recipe's verdict against the label.

Two details make the difference between a real test and a demo. The first is that it runs the whole recipe, every model and every rule, not one model in isolation. A station usually fails at the seams: two models that each behave correctly and a verdict rule that combines them wrongly. Testing the models separately never finds that.

The second is that the output is a confusion matrix you can click into. A number telling you there were four escapes is a start. Being able to open those four images and see what they had in common is what lets you fix the cause instead of nudging a threshold until the number improves.

Building a test set that does not flatter you

This is where most validation goes wrong, and it goes wrong in a predictable direction. Left to themselves, people assemble obvious passes and obvious failures, because those are the parts that are easy to label confidently. The resulting numbers look excellent and do not survive contact with production.

A useful sanity check on any test set: if the recipe scores perfectly on the first run, the set is probably too easy rather than the recipe being perfect. Genuinely marginal parts produce a few disagreements, and those are the interesting ones.

Repeatability, and why quality teams ask for it

Accuracy on a single pass is not the same as trustworthiness. A station can be right on average and still be unusable if it gives different answers to the same part on different runs, because disposition depends on the answer you happened to get.

This is what a gauge repeatability and reproducibility study measures, and it is already routine for a caliper, a bore gauge or a coordinate measuring machine. Running it on a vision inspection means passing the same parts through more than once and comparing the verdicts. Where the results diverge, you have found either genuine borderline parts sitting exactly on the decision boundary, or a source of variation in the station itself: lighting that is not as fixed as everyone assumed, a part that seats differently, an exposure that drifts.

It is also a fair question to put to a vendor. Asking for gauge R&R on a vision system tells you quickly whether they have deployed into regulated manufacturing or whether they are going to send you a single accuracy percentage and hope.

Yield by station separates two very different problems

Once a station is live, the most useful view is not the overall pass rate. It is the pass rate broken out by camera, line or machine, because the shape of that breakdown tells you what kind of problem you have.

An outlier station like that is almost never a model problem, because every station is running the same model. It is a physical difference: a fixture with more play, a light that has drifted or been knocked, a viewpoint where the part presents differently, or a genuine process difference at that position. The aggregate number would have hidden it completely, and averaging across a fleet is how a single bad station stays invisible for months.

The inverse case is worth naming too. If every station drops together, that is not a fixturing issue, it is either the process or the model. Same data, two very different investigations, and the breakdown is what tells you which one you are in.

A validation sequence you can put in a spec

1. Agree the defect set. Both parties sign off which parts are defective, including the marginal ones. This is the step that prevents the argument later.
2. Build the test set from real production, spanning lots and shifts, with ground truth labelled by agreement.
3. Run the full recipe offline and record escapes and overkill as counts, not as an accuracy percentage.
4. Open the failures. Every escape and every overkill gets looked at. Patterns in them are the actual output of the exercise.
5. Run it twice and compare, so repeatability is a measured number rather than an assumption.
6. Set sensitivity by cost, using the escape and overkill counts against what each one costs you. This is a business decision, not a technical preference.
7. Keep the test set with the recipe and re-run it as an acceptance step after any change, including a retrain.

Two things about that list are worth emphasising. It contains no accuracy figure anywhere, deliberately, for the reasons covered in our piece on escape versus overkill. And step seven is the one that decays first: a test set that is not re-run after retraining is how a station that passed validation in March quietly stops working in June.

None of this is specific to AI, which is rather the point. It is the same discipline a quality team already applies to every other measurement device, applied to a newer one. Where it fits into the wider station design is covered in our guide to machine vision systems, and if you are validating an anomaly detection station in particular, the marginal parts matter even more than usual, because the trip point is a single sensitivity setting rather than a learned boundary.