The reality gap
Sim-to-real transfer
Training on generated data is easy. Working on the real line is the hard part.
Sim-to-real, or sim2real, is the gap between a model that performs in simulation and one that performs on your parts. For visual inspection that gap is much narrower than it is in robotics, and this is why, plus what closes the rest of it.
What is sim-to-real transfer?
Sim-to-real transfer is getting a model trained on generated data to work on real inputs. The distance between the two is the domain gap, sometimes the reality gap. It is the central problem in using synthetic training data, because a model that learns the simulation rather than the task looks excellent in testing and fails on the line.
The useful asymmetry
Why inspection is the easy case
Sim-to-real has a reputation for difficulty that comes from robotics, and it is deserved there. Inspection is a much narrower problem, and the narrowness is the whole advantage.
| Has to generalise over | Robotics policy | Inspection model |
|---|---|---|
| Viewpoint | Changes constantly as the robot moves | Fixed. The camera is bolted down |
| Lighting | Whatever the room happens to be doing | Controlled and constant by design |
| Distance and scale | Varies with every approach | Fixed working distance |
| Contact physics | Friction, compliance, slip, all critical | Not involved at all |
| Environment | Changes while the agent acts in it | One product, one station |
| Consequence of error | Compounds over a whole trajectory | One wrong verdict on one part |
Nearly every axis a robot must generalise over is held constant on an inspection station. That leaves the defect itself as the thing that varies, which is exactly the thing worth generating.
Diagnosis
Where the domain gap actually comes from
Appearance
Rendered surfaces that are too clean, too uniform, or lit in a way no real lamp produces. A model trained on them learns a look rather than a defect.
Sensor characteristics
Real images carry the noise, compression, and colour response of a specific sensor. Synthetic images that lack them are detectably different at the pixel level.
Optics
Real lenses have distortion, vignetting, and a finite depth of field. A perfect pinhole render does not, and the difference concentrates at the edges of the frame.
Defect physics
The widest gap of all. A defect pasted onto a surface does not obey it. A real dent changes how the surface reflects; a scratch cuts through a coating to a different material underneath.
Method
Six ways to close the gap
In order of how much they matter. The first one does most of the work, because a gap that never opens needs no closing.
Start from real images
Photograph the actual part on the actual camera. Keep the surface, the finish, the lighting, and the sensor response genuinely real, and generate only the defect. Most of the domain gap never opens, because most of the image was never synthetic.
Respect the physics
A generated defect has to behave like the material it sits on. A dent on brushed metal has to distort the grain and move the highlight; a stain on fabric has to follow the weave. Pasting a defect texture is what produces a model that fails on the line.
Match the resolution
Generate at the pixel size the camera will actually see. A defect rendered at a higher resolution than production teaches the model detail it will never get, and the model then quietly misses the same defect at the resolution it really runs at.
Randomise what should not matter
Vary position, rotation, lighting angle, and exposure across the generated set, within the range the station will really see. Enough that the model learns the defect, not so much that it dilutes accuracy on the conditions that actually occur.
Validate on real parts
Hold back real images and test against them. This is the step that tells you whether the transfer worked, and it is the one most often skipped because the synthetic set is large and the accuracy on it looks excellent.
Close the loop in production
Feed the real defects the line produces back into training as they appear. Synthetic data is how you go live in days. Real data is how the model stays accurate once you are.
Steps one to four are what the OV Auto-Defect Creator Studio does by construction. Step six is what Haystack is for.
Manufacturers running Overview AI in production
Manufacturers running Overview AI in production
FAQ
Frequently asked questions
What does sim-to-real mean?
Sim-to-real, often written sim2real, is the problem of getting a model trained on generated or simulated data to work on real inputs. Training in simulation is attractive because you control everything and can produce unlimited labelled examples. The catch is that a model can learn the simulation instead of the task, and then fail the moment it meets a real camera. The distance between the two is called the domain gap or the reality gap, and sim-to-real is the work of closing it.
Why is sim-to-real easier for inspection than for robotics?
Because the thing being simulated is far narrower. A robotics policy has to transfer across contact physics, friction, latency, and an environment that changes while the robot acts in it. A visual inspection model has to transfer across one image of one product, photographed by a camera that is bolted in place, under lighting that does not change, at a fixed working distance. Almost everything a robot has to generalise over is held constant on an inspection station. That is why synthetic data reached production in manufacturing inspection well before it did in general robotics.
How do you avoid the model learning the synthetic images instead of the defect?
By keeping as much of the image real as possible. The strongest approach starts from photographs of your actual parts, taken on the actual camera that will run the inspection, and renders only the defect onto that real surface. The background, the finish, the lighting, the optics, and the sensor noise are all genuinely yours, so there is very little synthetic signal for the model to latch onto. Generating the whole scene from nothing is where the domain gap gets wide, and it is the step worth avoiding.
Does sim-to-real remove the need for real defect images?
It removes the need to wait for them, which is the part that blocks a deployment. It does not remove the need to validate against them. The honest pattern is a loop: generate defects to get a working model into production in days, then feed the real defects the line produces back into training as they appear. A set with some real defects and a larger body of synthetic variations usually outperforms either alone. In regulated environments, validation against real samples is generally required regardless of how the model was trained.
What is domain randomization?
Deliberately varying the things you do not want the model to depend on: lighting angle and intensity, part position and rotation, exposure, background, minor colour shifts. The idea is that if the model sees the defect under many irrelevant conditions, it learns the defect rather than the conditions. It is one of the main tools for closing the reality gap. For a fixed inspection station you need less of it than a robotics project would, because the real conditions genuinely are narrow, and over-randomising can cost you accuracy on the conditions that actually occur.
How do you tell whether the transfer worked?
Test on real images the model never trained on, and look at where it is uncertain, not only at an accuracy number. A model with a transfer problem usually shows it as low confidence clustered on a particular condition, one cavity, one shift, one lighting state, and not as uniformly worse performance. That is why an uncertainty view over production images is more useful than a single score: it tells you which slice of reality the training set failed to cover.
Related
Physical AI
The wider shift to AI that perceives and acts in the real world, and where inspection sits in it.
Read more →OV Auto-Defect Creator Studio
Physics-aware synthetic defects generated on images of your own parts, annotated as they render.
Read more →What Is Machine Vision
The field these models are deployed into, and the technology inside an inspection station.
Read more →Related Articles
World Models and Synthetic Data
Why the factories with the highest standards have the least defect data.
Read More →Synthetic Defect Data for High-Mix Lines
The arithmetic of waiting for real defects, and the honest limits of generating them.
Read More →What Is Physical AI?
World models, the perceive-predict-act loop, and why data became the bottleneck.
Read More →Test the transfer on your own parts
Describe the part and the defect you need to catch. You get the optics and an honest answer on whether it is solvable before anyone talks to you about hardware.