Computer Vision Quality Control: A Manufacturing Build Guide

A computer vision quality control system is an imaging station (camera, lens, lighting and trigger) feeding a model that decides pass or fail within the line's cycle time, and a PLC that acts on that decision. Most of its accuracy is decided before any model is trained: by how consistently the station images each part, and by how well the defect catalog and labels match the quality team's acceptance criteria. Use anomaly detection when defect examples are scarce and supervised detection or segmentation when you have labeled examples of each defect, then judge the system at the part level, by escape rate and false reject rate, on production runs the model has never seen.

This guide covers the whole system: what to define first, the imaging arithmetic, how to choose and train a model, how to evaluate it honestly, how it connects to the line, and what fails once it is running.

Key takeaways

• Define the defect catalog and acceptance criteria with quality engineers first. The model can only learn the boundary that the labels describe.
• Imaging decides what is detectable. If a defect does not show up clearly and consistently in the image, no model will find it reliably.
• Pick the model family by the data you have. Anomaly detection for scarce defects, supervised detection or segmentation for known, labeled ones, classic rules for dimensions.
• Evaluate parts, not boxes. Escape rate and false reject rate at a chosen threshold, with honest uncertainty when defect samples are few.
• A late or missing result must reject the part. The station fails closed, and every result carries the ID of the part it belongs to.
• Treat the station as a calibrated instrument. Shift-start checks with known parts, monitored image statistics, and change control on models and thresholds.

The system, end to end

Start with the defect catalog, not the model

Before any images are collected, write down what must be caught, with the people who sign off on quality. A catalog entry looks like this (values are placeholders for your own specification):

Two quality practices make the labels trustworthy. First, collect limit samples: physical parts at the boundary between acceptable and not, agreed by quality, and photograph them on the station. They become the reference for labelers and for daily checks. Second, run an attribute agreement analysis on a set of parts: have several inspectors judge the same parts independently and see how often they agree with each other and with the reference. If experienced people disagree on a defect class, the model will learn that disagreement as noise, and its errors on that class are a labeling problem before they are a modeling problem.

Imaging: where most of the accuracy comes from

Resolution

The pixel footprint on the part is the field of view divided by the number of pixels across it. A 100 mm field of view on a sensor 2,000 pixels wide gives 0.05 mm per pixel, so a 0.2 mm defect spans about four pixels. Machine vision practitioners generally want the smallest defect to span several pixels, and more if the model must tell defect types apart, so work the arithmetic backward from the smallest defect in the catalog to the camera and lens.

Motion blur

On a moving line, blur in pixels is line speed times exposure time divided by the pixel footprint. At 0.5 m/s (500 mm/s) with a 100 microsecond exposure, the part moves 0.05 mm during the exposure, which is one pixel at the footprint above. Keep blur at or below about a pixel. Short exposures need a lot of light, which is why inspection stations use strobed LED lighting synchronized with the camera.

Area scan or line scan

Area scan cameras capture a 2D frame and suit discrete parts. Line scan cameras capture one row at a time as material moves past, and suit continuous material (sheet metal, film, textiles, paper) or cylindrical parts rotated under the camera. Line scan needs an encoder to tie row capture to motion, or the image stretches and compresses as speed varies.

Lighting

Lighting is often more important than the camera. The right technique makes a defect stand out; the wrong one hides it.

Enclose the station to block ambient light. An opened door or a skylight changes the image more than most defects do.

Presentation and trigger

Fixture parts so they arrive in the same position and orientation, within a known tolerance. Trigger from a part sensor or an encoder count, not from a timer, so changes in conveyor speed do not shift where the part sits in the frame.

Choosing the model approach

Stations often combine them: rules for dimensions, an anomaly model as a safety net for unknown defects, and a supervised model for the critical defects in the catalog.

Collect data where the model will run

• Same station, same optics, same lighting. Images from a lab bench or a phone do not transfer. Build the station first, then collect.
• Cover the variation. Shifts, material lots, suppliers, product variants, positions on the part, and the normal range of surface finish.
• Keep the good parts. Anomaly models train on them, and false reject rate can only be measured on them.
• Write a labeling guide with photos of limit samples for every defect class, and double-label a subset to measure agreement.
• Split by production run or time, never at random. Consecutive images from the same batch are near-duplicates. A random split puts them on both sides and inflates every metric.
• Ignore accuracy. On a line where most parts are good, a model that passes everything scores high accuracy and catches nothing.

Training a defect detector

A fine-tuned detector is a reasonable baseline when you have labeled boxes. The part that deserves attention is augmentation: it should reproduce real variation on the station, and nothing else.

import torch
from torchvision import tv_tensors
from torchvision.models.detection import (
    FasterRCNN_ResNet50_FPN_V2_Weights,
    fasterrcnn_resnet50_fpn_v2,
)
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.transforms import v2 as T
 
CLASSES = ["background", "scratch", "dent", "stain", "crack"]
 
model = fasterrcnn_resnet50_fpn_v2(weights=FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, len(CLASSES))
 
train_tf = T.Compose([
    T.ToImage(),
    T.RandomHorizontalFlip(0.5),                          # only if the part is symmetric
    T.RandomAffine(degrees=3, translate=(0.02, 0.02)),    # fixture tolerance, not arbitrary rotation
    T.ColorJitter(brightness=0.1),                        # LED drift; no hue jitter, stains are color defects
    T.ToDtype(torch.float32, scale=True),
    T.SanitizeBoundingBoxes(),
])
 
 
def make_target(boxes_xyxy, labels, height, width):
    return {
        "boxes": tv_tensors.BoundingBoxes(boxes_xyxy, format="XYXY", canvas_size=(height, width)),
        "labels": torch.as_tensor(labels, dtype=torch.int64),
    }
 
 
optimizer = torch.optim.SGD(model.parameters(), lr=0.005, momentum=0.9, weight_decay=1e-4)
model.train()
for images, targets in train_loader:          # dataset applies train_tf to (image, target) pairs
    loss_dict = model(list(images), list(targets))
    loss = sum(loss_dict.values())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Augmentations copied from general image tutorials can erase the defect signal. Hue and saturation jitter teaches the model that color changes are normal, which is exactly wrong when a stain or discoloration is the defect. Large rotations create part orientations the fixture never produces. Ask of every augmentation: does this variation happen on the station?

Evaluate at the part level

Detection metrics such as mAP are useful while developing a model. The line cares about two numbers per part:

• Escape rate: defective parts passed, divided by defective parts.
• False reject rate: good parts rejected, divided by good parts.

They trade off through the decision threshold. Quality sets the escape requirement; you pick the threshold that meets it and report the false reject rate that results.

import numpy as np
 
 
def part_scores(detections_per_part, critical_classes):
    """Highest confidence of any critical-class detection on each part."""
    return np.array([
        max((d["score"] for d in dets if d["label"] in critical_classes), default=0.0)
        for dets in detections_per_part
    ])
 
 
def threshold_for_escape_target(scores, is_defective, max_escape_rate):
    """Highest threshold whose escape rate on validation parts stays within the target."""
    defect_scores = np.sort(scores[is_defective])
    allowed = int(np.floor(max_escape_rate * len(defect_scores)))
    threshold = defect_scores[allowed]        # at most `allowed` defective parts score below it
    escape = float(np.mean(scores[is_defective] < threshold))
    false_reject = float(np.mean(scores[~is_defective] >= threshold))
    return threshold, escape, false_reject

Three rules keep this honest:

1. Validate on production runs the model never saw, including at least one material lot or shift that is not in training.
2. Report per defect class and per product variant. An overall escape rate can hide a class the model never catches.
3. State the uncertainty. With few defective parts in the validation set, the escape estimate is loose. The statistical "rule of three" says that if you observe zero escapes in n defective parts, the 95% upper bound on the true escape rate is roughly 3/n. Fifty clean defective parts do not prove a low escape rate; they bound it near six percent.

Run the current inspection process (people or the old system) on the same reference set. That comparison, not a number from a brochure, is the basis for deciding whether the new station is ready.

Deploying on the line

The latency budget

The deadline is not the cycle time. It is the time between the image and the moment the part reaches the reject mechanism, minus the actuator's response time. With 300 mm between camera and reject gate at 0.5 m/s, the result must reach the PLC in well under 600 ms. Measure every stage on the real hardware: trigger to exposure, transfer, preprocessing, inference, result write.

Edge hardware and model export

Inference runs next to the line, on an industrial PC with a GPU, an embedded module such as NVIDIA Jetson, or a CPU with OpenVINO. Export the model to ONNX and compile it for the target. TensorRT 11 removed the --fp16 flag from trtexec, so set FP16 in the ONNX model first, for example with NVIDIA Model Optimizer's AutoCast:

python -m modelopt.onnx.autocast --onnx_path defect_detector.onnx --output_path defect_detector_fp16.onnx
trtexec --onnx=defect_detector_fp16.onnx --stronglyTyped --saveEngine=defect_detector.plan
ovc defect_detector.onnx --output_model defect_detector_ov

Reduced precision (FP16, INT8) shifts scores slightly. Rerun the threshold selection on the exported engine, on the validation set, before release. The threshold belongs to the deployed artifact, not to the PyTorch checkpoint. For more on fitting models onto edge hardware, see our guide to edge AI on IoT devices.

Talking to the PLC

Results go to the PLC over digital I/O for a simple pass or reject, or over the plant's fieldbus or OPC UA when you also send defect class, scores or recipe changes. Two design rules prevent the worst failures:

• Every result carries the part ID (an encoder count or the PLC's tracking index). A missed trigger must not shift every later result onto the wrong part.
• Fail closed. The PLC treats a missing, late or malformed result as a reject, and a heartbeat between the PLC and the inference PC stops the line or diverts parts when the vision system is down.

def handle_part(part_id: int, frame) -> None:
    started = time.monotonic()
    detections = engine.infer(preprocess(frame))
    score = part_score(detections, CRITICAL_CLASSES)
    decision = "REJECT" if score >= THRESHOLD else "PASS"
    if time.monotonic() - started > BUDGET_S:
        decision = "REJECT"                   # late counts as fail, never as pass
    plc.write_result(part_id=part_id, decision=decision)
    archive.put_nowait((part_id, frame, detections, decision, MODEL_VERSION))  # never block on storage

Operating the station

An inspection station is a measuring instrument and needs the same discipline.

• Shift-start verification. Run golden parts and limit samples through the station. It must pass the good ones and reject the known defects, or the line does not start.
• Image health metrics. Track mean brightness, contrast and a sharpness measure on every image. Slow drift means aging LEDs or a lens getting dirty; a step change means something moved.
• Score distributions on good parts. A shift in scores on parts that pass is the earliest sign that the model's view of "normal" no longer matches the line.
• A review queue for borderline scores. Inspectors judge the uncertain cases; their decisions become labels for the next model.
• Change control. Model version, threshold and lighting settings are tied to the product recipe and released through validation and quality sign-off. Keep the previous version deployable.
• Traceability. Store image, score, decision and model version against the part serial or batch, for as long as your quality system requires.

Retraining and release follow the same evaluate, gate and roll out pattern as any production model; our introduction to MLOps covers the pipeline side.

What goes wrong in production

• The optics drift. Dust on the lens, LEDs dimming with age, a camera knocked a millimeter out of focus. Image statistics catch it; nothing else will.
• Ambient light leaks in. A door left open or sun through a window changes contrast across the shift. Enclose the station.
• Line speed changed. A faster conveyor adds motion blur and moves the part in the frame. Re-verify after any change to speed or timing.
• A new material lot or supplier. A different surface texture looks anomalous to the model, false rejects climb, and the line pushes back.
• The threshold was lowered under pressure. False rejects cost money every hour, escapes cost money later. Thresholds change only through validation, never on the floor.
• Results landed on the wrong part. A missed trigger shifted the result queue by one. Every result needs the part ID.
• Lab images in the training set. The model learned the lab's lighting, not the defect.
• Metrics from a random split. Near-duplicate images on both sides of the split made a weak model look excellent.
• Storage stalled inference. A full disk or a slow network share blocked a synchronous image write and the station missed its deadline. Archive asynchronously.

Frequently Asked Questions

How accurate is computer vision for quality inspection?

There is no general number, and any vendor quoting one without your parts is guessing. Accuracy depends on how visible each defect is under the station's optics and lighting, how consistently parts are presented, and how consistent the labels are. Measure it yourself as escape rate and false reject rate at the part level, on parts from production runs the model never saw in training.

How many images do you need to train a defect detection model?

It depends on the approach and the variation in your parts, so plan by coverage rather than by count. Anomaly detection can start from good parts only, while supervised detection needs labeled examples of every defect type across the variation the line produces: shifts, material lots, product variants and positions on the part. Keep collecting after launch, because rare defects are usually under-represented at the start.

Can computer vision replace human inspectors?

It can take over repetitive pass or fail decisions on defects it has been validated for, and it does not tire over a shift. People remain essential for reviewing borderline cases, labeling, investigating new defect types and root cause analysis. A common pattern is to move inspectors from looking at every part to handling the cases the system is unsure about.

Should inspection models run at the edge or in the cloud?

Pass or fail decisions should run at the edge, on an industrial PC or embedded GPU next to the line, because the result must reach the PLC within the cycle time and the line cannot stop when the network does. The cloud or a central server is useful for storing images, labeling, retraining and fleet-wide dashboards. Design the edge station to keep inspecting when that connection is down.

What is the difference between machine vision and deep learning inspection?

Rule-based machine vision measures explicit features, such as an edge position, a hole diameter or a color threshold, and is precise and explainable when parts and defects are geometrically consistent. Deep learning learns what defects look like from labeled or normal examples, which handles variable, irregular defects such as scratches, stains and cracks. Many stations use both: rules for dimensional checks and a learned model for cosmetic defects.

What is anomaly detection in visual inspection?

Anomaly detection trains only on images of good parts and flags anything that looks different, producing a score and often a heatmap of where the difference is. It is useful on new lines or for rare defects where labeled defect images are scarce. Its weakness is that it flags every unusual appearance, including acceptable cosmetic variation, so thresholds need careful validation against the acceptance criteria.

Need help putting vision on a line?

We build inspection systems as engineering systems: imaging arithmetic first, honest part-level validation, fail-closed integration with the PLC, and monitoring once it runs. If we built it, we are on the pager for it, which is why we care about the station at 3 a.m., not only the demo.

• AI & Machine Learning practice
• POC-to-Production Engineering
• Talk to an engineer