Scratching the Surface

Classical CV vs Vision-Language Models for steel defect detection

Jonathan Jayes

2026-05-18

A. Hook

The 30-second pitch

How can we prepare for the unknown?

Is computer vision a solved problem?

How can we save time with new tech?

You at the end of this talk

The goal of today’s talk is to help us think about preparing for a project when we don’t have all of the information that we need from the client.

For AQ Group one of the tasks that we could work on was rolled-metal defect detection. This is a problem that has been worked on for decades, and there are many classical computer vision techniques that can be used to solve it. However, there are also new vision-language models that have been shown to be very effective at a wide range of computer vision tasks.

So the question is, how can we use this new technology to save time and resources on a project like this? And how can we prepare for the unknowns that we might encounter along the way?

What I want you to take away from today’s talk

Classical CV

ResNet · YOLO · SVM

⚡

Lightning-fast inference

Milliseconds per image on a cheap GPU — suitable for real-time production lines.

💰

Negligible inference cost

Fractions of a cent per 1 000 images on rented compute.

🏷️

Needs labelled data to work

Requires hundreds to thousands of annotated images per domain to reach production quality.

📦

Poor domain generalisation

Change the camera, the lighting, or the defect type — and you are back to the annotator queue.

vs

Speed

Cheapness

Generalisation

Vision-Language Models

GPT-5 · Gemini 2.5 · Claude

🐢

Slow inference

Seconds per image via API — three to four orders of magnitude slower than classical CV.

💸

Higher inference cost

~$0.80–$4 per 1 000 images depending on provider — token pricing applies per call.

🌍

Zero-shot generalisation

Pre-trained on vast internet data — can reason about novel domains without any labelled examples.

💬

Explains its reasoning

Returns a natural-language rationale alongside each verdict — not just a number.

This slide frames the whole talk. Classical CV is fast and cheap but brittle — move the camera and you start over. VLMs are slow and expensive per call but generalise out of the box. The question we spend the rest of the talk answering: can we use the VLM as a teacher to build the cheap classical model, rather than putting it in the hot loop?

B. Context

What is rolled-metal?

What are rolled-metal defects?

You can see that this is quite challenging - the no defect ones still have a lot of visual clutter, and the defects are often very subtle.

What are rolled-metal defects? — your turn

Two Severstal hot-rolled steel-strip close-ups. One is labelled defective, one is not. Which is which?

Pair A: a normal strip and a pitting defect. Let the audience commit before clicking through. The “normal” still has texture — that’s the point: there’s no clean baseline.

…and here is the pixel-level ground truth

The bottom strip is the defect: a single pit (pink mask). The top strip is what Severstal annotators flagged as normal — texture and rolling marks, no actionable defect.

The defect is tiny — maybe 600 pixels in a 1600×256 image. Note how the “normal” still has visible texture. This is exactly why classical CV struggles: the signal-to-noise ratio at the pixel level is brutal.

Try another pair — both have defects

Two more close-ups. Both strips are defective. Can you spot the marks, and would you call them the same kind of defect?

Pair B: a class-3 (“scratch”) and a class-4 (“patch”). Audience usually spots something but rarely names the class — that’s the multi-class problem on top of the binary one.

…annotated by a metallurgist, pixel by pixel

The top strip carries small “scratch” blobs (orange); the bottom strip has large “patch” regions (purple). Same dataset, same camera, four totally different defect taxonomies — and the annotator decides which one to call.

Every coloured pixel was painted by a human. ~12,500 of these images were annotated for the Kaggle Severstal challenge. That’s the bill that classical CV asks you to pay every time the camera changes.

C. What do we know from AQ Group?

What is their setup??

We just don’t know what the setup at the various AQ group factories is. Ideally, we would have a team of engineers who have set up a high quality imaging camera system that is capable of capturing high resolution images of the rolled-metal as it is being produced, as in the top left quadrant of the image. They would also have a team of data annotators who have labeled a large dataset of images with the various types of defects that can occur, as in quadrants two and three. This would allow us to train a supervised learning model to detect the defects in real-time as the metal is being produced. I think this is unlikely to be the case, because if they had gone to all of the work of putting in the cameras and annotating the data, they probably would have already trained a model to do this, and celebrated the completion of a successful project, as in the bottom right quadrant of the image. So I think it’s more likely that we will have to work with a less than ideal setup, which is something that we will have to prepare for.

We make some assumptions about the project spec

We will treat the problem as if they have

• no labelled data,

• but want us to as quickly as possible provide them with a system that can identify defects on the production line.

Then in this demo, I choose

• two datasets that have some similarities to what we might expect from AQ Group as training data

• and use another dataset as a holdout set to test how well our models generalise to new domains.

• I collapse all the labels to a single binary “defect / no-defect” target, because we don’t know what the defect taxonomy is at AQ Group.

So how do we go about this??

D. Is computer vision a solved problem?

Yes — classical CV is a (mostly) solved problem!

The standard CV taxonomy: classification, detection, segmentation.

The people who solved it

Geoffrey Hinton

Ilya Sutskever, Geoffrey Hinton, Alex Krizhevsky

…and the network that started it

AlexNet (2012) — the architecture that ended hand-crafted feature engineering and started the deep-learning era of computer vision.

Talk about the progression in Computer vision, from Mnist to ImageNet to the present day. Talk about how resnet and yolo models have very fast inference, but need the images that they are trained on to be very similar to the ones that they predict on. They are good at identifying the patterns that they have seen in the training data, but they are not good at generalizing to new domains. This is a problem for AQ group, because we don’t know what the domain of the images that we will be working with is. We might be working with images that are very different from the ones that we have seen in the training data, and this could lead to poor performance.

But the issue is when we change the domain…

Rolled metal defects are not objects the models are trained on

YOLO11s (COCO pretrained, 80 classes) run on a Severstal hot-rolled steel strip with real defects. The model boxes the actual anomalies — a scratch cluster and a large surface patch — and labels every one a bird. “Steel defect” simply does not exist in its vocabulary.

The problem is generalization - these models are exceptionally good at identifying edges, lines, etc, but it needs to see examples of the objects it is trying to predict in our domain. COCO contains 80 everyday object categories built from photographs of the natural world. A rolled-metal surface defect — a pit, a scratch, a silk-spot — looks like nothing in that vocabulary. The model does not fail because it is a bad model; it fails because it has never been shown what a defect looks like. That is the annotation bottleneck in one image.

What has the model learned to see?

The annotation bottleneck

Classical CV runs on labels.

Every time the camera moves, the lighting changes, or a new defect class shows up, the whole pipeline goes back to the annotator queue.

For AQ Group we don’t yet know:

• which mill
• which camera, which lighting rig
• which defect taxonomy matters in production

So the safe bet is a system that can start without labels.

The problem is that we will have to spend a lot of time speaking to engineers and asking them to annotate data, and this is a bottleneck that we want to avoid. We want to be able to start working on the problem without having to wait for the annotations to be done, and this is where vision-language models come in.

The domain-gap problem, visualised

Train: Severstal + NEU-DET (top). Test: KolektorSDD2 (bottom).

Same task — “is this strip defective?” — wildly different pixels.

This is the visceral “look-and-feel” of distribution shift. We didn’t change the task — we changed the camera.

• Different sensors, illumination, exposure, magnification
• Different defect taxonomies (scratches vs silk-spots vs welding lines)
• Different aspect ratios (1600×256 vs 229×645)
• Different surface treatments

A classifier trained on the top row has never seen anything that looks like the bottom row.

Two classical workhorses

ResNet50 — image-level classifier. One number out: \(\hat{p}(\text{defect})\).

YOLOv8 / YOLO11s — object detector. Typed bounding boxes with per-box confidence.

Two architectures we’ll lean on. ResNet50 is the canonical image classifier — convolutional backbone, pooled features, one sigmoid output. YOLO is the canonical real-time detector — same kind of backbone, but the head predicts a grid of bounding boxes with class scores. ResNet gives us “is this strip defective?”. YOLO gives us “and here’s where, with a box per instance”. Both are pre-trained on ImageNet / COCO and then fine-tuned on Severstal + NEU-DET.

The VLM tells you why — not just what

For each image: the raw crop, the ResNet Grad-CAM heatmap, and the GPT-5.4 natural-language rationale. The classical model gives a number; the VLM gives a sentence.

Row 1 (Severstal, normal): ResNet confidence is low (p̂=0.06) — correctly uncertain. The CAM lights up mid-strip texture. VLM correctly calls NORMAL and explains why: “darker vertical streaks are consistent with benign surface texture/illumination variation rather than a distinct localised manufacturing defect.”

Row 2 (KolektorSDD2, defect): ResNet says p̂=0.11 — confidently wrong. The CAM scatters attention at the bottom, nowhere near the actual scratch. VLM correctly calls DEFECT (conf 0.94) and explains: “a single bright diagonal linear mark on the left side is inconsistent with the surrounding texture … most plausibly caused by mechanical scratching.”

Row 3 (Severstal, defect): ResNet gets it right (p̂=0.75) and CAM fires broadly. VLM also correct (conf 0.79): “a localized bright vertical mark near the center … most consistent with a mechanical scratch or roller contact mark.”

The key contrast: the Grad-CAM tells an operator where the network looked; the VLM tells them what it saw and why it matters. Run with: uv run python scripts/make_cam_vlm_slide.py.

More labels won’t save us

Held-out F1 for every classical recipe we tried.

No classical recipe gets us there.

Fine-tuning on Kolektor helps — F1 climbs from 0.16 to 0.56 — but still leaves us far short of production-ready.

~11 700 labels in, and the ceiling is a passable model on one mill.

This slide earns the pivot to VLMs. We tried the obvious classical things — bigger model, fine-tune on a slice of the new domain, detection head (YOLO11s), self-labelling Severstal and re-training YOLO on those pseudo-labels. None of them clear the bar on KolektorSDD2. Fine-tuning on Kolektor lifts F1 3.5× but plateaus around 0.56. Numbers (held-out F1 on Kolektor):

• ResNet50: 0.16
• ResNet50+FT: 0.56
• YOLO11s: 0.38
• YOLO-bootstrap: 0.20

The figure is deliberately all classical — the VLM hasn’t entered the story yet. Audience should leave this slide asking “what else could possibly help?”. Source: results/{resnet50,resnet50_kolektor_ft,yolo,yolo_bootstrap}/metrics.json, rendered by scripts/make_comparison_figures.py (f1_comparison_classical.png).

A YOLO box says something a heatmap can’t

Same three images, two outputs. Middle: ResNet50 Grad-CAM — one scalar p̂(defect) plus a soft “where roughly”. Right: YOLO11s — one or more typed bounding boxes, each with its own confidence. Top two rows are in-domain Severstal; bottom row is out-of-distribution KolektorSDD2.

Qualitative companion to the scoreboard. The F1 numbers don’t tell you what each output looks like. Walk the rows:

Row 1 (Severstal, in-domain): ResNet fires on the whole bright region (p̂=1.00) — useful as a yes/no, useless if you want to know which defect. YOLO drops four tight boxes on the actual scratches with per-instance confidence. That’s a work order for an operator.

Row 2 (Severstal, in-domain): Same story — three crisp boxes vs one broad blob.

Row 3 (Kolektor, OOD): Both models break, but differently. ResNet says “no defect” (p̂=0.11) and the CAM is on irrelevant texture. YOLO does fire — two low-confidence boxes — but they’re at the top, not on the actual scratch in the middle.

Neither is great on the OOD row. The point of the slide is different output shape, same domain-gap problem — bringing more classical machinery to bear doesn’t close the gap, it just changes what “failure” looks like. Time to try something different.

Why are VLMs better at CV tasks now?

Nov 2022

GPT-3.5

No Fusion

Launch of ChatGPT. Core engine handles text tokens exclusively.

Mar 2023

GPT-4 (Base)

No Fusion

Massive logical upgrade, but architecture remains text-only.

Sep 2023

GPT-4V / Voice

Late Fusion

Stitched pipeline. External vision encoders and audio transcribers translation layers.

May 2024

GPT-4o

Early Fusion

Unified architecture. Processes text, native sight, and live audio in one neural space.

Jul 2024

GPT-4o mini

Early Fusion

Streamlined omni variant maintaining native multimodal processing.

2025+

GPT-5 Family

Early Fusion

Scales early-fusion to native video generation and seamless token blending.

Slide goal: Highlight the fundamental architectural paradigm shift OpenAI went through in under 3 years.

• GPT-3.5 & GPT-4 base — traditional, blind LLMs. Text-in, text-out. No fusion: there were no other modalities to handle.
• GPT-4V & classic Voice — late fusion. Components stitched together; information is lost (tone of voice flattened to plain text; vision pre-filtered through a separate encoder). It’s a pipeline bottleneck.
• GPT-4o onwards — early fusion. No translators or adapters: images and raw audio go straight into the core network alongside text. Enables human-level latency and native audio outputs.

E. How do Vision-Language Models see images?

Late Fusion

Train a Large Language Model on text, then add vision as a separate input channel with a translation layer.

Output Text

LLM

Projection (Feature Vectors)

Vision Encoder

Image Input

Text Prompt

In short, a CLIP-based LLM is like a monolingual English speaker relying on a highly skilled translator (the adapter) to understand French (images). A natively multimodal model is truly bilingual from birth, processing and generating text, images, and audio seamlessly with the same “brain.”

Early Fusion

Train a Large Language Model on text and images together, with a shared internal representation.

Text

Image

Video

MM

Any-To-Any

Generation

In short, a CLIP-based LLM is like a monolingual English speaker relying on a highly skilled translator (the adapter) to understand French (images). A natively multimodal model is truly bilingual from birth, processing and generating text, images, and audio seamlessly with the same “brain.”

Early Fusion shared embedding space

F. Flagship VLM Bake-Off

Same images, frontier models, zero training

F1 and per-call cost for gpt-5.4, gemini-2.5-pro, gpt-4.1-mini on the unseen Kolektor domain.

Setup

• 50 images (balanced) from held-out KolektorSDD2
• Identical CoT-with-JSON-schema prompt
• 3 providers via src/vlm_clients.py
• Costs logged to results/cost_ledger.csv, capped at $50

Reading the chart

• All three frontier models beat the supervised ResNet50 on Kolektor.
• Gemini 2.5 Pro tops the chart at F1 0.83.
• Azure mid-tier (gpt-4.1-mini) is the price–performance knee at F1 0.80 for ~10× less per call.

Source: results/vlm/flagship_summary_.json, rendered with scripts/make_flagship_bakeoff_figure.py.

Best VLM vs classical — KolektorSDD2 F1

Held-out F1 on KolektorSDD2: every classical recipe vs the best frontier VLM (zero-shot).

The headline.

• Best classical recipe (ResNet50 + Kolektor FT): 0.56
• Best zero-shot VLM (Gemini 2.5 Pro): 0.83

No labels on the new domain, no fine-tuning, no schema changes — just an API call with a prompt.

This is the punch-line of the bake-off. We poured ~11 700 labels and a fine-tune into the classical side; the VLM saw nothing of Kolektor before inference. Same metric, same images, very different ceiling. Rendered by scripts/make_comparison_figures.py (f1_kolektor_vlm_vs_classical.png), VLM number pulled from results/vlm/flagship_summary_*.json.

G. Prompt-tuning as gradient descent on words

A structured-output teacher critiques a structured-output student

F1 vs prompt-tuning round, per provider. Star = held-out F1 after final addendum.

The loop (scripts/prompt_tune_vlm.py)

1. Score base prompt on a 12-image calibration set
2. Collect every disagreement with its parsed reasoning + attributes
3. Feed disagreements to gpt-5.5 as a “domain prompt engineer”
4. Tuner emits JSON: summary, common_mistakes[], prompt_addendum
5. Append the addendum, re-score, repeat

The tuner output is schema-enforced — strict JSON, no free-form prose. The addendum becomes a # DOMAIN-SPECIFIC GUIDANCE block prepended to the student’s user prompt.

The tuner’s schema

TUNER_SCHEMA = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "common_mistakes": {
            "type": "array",
            "minItems": 1,
            "items": {
                "type": "object",
                "properties": {
                    "pattern": {"type": "string"},
                    "fix": {"type": "string"},
                },
                "required": ["pattern", "fix"],
            },
        },
        "prompt_addendum": {
            "type": "string",
            "description": (
                "Markdown to append under '# DOMAIN-SPECIFIC GUIDANCE'. "
                "Explains the domain, what counts as a defect, and "
                "the visual heuristics that disambiguate the hard cases."
            ),
        },
    },
    "required": ["summary", "common_mistakes", "prompt_addendum"],
}

The “explain the domain” bit is what makes this generalise. We’re not just patching mistakes — the tuner has to teach the student what a hot-rolled steel strip is, what counts as a scratch vs. a silk spot, why oil residue isn’t a defect, etc.

H. Two languages of explanation

Grad-CAM points. The VLM tells you why.

Four images — one normal Severstal frame at the top, then three defective close-ups from KolektorSDD2 and Severstal. Middle: ResNet50 Grad-CAM heatmap (red = pixels that pushed p(defect) up). Right: live GPT-5.4 reasoning for the same image (no fine-tuning, structured-output prompt).

This is the slide that justifies pairing them. Grad-CAM = where. VLM rationale = what & why. Walk through each row: the normal Severstal is correctly called by both (the VLM volunteers “benign longitudinal texture”). The Kolektor scratch is missed by the ResNet (p=0.11, CAM points at the edges) but caught by GPT-5.4 with a clean diagnosis. The Severstal row shows both flagging it as a defect, but Grad-CAM gives you a blob and the VLM gives you “localised bright vertical mark, mechanical scratch or roller contact.” Last row is the honest failure: a partial-occlusion frame where the ResNet fires on the bright edge and the VLM declines — and tells you it’s declining because “much of the frame is dominated by a large geometric edge/occlusion.” For a process engineer triaging a flagged image, the second column is what closes the loop.

I. The economics: when does a VLM pay for itself?

How fast is each model on one image?

• Log scale
• Linear scale

End-to-end inference latency per image (log scale, lower is better). Classical models are measured on an Azure NC4as_T4_v3 (T4 GPU) and an Apple-silicon CPU; VLM numbers are the mean wall-clock time per call from results/vlm/flagship_summary_*.json.

Same data on a linear axis (lower is better). The classical bars are visually invisible — that’s the point: VLMs are three to four orders of magnitude slower per call.

This is the latency reality check. A T4-served ResNet handles an image in ~6 ms — that’s ~160 images per second per GPU. The cheapest VLM in our bake-off (GPT-4.1-mini on Azure) takes ~3 seconds; Gemini-2.5-pro takes ~17. That’s three to four orders of magnitude. A 60 frames-per-second inspection line is fine for the classical model and impossible for the VLM. The point of the next slide isn’t to say VLMs are bad — it’s to say: that latency comes from somewhere, and someone has to pay for it. Flip to the linear tab to make the scale gap visceral.

…and what does each call actually cost?

• Log scale
• Linear scale

Same models, but USD per 1 000 images (log scale, lower is better). Classical bars are pure GPU/CPU rental on Azure ($0.526/hr for a T4); VLM bars are real API token spend pulled from results/cost_ledger.csv.

Same numbers on a linear axis (lower is better) — the classical bars collapse to a line at zero next to the VLM bars.

The latency gap shows up almost 1:1 as a cost gap. Running ResNet on a rented T4 costs fractions of a cent per thousand images — most of the bill is the VM, not the compute. Calling Gemini-2.5-pro on the same 1 000 images costs about $4.35 — roughly 5 000× more. GPT-4.1-mini sits in between at ~$0.83. So the choice isn’t “smart vs dumb model” — it’s “is the marginal accuracy from the VLM worth 4 to 5 orders of magnitude of cost?” Which sets up the break-even curve on the next slide.

One-off labelling + classical inference vs per-image VLM

Break-even between paying a VLM per image forever vs. paying once to label N_train = 2,332 images and serving a near-free ResNet/YOLO.

Three label-cost worlds

• $0.05 / label — in-house GPU-side annotators
• $0.20 / label — offshore crowd workers
• $0.50 / label — expert metallurgist sign-off

Break-even images processed (VLM-only vs label-once-then-classical):

• $0.05/label → ~26k images
• $0.20/label → ~104k images
• $0.50/label → ~261k images

A line running 24/7 hits 26k images in hours. The hybrid (YOLO → escalate 30% to VLM) buys back another 3×.

Numbers from scripts/make_cost_tradeoff_figure.py, sourcing per-call VLM cost from results/vlm/flagship_summary_*.json.

Does the student ever catch the teacher?

ResNet50 trained on VLM pseudo-labels at increasing N (3 seeds per N). Dashed line = GPT-5.4 zero-shot F1 = 0.79.

Setup (scripts/labelling_curve.py)

• Pool: Kolektor train (~2 300 images, ~10 % defective)
• Labels: VLM pseudo-labels with measured noise (TPR=0.68, TNR=0.96)
• 3 seeds per N — no class balancing
• 6 epochs, AdamW, ResNet50

Reading the chart

• At N ≤ 1 600, the student has not caught the teacher (F1 peaks ~0.37 vs 0.79).
• The gap signals that more labels and/or a cleaner label budget would be needed.
• The dollar axis makes the trade-off concrete: $10.74 of labelling still falls short.

The student (ResNet50) never catches the teacher (GPT-5.4 zero-shot) within the 1 600-image budget tested. This is an honest result — the teacher is very good and the labels are noisy. The practical implication is that you need either more images, better labels, or a fine-tuned VLM starting point before the classical model matches.

J. The takeaway

Use VLMs to teach classical CV. Don’t put them in the hot loop.

What we showed

1. Classical CV fails under realistic domain shift — and Grad-CAM tells you why.
2. Frontier VLMs generalise zero-shot to unseen mills and defect taxonomies.
3. A prompt-tuning loop, with a structured-output teacher, moves F1 measurably on a 12-image calibration budget.
4. VLMs and Grad-CAM give complementary explanations — where and why.
5. At line-rate volumes, the break-even versus paid labelling lands in hours, not months.

The recipe for AQ Group

• Day 1 — drop a VLM in shadow mode behind the existing camera. Zero labels needed.
• Week 1 — harvest the VLM’s pseudo-labels. Tune the prompt against your disagreements.
• Week 2 — train a small ResNet / YOLO on the pseudo-labels. Put that in the hot loop.
• Ongoing — escalate the bottom-30%-confidence frames back to the VLM. Re-train weekly.

The VLM is the teacher and the safety net, not the production classifier.

This is the punchline. Everything before this slide is evidence; this slide is the recommendation.