Six things that failed when we trained traffic-sign detectors on rendered images only

Stepan Galickov · 19 September 2026

For about a year I have been building training data for traffic-sign detection by rendering it: the sign and its scene are built in 3D, and the labels are computed from geometry instead of drawn by hand. No photographs in training at all.

The part that gets written about is the headline number. The part that actually cost the time is everything that did not work, and most of that is not written down anywhere. This post is mostly the second part.

Everything below is measured on real photographs the model never saw, with three independent training runs per configuration. I will say where the numbers are weak.

The setup, briefly

Detector is YOLOv8s, input 640, 100 epochs, batch 16, AdamW, three seeds. Evaluation is at input 1280 — more on that later, because it turned out to matter more than most of the render settings.

Exams are four sets of real frames: seasonal slices of the open Russian sign dataset RTSD (October, 905 frames / 1362 signs; winter, 157 / 210; spring, 285 / 655), the international Mapillary traffic-sign set (1235 / 1960) as a cross-domain check, and our own night drive (315 frames / 735 boxes, different city, different camera, hand-labelled) because no open Russian night test exists.

One rule from early on: we never evaluate on our own synthetic frames. When we did, an early Faster R-CNN run scored 0.95 on them and 0.61 on the real October exam. Roughly 1.6× inflation. The YOLO recipe used throughout the rest of this post has the same gap between its own frames and real ones. Every decision in the project after that point was made on real exams only, and a good share of the failures below were discovered precisely because of that rule.

What did work

Training on rendered frames alone, with no real photograph in training, reaches 75–84% of the score of the same architecture trained on 5.6k real labelled frames, depending on season:

mAP50, three seeds each. The ceiling is a small-data baseline, not a production-scale model, and that matters when reading the percentage.

The more useful result is what happens in the way data is actually used: pretrain on rendered frames, then fine-tune on real ones. On the cross-domain day exam that gave +0.15 mAP50 (0.333 → 0.480), three seeds, all three positive, spread 0.04.

But the average hides the interesting part. The gain is not spread evenly across classes — it tracks how many real examples of that class already existed in training:

A class with no real examples at all starts being detected. A class with 4330 examples — nearly half the dataset — gets slightly worse. Rendered data adds what is missing and gets in the way of what already works.

The peak is not at zero, and that is worth explaining rather than glossing over. The class with no real examples gains only 0.11 in absolute terms: the model has to learn it from rendered frames alone, with nothing real to anchor it, and it ends up detectable rather than good. The largest absolute gains land on classes that have a few dozen to a few hundred real examples — enough to anchor the appearance, not enough to cover the variation. Relative to where they started, though, the zero-example class is the only one that goes from impossible to possible.

Note that "rare" here means rare in the dataset, not rare on the road. "No parking" stands on every street in the country and still had only 54 examples in the reference set.

One caveat on this table: the additive measurement was run on the previous version of the generator. The current version differs only in where the road backgrounds come from, and on a comparable slice the two scored equally (0.571 vs 0.568, three seeds), but the additive run has not been repeated yet.

Now the six that failed

1. We modelled snow. There was no snow to model.

Snow-covered signs seemed like an obvious gap: you cannot schedule a blizzard, and a camera crew costs money per day. So we built snow accumulation on sign faces, trained with it, and it worked — on our own snow filter, day 0.414 → 0.465 and 0.484.

On real winter photographs it did not hold up: −0.055 and −0.052 on the non-winter slice, both seeds — and on the winter slice the two seeds disagreed outright, −0.014 and +0.017.

Then someone finally looked. We pulled the 24 largest signs (70–166 px) out of 1154 real winter frames and inspected them by eye. Not one was covered. A road sign is vertical and its retroreflective sheeting is smooth; snow does not stick to it. The gap we had spent weeks modelling did not exist in the data.

The general version of this: your own generated test set will tell you that your generator works. It is measuring whether your generator is self-consistent, which it always is. Check that the phenomenon exists in real frames before you model it.

We built rain twice. Once as an effect — streaks, droplets on the lens, the thing that looks like rain in a film. Once as physics — wet road with changed reflectance, atmospheric veil, defocus, no visible droplets at all.

The painted version lost on every real exam and both seeds: −0.02 to −0.055. The physical version gained +0.041 on a real winter exam. Fog, built from the Koschmieder extinction law rather than a blur filter, gained +0.031 on the same one. One caveat on both numbers: that exam is an older 889-frame winter set, not the 157-frame winter slice listed above, and the baseline is a day recipe from an earlier version of the generator. The two generations of exam are not comparable, and we do not add deltas across them.

We ran that comparison four separate times on different effects before believing it. The pattern held every time: if the effect is a post-process applied to the image, it does not survive the transfer to real photographs. If it changes how light actually reaches the camera, it does.

3. A condition on its own can be worse than no condition at all

Having established that physical rain helps, the obvious next step is to train a rain-heavy dataset. That lost: physical rain alone scored 0.504 and 0.489 on the October exam against 0.546 for a plain day recipe. Two seeds, both down.

Night behaved the same way — a night-heavy recipe scored below the day recipe on the October exam. Both conditions only pay inside a mixture, where they act as variation rather than as the dominant mode.

We nearly shipped the conclusion "rain helps" from the mixture experiment alone. It is only true in that context.

4. Matching the real distribution lost, twice

Our rendered signs occupied about 4.6% of the frame on average. Real ones occupy 2.3% — half that. An obvious defect, and an obvious fix.

Shrinking them to match real statistics lost 0.033. Making them larger gained 0.023.

Second instance: we measured that our signs were brighter relative to their surroundings than real ones — our signs sat about 31 units above their scene mean, real ones about 26 below. A clear discrepancy. We built a version that removed it. It lost 0.026 on the main exam, and the daytime case it was supposed to fix did not improve either.

A measured discrepancy between your renders and reality is not the same thing as a discrepancy worth fixing. We now require a hypothesis about why a given mismatch would hurt the detector before spending a cook on closing it.

5. Licensed stock backgrounds lost badly, and the fix was not a rendering fix

For a while our road backgrounds came from a mixed pool with unclear provenance. Rebuilding on fully licensed stock imagery seemed like pure upside: same pipeline, clean rights.

It lost 0.044 on the main exam, pairwise 1 of 9 runs.

The reason came out of a simple probe: we ran a detector over both background pools and counted how often it fired on road content. The clean stock pool hit 26.5% — against a false-positive floor of 25.9%. In other words, no signal at all: stock "road" imagery is mostly landscapes with a road somewhere in them. A pool of frames shot from a driver's viewpoint hit 47.8%.

The fix was not a better shader. We went out and filmed our own road backgrounds. On a comparable slice the rebuilt version matches the old one (0.571 vs 0.568, three seeds) while being clean on rights.

Of everything in this post, this is the finding I would most want to have known a year earlier. Where the backgrounds come from moves sim-to-real transfer more than most of the render settings we spent time tuning.

6. More epochs is not more data

A small factorial, equal compute per cell, three seeds, judged on the October exam:

three times the data, same epochs: +0.003, +0.049, +0.055 — three of three positive;

three times the epochs, same data: −0.015, −0.041, −0.048 — three of three negative.

Worth running because the opposite is cheap to believe when a training run is the expensive part and generation is the cheap part.

Three measurement traps that cost us more than any single experiment

Evaluation resolution.

The same weights scored 0.192 at input 640 and 0.363 at 1280 — nearly double, from a flag. A real sign is about 23 pixels at 1280. Training at 1280, however, failed on every real exam. The working recipe is train at 640, evaluate at 1280, and if someone reproduces our numbers and gets half of them, this is the first thing to check.

Aggregate versus judged classes.

On the night exam, the aggregate across all 15 classes showed +0.055 from pretraining. Restricted to classes with enough boxes to judge, it showed −0.001, and the sign flipped between seeds. The claim would have rested entirely on classes we had ourselves declared unmeasurable. We do not make a night claim.

Two generations of test sets.

Our weather deltas were measured against one baseline on one set of exams; the percentage-of-ceiling numbers against another. They are not additive and we keep them in separate tables. This is boring and it is the single easiest way to publish a number that is quietly wrong.

What we still cannot claim

Night.

Our night numbers come from one drive on lit city streets. 60% of the boxes in that exam are a single class. Read it as "pedestrian crossings at night", not as night.

Spring got worse.

The current version scores 0.641 on spring against 0.687 for the previous one, consistent across three seeds, while autumn and winter improved. We do not have an explanation yet.

Coverage.

15 classes out of 310 in the national regulations. A new class takes days; the full set is planned engineering, not a weekend.

Generality.

Everything here is rigid, standardised, regulated objects with exact geometry. I would not extrapolate any of it to deformable objects or to scenes where contact physics matters.

The result that narrowed the claim

We had been saying that rendered data helps rare classes. A researcher who has published on sign detection re-ran his own archived augmentation experiments against that claim and found that classical augmentation shows the same tail-favouring shape, though measured on stricter localisation rather than AP50.

So the honest statement is narrower than the one we started with. Rendering is not special across the tail in general. It is special where a class has zero real examples, because augmentation has nothing to augment there. That case is the first row of the table above, and it is the only one where nothing else can substitute.

He did that on his own time, on a claim that was not his, and it cost us a marketing line. It was the single most useful piece of feedback the project has had.

If you are about to do this

Never evaluate on your own generated data. Ours inflated results by 1.6×.

Check that the phenomenon exists in real frames before modelling it. Look at the frames yourself.

Prefer physics over post-processing. Painted effects do not transfer.

Spend your first week on where the background frames come from, not on the shader.

Decide the class census before looking at results — which classes have enough boxes to judge at all.

Three seeds minimum. Two once hid a spread of 0.045 from us.