9.8
Split-Run Testing
Running two versions of the same advertisement to randomly divided halves of one audience and keeping the version that produces more response.
Claude Hopkins, Scientific Advertising (1923), on keyed coupons; systematised by John Caples, Tested Advertising Methods (1932)
What it does
Replaces opinion about advertising with a measurement of it. Random assignment, identical conditions, a counted response — the design every A/B test, geo-lift and incrementality experiment on this site inherits. Its real product is not the winning ad; it is the habit of finding out you were wrong cheaply and often.
When it breaks
Most tests are underpowered for the difference they are looking for, so a large share of declared winners are noise that will not repeat. The design also only compares the variants you thought of: it optimises inside a set and never reports that the set was wrong, which is how a headline test returns a clean 12% lift on a page selling the wrong thing. And it measures response, not profit — Hopkins' keyed coupon counted replies and the modern equivalent counts clicks.
Case
Kohavi and colleagues report the base rate from running the design at industrial scale: only one third of the ideas tested at Microsoft improved the metric they were designed to improve. Their paper collects the same numbers from elsewhere — around 10% of Google's controlled experiments led to business changes, and Netflix considers 90% of what it tries to be wrong. The method's value is in what it stops, not in what it finds.
Kohavi, Deng, Frasca, Walker, Xu & Pohlmann — Online Controlled Experiments at Large Scale, KDD (2013) ↗Diagram not yet drawn
Hopkins' keyed coupon and a modern A/B split drawn as the same diagram; underneath, a hundred tested ideas with a third shaded as wins and the remaining two thirds shaded as the thing the method was actually bought for.
In the wild
Unchecked · not part of the tier
What people have written about this tool in the last twelve months. A machine found these and nobody has read them. Everything above this line was checked by hand.
Loading…