8.5

Incrementality / Geo-Lift Testing

A randomised experiment that withholds spend from matched markets or audiences in order to measure what that spend actually caused.

Field-experiment practice; canonical demonstration Blake, Nosko & Tadelis, Econometrica (2015)

What it does

Randomly withholds spend from matched markets or audiences to measure what it actually caused. The only honest measurement of causal contribution.

When it breaks

Needs sample, time and patience. The statistical power to detect a small effect is absent from most budgets, so “no effect” usually means “we could not measure it”.

Case

eBay ran a large randomised shutoff of paid search. Turning off brand keywords recovered 99.5% of the lost paid clicks through organic search: essentially zero incremental value. For non-brand keywords the true experimental ROI was −63%, against naive observational estimates of +4,173%.

Blake, Nosko & Tadelis · eBay paid search experiment ↗
+1,000% +2,000% +3,000% +4,000% Break even Naive +4,173% Controlled, still positive Experimental −63% One campaign, three answers 99.5% recovered organically
Incrementality / Geo-Lift Testing · 8.5

In the wild

Unchecked · not part of the tier

What people have written about this tool in the last twelve months. A machine found these and nobody has read them. Everything above this line was checked by hand.

    Loading…