Incrementality / Geo-Lift Testing

A randomised experiment that withholds spend from matched markets or audiences in order to measure what that spend actually caused.

Field-experiment practice; canonical demonstration Blake, Nosko & Tadelis, Econometrica (2015)

What it does

Randomly withholds spend from matched markets or audiences to measure what it actually caused. The only honest measurement of causal contribution.

When it breaks

Needs sample, time and patience. The statistical power to detect a small effect is absent from most budgets, so “no effect” usually means “we could not measure it”.

Case

eBay ran a large randomised shutoff of paid search. Turning off brand keywords recovered 99.5% of the lost paid clicks through organic search — essentially zero incremental value. For non-brand keywords the true experimental ROI was −63%, against naive observational estimates of +4,173%.

Blake, Nosko & Tadelis — eBay paid search experiment ↗

Diagram — not yet drawn

Three ROI estimates for one campaign on a single axis — naive, controlled, experimental — with the distance between them as the whole content of the drawing.

In the wild

Unvetted · not part of the tier assessment

What has been written about this tool in the last twelve months. Machine-retrieved and unchecked — everything above this line was checked.

Loading…