Fisher (1935); Kohavi, Tang and Xu (2020)
Randomized experiments
Randomly assigning people, regions or periods to receive the marketing or not makes the groups alike in everything except the marketing, so the difference in outcomes estimates its causal effect. Holdout groups and geo tests apply the idea to campaigns.
How to use it
Define the outcome and the minimum effect worth detecting before the test, calculate the sample needed, randomize, and resist stopping early when results look good. Use experiments to calibrate models and to settle the largest budget questions.
Where it fails
Small effects need very large samples. Tests measure short-term effects in a specific context, spill over between groups, and can be expensive when the holdout forgoes sales.
R. A. Fisher (1935). The Design of Experiments. Oliver and Boyd. Ron Kohavi, Diane Tang and Ya Xu (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.