Toolkit and decision tool Enrichment: beyond the IB course

A/B test

Also called Split test, A/B testing

Where you meet this term: UndergraduateMBAProfessional

An experiment in which two versions of something, such as a web page, a price or an email, are shown at random to comparable groups of users, to measure which performs better on a chosen measure.

In plain words

Showing half your customers one version and half another, at random, to see which one works better.

In more depth

Random assignment makes the groups comparable, so a difference in the result can be put down to the change. The sample size needed for statistical significance should be decided in advance, because stopping a test early or testing many variations at once inflates false positives. A/B tests measure short-term behaviour well but can miss long-term effects, such as damage to a brand.

Example

A fictional online bookshop shows half its visitors a green buy button and half a blue one; over two weeks 4.1% of the first group buy against 3.8% of the second, and the shop checks whether the gap is statistically significant before making the change.

Common misconceptions

  • Misconception

    Any difference between the two versions shows which is better.

    In fact

    With small samples, differences arise by chance; the result needs to be statistically significant and large enough to matter.

Suggested next