Toolkit and decision tool Enrichment: beyond the IB course
A/B test
Also called Split test, A/B testing
Where you meet this term: UndergraduateMBAProfessional
An experiment in which two versions of something, such as a web page, a price or an email, are shown at random to comparable groups of users, to measure which performs better on a chosen measure.
In plain words
Showing half your customers one version and half another, at random, to see which one works better.
In more depth
Random assignment makes the groups comparable, so a difference in the result can be put down to the change. The sample size needed for statistical significance should be decided in advance, because stopping a test early or testing many variations at once inflates false positives. A/B tests measure short-term behaviour well but can miss long-term effects, such as damage to a brand.
Example
A fictional online bookshop shows half its visitors a green buy button and half a blue one; over two weeks 4.1% of the first group buy against 3.8% of the second, and the shop checks whether the gap is statistically significant before making the change.
Common misconceptions
- Misconception
Any difference between the two versions shows which is better.
In factWith small samples, differences arise by chance; the result needs to be statistically significant and large enough to matter.
Related terms
Suggested next
- Sampling error Linked through Statistical significance and Sampling
- Sales funnel Beyond IB Linked through Conversion rate
- Descriptive statistics Linked through Statistical significance
- Customer acquisition cost (CAC) Beyond IB Linked through Conversion rate
- Acid test (quick) ratio A related name
- Convenience sampling Linked through Sampling