Measure or calculation Enrichment: beyond the IB course
Statistical significance
Where you meet this term: UndergraduateMBAProfessional
A judgement that a result seen in a sample, such as a difference between two groups, is unlikely to have arisen by chance alone, against an agreed threshold.
In plain words
A way of checking whether a difference you see in data is real or just luck. If it is unlikely to be luck, it is called statistically significant.
In more depth
A test computes a p-value, the probability of a result at least as extreme as the one observed if there were really no effect; a p-value below a chosen level, often 5%, is called significant. Significance is not importance: with a huge sample a trivial difference can be significant, and with a small one an important difference can fail the test. Testing many comparisons at once raises the chance of false positives.
Example
A fictional retailer's test finds conversion of 4.1% for one page and 3.8% for another; with 50,000 visitors in each group the gap is significant at the 5% level, but with 5,000 in each it could easily be chance.
Common misconceptions
- Misconception
A statistically significant result is an important one.
In factSignificance says a difference is unlikely to be chance, not that it is large enough to matter.
Related terms
Suggested next
- Random sampling Linked through Sampling and Sampling error
- Research bias Linked through Sampling and Sampling error
- Conversion rate Beyond IB Linked through A/B test
- Mean Linked through Descriptive statistics
- Median Linked through Descriptive statistics
- Mode Linked through Descriptive statistics