Measure or calculation Enrichment: beyond the IB course

Statistical significance

Where you meet this term: UndergraduateMBAProfessional

A judgement that a result seen in a sample, such as a difference between two groups, is unlikely to have arisen by chance alone, against an agreed threshold.

In plain words

A way of checking whether a difference you see in data is real or just luck. If it is unlikely to be luck, it is called statistically significant.

In more depth

A test computes a p-value, the probability of a result at least as extreme as the one observed if there were really no effect; a p-value below a chosen level, often 5%, is called significant. Significance is not importance: with a huge sample a trivial difference can be significant, and with a small one an important difference can fail the test. Testing many comparisons at once raises the chance of false positives.

Example

A fictional retailer's test finds conversion of 4.1% for one page and 3.8% for another; with 50,000 visitors in each group the gap is significant at the 5% level, but with 5,000 in each it could easily be chance.

Common misconceptions

  • Misconception

    A statistically significant result is an important one.

    In fact

    Significance says a difference is unlikely to be chance, not that it is large enough to matter.

Suggested next

  • Random sampling Linked through Sampling and Sampling error
  • Research bias Linked through Sampling and Sampling error
  • Conversion rate Beyond IB Linked through A/B test
  • Mean Linked through Descriptive statistics
  • Median Linked through Descriptive statistics
  • Mode Linked through Descriptive statistics