Ah, statistical significance! It’s something we hear a lot about, but it’s not something we’re all familiar with figuring out on our own.
How do we define statistical significance? I went to google to find out how to define it succinctly. Here’s the best explanation I found:
“Statistical significance helps quantify whether a result is likely due to chance or to some factor of interest,” says Redman. When a finding is significant, it simply means you can feel confident that’s it real, not that you just got lucky (or unlucky) in choosing the sample.
Tom Redman in an HBR article: https://hbr.org/2016/02/a-refresher-on-statistical-significance
When dealing with statistical significance, one must look at a couple components:
- Sample size
- Control data
- Statistical significance risk tolerance
- P-value – video below!
The sample size is how many users are in your group that you are testing against.
Your control data is whatever your baseline is.
Your risk tolerance is how close to a 100% degree of certainty do you need?
In early days of experiments I like to take on a lot of risk – what I’m generally looking for is a signal that we’re on to something. As I test through different variants, I like to get more and more certain that what we’re testing and the results we are getting are closer to a high degree of certainty.
When running A/B tests, it’s important to understand what your goals are, and once you have those in place, you need to understand sample sizing and how to build a proper experiment.
Here are some tools I use to help determine statistical significance and sample sizing.
A/B Test Sample Size Calculator from Optimizely
Plug in your baseline / control conversion rate, plug in what relative change you would like to see, review the statistical significance % and it’ll give you the sample size needed.
Notice what happens when I bring this down to 80% confidence:
Neil Patel’s A/B Testing Significance Calculator
With this tool you can put in your control numbers and then one or more variants to see how your conversion rates compare and what the % certain his model shows for statistical significance.
Survey Monkey AB testing calculator
Blast Statistical Significance Calculator
The Last Statistical Significance Calculator You’ll Ever Need
Blast website 🙂
Blast is being pretty bold here but I must say – this is one heck of a calculator. It’s complex, it’s detailed, it’s, well, really good!
