Statistical significance in A/B testing (and free calculators to help)

Ah, statistical significance! It’s something we hear a lot about, but it’s not something we’re all familiar with figuring out on our own.

How do we define statistical significance? I went to google to find out how to define it succinctly. Here’s the best explanation I found:

“Statistical significance helps quantify whether a result is likely due to chance or to some factor of interest,” says Redman. When a finding is significant, it simply means you can feel confident that’s it real, not that you just got lucky (or unlucky) in choosing the sample.
Tom Redman in an HBR article: https://hbr.org/2016/02/a-refresher-on-statistical-significance

When dealing with statistical significance, one must look at a couple components:

Sample size
Control data
Statistical significance risk tolerance
P-value – video below!

The sample size is how many users are in your group that you are testing against.

Your control data is whatever your baseline is.

Your risk tolerance is how close to a 100% degree of certainty do you need?

In early days of experiments I like to take on a lot of risk – what I’m generally looking for is a signal that we’re on to something. As I test through different variants, I like to get more and more certain that what we’re testing and the results we are getting are closer to a high degree of certainty.

When running A/B tests, it’s important to understand what your goals are, and once you have those in place, you need to understand sample sizing and how to build a proper experiment.

Here are some tools I use to help determine statistical significance and sample sizing.