Reading Time: 7 minutes
AB Testing Sample Size: The 4 Levels of Difficulty
Rate this post

The question that comes often is “Is AB Testing Possible With Under 100k Monthly Visitors?”. The main decision factor is the volume of traffic going to your website (or page), it is then important to calculate the AB Testing Sample Size before launching any AB test.

We have created a simple model, divided in 4 levels of difficulty, to help you evaluate if the amount of visitors going to your website is enough to run an AB test on your website:

1. Fear Factor zone
2. Thrilling zone
3. Exciting zone
4. Safe zone

AB Testing Sample Size

AB testing is omnipresent now. But many of these tests are close to being useless. In this post I’ll show you a graphical representation of when AB testing actually works and when it is a waste of time and money (that means when the AB Testing Sample Size is higher than the available volume of traffic).

But before a few basics that are important:

What is AB Testing?

AB testing is an optimisation technique that consists of comparing two different page options, say page A and page B. Website visitors are split in to two groups with each seeing only one of the two pages so that you can measure what effect each page has on their online behaviour and on conversions. The goal being, of course, to see which version results in the better conversion rates.

To achieve reliable AB tests it is essential that they are statistically significant so that you don’t end up with false results and waste your marketing budget.

It’s important to take into consideration:

Unless you’re a statistician, these specifications might not mean much to you but, don’t worry, carry on reading this article and all will become clear!

How to calculate the AB Testing Sample Size?

There are many tools online that can do the necessary calculations for you. One of the best one is the AB Testguide.

AB Test Guide - AB Testing Sample Size

To know how many visitors are required for a test and calculate the AB Testing Sample Size, you will need to provide 3 pieces of information:

1. Your Analytics data

The conversion rate of the page being tested: this is often the Subscription/Visitor ratio or the Sales/Visitor ratio.

The number of unique weekly visitors on the page being tested. Providing this information will enable you to see how many days your test will need to run for in order to reach a statistically significant number of visitors.

2. Your expectations

You need to specify the expected percentage increase that your test will generate. This can e the tricky part because it shouldn’t be the increase you would like but rather a more realistic and objective projection.

So why do you need to estimate the increase in conversion when it’s just this that you’re trying to test in fact? Well, actually, the value that you project will act in this case as the threshold for result detection. The higher this threshold, the greater the number of visitors (and AB testing sample size) required will be for a reliable and accurate result. In fact, contrary to what you might think, a large improvement percentage is easier to detect and so can be confirmed more quickly but it’s this very rapidity that can make it less reliable. Whereas, a smaller improvement percentage will be more subtle and discreet, meaning more time and visitors will be necessary to detect the change and it will therefore be representative of a more reliable result.

Conversion optimisation experts agree that it is very difficult to see an improvement of any more than 10% on your overall conversion rate (Sales/Visitors). Reaching anything beyond 10% will require “innovative” changes, meaning quite significant changes to your site. More “iterative” changes –  those that consist of a simple alteration to a message or button, for example – will generally only bring about an increase of less than 5%, often far less! Therefore, in most cases, we would recommend “5%” as a realistic projected increase.

3. Statistical Parameters 

Without going in to too much technical detail, these are the parameters that should give you a reliable result:

Hypothesis: “2-sided”

Power: 80%

Confidence level: 95%

Using these values, AB Test Guide will give you an estimate duration required for your test but it’s always important to verify that the test will last at least 1 sales cycle and should ideally last for 2.

In general, it’s always best not to set a test to run for any longer than 30 days, with 60 days being the absolute maximum. Running a test for more than 3 months is risky because many of the parameters will change over such a long period (on average, 10% of internet users erase their cookies once a month, which will contaminate your test results).

pdf_icon

AB Testing Sample Size: The 4 Levels of Difficulty

Click here to download a PDF copy of this article Save to read later or share with a colleague.

So now you know the AB Testing Sample Size required for you to carry out an effective AB test on your site but what happens if the amount of visitors required is larger than your actual traffic?

AB testing sample Size: 4 levels of difficulty

The graph below can help you to estimate the level of difficulty related to the AB Testing sample size required to carry out reliable AB tests.

In other words: what is the minimum increase in conversion rate needed to be confident that test results are reliable in relation to the number of unique monthly visitors?

We have fixed certain parameters for this graph:

  • Site conversion rate: 2%
  • Test duration: 30 days (you should NOT test longer: Cookie deletion will make your results unreliable)
  • 1 control page (A) and 1 variation for testing (B)
  • Confidence level: 95%
  • Power: 80%

Visitors are organised in to 4 zones that represent the difficulty level, or in other words the increase in conversion that would be required for an accurate result, for reaching an Uplift (an increase in conversion rate).

AB Testing Sample Size - 4 Levels of Difficulty

What Are The 4 difficulty levels?

Make sure to always consider the data of the page tested rather than that from the site as a whole.

With less than 10,000 visitors a month, AB testing will be very unreliable because it’s necessary to improve the conversion rate by more than 30% in order to have a “winning” variation.

With between 10,000 and 100,000 visitors a month, AB testing can be a real challenge as an improvement in conversion rate of at least 9% is needed to be reliable.

With between 100,000 and 1,000,000 visitors a month, we’re entering the “Exciting” zone: it’s necessary to improve the conversion rate by between 2% and 9%, depending on the number of visitors.

Beyond one million visitors a month, we’re in the “Safe” zone, which allows us to carry out a number of iterative tests.

What should you do if your traffic is below the required AB Testing Sample Size?

It is really worthwhile doing AB tests in order to validate hypotheses so, even if your traffic is low, it’s important to still find a way of testing at least certain elements.

If your traffic is less than 10,000 visitors a month, however, it’s going to be very difficult to optimise using “iterative” methods.

Here are two things you should be doing in parallel:

  1. Trying to increase your traffic!
  2. Working on optimising your site:
    • solidify your USPs: why visitors should buy from you rather than your competitors, and why they should do it now!
    • test functionality: does your site work well on all browsers, devices, etc.
    • gain a better understanding of your visitors and customers (mini-surveys, user testing,…).

In the 10,000 to 100,000 visitors a month zone, it’s still quite difficult: iterative tests aren’t very powerful and so it becomes necessary to create “innovative” tests in order to obtain reliable results. Using Neuroscience and Consumer Behavioural techniques can help to boost a test’s performance on top of those elements mentioned above.

Beware of micro conversions!

Micro conversions are steps that occur during your conversion funnel, for example when someone adds a product to their basket, but the final conversion is still making the sale (which is a macro conversion).

AB Testing Sample Size -  Roberval Effect
Effet Roberval  (© Convertize)

It’s pretty easy to increase the rate of micro conversions and so it can be tempting to try and base tests on these. However, you must always also measure the impact the test is having on macro conversions to have an objective understanding of the true performance of your tests.

If you optimise “too much” on one side, then it could result in a decrease in conversion on the other side: for example, if you over-optimise one page through the use of motivating messages but then the promised incentive isn’t realised on the following pages, your overall conversion rate is likely to drop. This is called the Roberval Balance.

Philippe Aimé Philippe’s bonus tip:
Want to learn more about successful A/B testing?

You can read more about A/B Testing on our blog, take the guesswork out of testing with our suggested conversion rate tactics and automatically deliver winning pages with Convertize Autopilot™.

In conclusion

AB testing is an important technique for validating ideas for increasing conversion but it’s necessary to know the AB Testing sample size for the page you want to test.

Subscribing to an AB testing solution doesn’t guarantee results because often the main objectives of these is to sell you a 12 month subscription which can lead to certain misinformation and confusion about results given. The best way to optimise your conversions is to have wider knowledge and support in order to make informed and effective changes.

Philippe Aimé

by Philippe Aimé

Philippe is the CEO of Convertize. He is based in London. Philippe created his first website in 1998, and has worked many years as a Cost Optimisation Consultant for several important groups. He now heads a team of Conversions Optimisation consultants.