A/B Testing: Stop Guessing, Start Testing

A/B testing is one of the most effective ways to increase conversion rates available to an eCommerce business. As you know, with eCommerce there is always room for improvement. There is no end to the things you can tweak and test on your site, and one small change can lead to drastically improved conversions. There is no one-size-fits-all testing approach, but rather it is all down to the way each unique website and its customers functions.

Running tests is a tried and approved method, Amazon itself owes a huge amount of its success to running tests. As Jeff Bezos said himself; “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.” Bezos’ insistence on constant testing, experimentation and innovation led Amazon to features such as; one-click ordering, two day and free shipping, and personalized recommendations. These improvements didn’t come from nowhere; they are as a result of intense variant testing, and implemented action based on reliable results.

A/B testing means making the right changes, at the right time. Let’s take a closer look.

What is A/B Testing?

A/B testing, also known as ‘Split Testing’ is a scientific way of arriving at the best option given a set of conditions. A change is made to a website, and tested against the original version to see which creates the most positive outcome from the audience based on the original intention.

It means testing two versions of a web page to see which produces the best results. The audience is split into two segments; A and B, and the results will prove which version is more successful. The variable that has been changed will be measured by audience interaction, creating real time feedback and insights that can be used to improve the overall quality of the website.

Why Should I A/B Test?

A/B testing removes the guesswork from optimization. It means that any changes made to the user-experience are only implemented after data has been collected confirming that this is the right decision. Shoppers’ needs are able to be understood without any coding or integrations. These kinds of tangible insights into customer preferences are hugely important.

This method can save a business time and money, as it’s not every part of the business that needs to be tested. Just the key elements are tried, and therefore less resources are spent whilst more traction with the customers is gained.

From an HR perspective, the data-driven aspect of A/B testing means that personal opinions or ‘gut feelings’ within a business are irrelevant. Here, the data speaks for itself and opinions, positions or salary doesn’t make a difference. The figures lead the way. 

The overall benefits of A/B testing are huge:

• Increased conversion rates
• Improved user experience
• Increased email opening rate
• Increased purchase frequency rate
• Data-driven decision making
• Reduced customer acquisition costs
• Removed mystery from marketing

In short, everything that your eCommerce business strives for; A/B testing holds the key to. Seems like a pretty good reason to do A/B testing then.



How Do I A/B Test?

When starting out with A/B tests, a good idea is to choose 1-3 objectives that you as a business would like to focus on. These can be based on product roadmaps, and should be ranked in order of priority.

There are five steps to successful A/B testing:

1. Research

Before the tests can begin, it’s important to understand how your brand is doing at the moment. Being clear on the brand’s current performance is key in order to know what changes to make. The current performance metrics and key performance indicators should be analyzed, and compared with previous goals or indicators.

2. Hypothesis

Once the results have been viewed, it should be possible to see the gap between the goals and the results. The hypothesis should state the intended goal; what is the aim that your business actually wants to achieve? When setting hypothesis parameters, one variable should be chosen that you want to view. With one variable, you can see what is driving the changes. If this variable change makes no difference, you can try something else. At this stage you can also determine the length of time to run the test, which audience to expose the campaign to and a breakdown of who is in each audience group. Viewers should be randomly assigned to both groups, with a 50/50 breakdown in each. This is so that there are no biases and the results will be objective. 

3. Create Variable Design

Since [A] is the design already present on the website, the variable [B] will need to be designed. This can be through small changes, edited details to the original design, or creating a secondary design. This will be what is tested on the audience.

4. Run the A/B Test

It’s important to make sure that the test runs for a suitable amount of time. If it’s too short, it might not produce any results. The average daily and monthly visitors, estimated existing conversion rate, minimum improvement in conversion expected and percentage of visitors included in the test should all be kept in consideration here.

5. Analyze Results

Once the results are in, these should be analyzed and measured against the hypothesis put forward. Then changes can be made according to whatever produced the best results. 

This process can then continue to be repeated, as the effectiveness of the website goals and aims can be measured and monitored. 

Where Should I A/B Test?

There is no one answer to this question; A/B testing can be run all over the website. For example:

• Personalization: testing different personalization strategies means you know which ones will produce the optimal results 
• Merchandising: this can apply to rules and collection merchandising, or search merchandising, e.g. visual layout and colors
• Copy & Wording: changing a few words can have a huge difference
• Pricing and promotions
• Landing pages
• Emails and marketing campaigns



When Should I A/B Test?

A/B tests should not just run when figures drop, they should be run constantly if you want your site to be continuously optimized to its maximum potential. This is because what once yielded the best results may not continuously do that over time; customer behavior changes, trends change and site performance changes. Your business needs to have the knowledge and flexibility to adapt and change with this.

Other than that, there are some times when it can come in particularly useful, for example when the website is launching a new feature or redesigning or relaunching something. This can ensure that any changes will be positive; with all the guesswork removed, and no risks taken.

A/B testing is also a great way of understanding why there might be sudden changes in the performance of a certain page or feature on your website. If there is something problematic occurring (such as the color, content, layout) A/B testing is a great way of uncovering what exactly the problem is and how to fix it. 

It can also be useful in helping to decrease bounce rate, for example if there are too many options, or a message mismatch or not enough clarity, this will be able to be identified and changed. If there is a low converting channel, for example through a landing page problem, A/B testing can help identify this.

Best Practices

On average an A/B test should run for at least 2 weeks in order to get the best results, but it could run for up to 2 months. The longer the time the more accurate the results will be. Using as large a sample size as possible will also produce more reliable results.

Before conducting the A/B test, conduct an A/A test. If the results are noticeably different between the two, something is wrong with the setup. 

Don’t jump to conclusions! Patience is key when it comes to A/B testing. Putting wishful thinking aside, look at the results objectively, and the longer you can wait, the more accurate and reliable the outcomes will be.

Finally, don’t stop optimizing. The more customers you have, the more valuable the insights will be, and the more data collected the better the website performance overall. Past results don’t determine future results, in the world of eCommerce things change rapidly, and you want to be on top of this.

Overall, A/B testing is an amazingly helpful tool for your business. If it isn’t already set as a business goal, it should be a main one. It can have long term, significant impacts on your business and the results that it produces. Increase in conversions, decrease in bounce rates, increase in customer satisfaction and improvement in brand image are all areas that can benefit. 

Without A/B testing, your eCommerce performance is a game of risks and chances. With A/B testing, the game you’re playing is that of assured optimization and success.