A/B Testing for your Online store

Many online store owners look to A/B Testing to help them decide on key marketing strategies and techniques to increase conversion. The danger with this approach is if the testing is performed incorrectly, it could majorly impact your online store performance and hit your revenue. In this article, we will cover the different types of A/B testing that's performed so that it can be simple to implement, understand how to interpret the results and what takeaways you might be able to get from the exercise.

A/B Testing explained

Also known as split-testing is the use of 2 versions of the same page, email, or digital media for you to measure the response and which one performs better. This process helps you to perform better through the knowledge of statistical response from your audience on which landing page generates a better call-to-action, social media lead generation, and at times when it comes to product pages - which one converts better in the shopping cart.

How A/B Testing works

Using an existing page as the base/control, replicating the page with ideas on what might enhance the engagement with the audience gets created as a "variant". Splitting the traffic between both pages gives you a fair exposure to both pages, and if the "variant" performs higher in a call-to-action such as a product page "add to basket", or "learn more" on a landing page - you can determine if this is an improvement. If the "variant" outperforms the "control", then it is accepted as the new replacement, and thus the "variant" becomes the new "control", and then you repeat the same experiment on improving the new "control" with a new "variant".

There are other factors to consider in A/B Testing, such as promotion, seasonality, and recent events (like toilet paper sales during a pandemic) - all of these external and internal variables will affect the test results when considering your experiment that will determine your digital marketing strategy. Keep the variables in mind in seeing if there are any of these that can impact the "control" extrapolate the "variant" and bring it back to be normalized for a fair comparison.

A/B/n Testing, and what does it mean

A/B/n testing means that you can have multiple "variants" against a control. If you have a large pool of traffic, you can split if 4-ways with the "control" being 25% of the traffic, and 3 "variants" with 25% of the traffic equally. These are a bit more complicated to analyze since understanding the variations between pages need to be explicit, and capturing the feedback requires some UI/UX design to ensure that the elements are being recognized by the users as well as captured by the testing. This is done also in the likes of marketing research, such as primary research in product design, as well as product acceptance on how focus groups determine what would they be happy to purchase, and at what value. Colors, designs, and user behavior of your screened target audience will have one response, while new potential customers may have a test run and have a different response. Having the right desired action for each of your "variants" will rely on UI/UX and your content marketing with testing tools that record their navigation of the page. Lastly, the analysis of A/B/n testing will require specific criteria on what made the improvement partially, or completely on the page so as to arrive at what was learned for your marketing tactics.

How long should your testing run?

Ideally, for two business cycles - which means at least 2 full weeks, capturing the weekdays and weekends for all tests. Also, make sure there are no "sales" or national holidays within these cycles. You can account for a couple of visits by a new buyer, and the various traffic sources you are reaching with various advertisements or posts. You can also account for newsletters going out to your audience within these two cycles. The test must run in full cycles, and not end part-cycle. End the test once you have reached a minimum of 2 cycles and you have reached your MBE below, and do not let it run any longer than you have to.

Statistical significant threshold

Statistically significant means you have reached a level of enough users through your test. The tools that are installed on the site might have calculated that you have had enough subjects to reach the statistically significant threshold, but your test _must_ still run for the two business cycles. However, if you have not reached this threshold within the two business cycles, then you should consider running it for a third or fourth cycle until it is significant. Regardless of what is being tested, being statistically significant is an important measure to ensure you have enough sampling data to make a good determination of what has worked, and what wasn't working well.

Example: If you have a baseline of 5% conversion rate, and a minimum detectable effect of 8% your sample size for a relative measurement would be 47,127 = (1-B)% of the relative time. Your MBE (Minimum Detectable Effect) will require double the relative measurement for the 2 pages, thus about 100,000 visitors to do the test.

Your sample size either by advertisement, SEO, or Social Media needs to be determined before your test starts. You must make sure the traffic is going to be significantly significant to ensure coverage of the two or more business cycles. If the test does not reach the sampling or the length of the test run, then the test should be invalid.

Why is A/B Testing important

Here is a scenario - you spend $200 on a FB Advertisement, and 20 visitors arrive on your website, 4 of them add to the basket and spend $25 each, with a total sales of $100. Based on $100, say your gross profit is $50, then you lost $150. Alternatively, if your 20 visitors arrive and purchase $50/each and you had 8 of the 20 buying from your online store, you are now looking at $400 in revenue, and with the same profit marking, you are breaking even. This means that you have to have a strong understanding of your advertising, your conversion, and the ability to incentify both an increase in basket value as well as the increase in conversion percentage. This is not factoring in the long-term residual returning customer that will help amortize your advertising spend when they re-engage in purchasing your products and services. Last but not least, if you are looking to measure your landing pages, Google analytics will help you see the sources of traffic on how the A/B Pages are being reached, determining the types of landing pages and how they are being used.

Contrary to popular opinion, Conversion Rate Optimization (CRO) isn't based on A/B testing alone. Strategies in looking at the full customer experience are important to measure and understand the barriers and frustrations of your audience and how you make it easy to navigate, add to basket, and checkout on your online store.

What items should be A/B tested?

While you could be testing every aspect of your site, however, there are key areas that are critical that make a significant impact on your revenue and conversion and become stronger competitively. It deals with qualitative and quantitative research strategies - where you can use A/B Testing for the focus group studies on the qualitative end, and use quantitative studies for price sensitivity and trade-off studies like a Conjoint study. Without turning your online store into a marketing research experiment, placing your questions into these two groups will help determine where and at what point of your marketing funnel you believe that you have to improve your customer engagement. For every page you can lose your customer's interest in your funnel. Keeping it simple, and effective in communication and helping them see the value quickly is important. Landing pages, Product Pages, and Basket/Checkout are key areas to consider.

Conclusion

A/B testing is a bit of a chicken and egg scenario - you need to have enough traffic to do the test, and you have to reach a certain level of success to gain that traffic. However, if you have the traffic and are looking to grow both revenue and conversion with your existing advertising spend, then A/B Testing should be considered to maximize your funnel's conversion rate. If you have any questions regarding the tools and how to engage in A/B Testing, feel free to reach out to wish@thegenielab.com