Website A/B testing explained

    In this post, Bart Albrecht discusses 3 questions about A/B testing: what should you test, how long should you test and which tools should you use?

    Subscribe to our blog

    A/B testing is the practice of running two variants of a website page (or email, display advertisement or social advertisement) in parallel, to test if website design and content choices are hitting the mark. It is a popular topic in digital marketing, since testing is the best way to check which website changes have the greatest effect on your KPIs, be it conversion rate, bounce rate or anything else.

    By testing and adjusting the pages according to your A/B test results, you will be able to fine-tune your pages to the needs of your exact visitors – no guesswork needed.

    When you start out A/B testing it is best to focus on your main KPIs, such as the number of contacts who successfully complete a contact form. In a later stage, you can focus on smaller goals and adjustments.

    The image below (courtesy of Tourque) explains the basic concept of A/B testing very well.

    AB Testing explained

    There are three main questions considering A/B testing I would like to discuss today:

    • What should you test?
    • How long should a test run?
    • What are good tools to run A/B tests with?

    What should you test?

    Before you start testing, you should make very clear what you want to test. If the A and B versions of the test are very different, it will be hard to define which change made the greatest impact.

    To start, focus on small, easy modifications – only one at the time works best:

    • Change the location of a call to action
    • Make the call to action more attractive (change wording or image or color)
    • Simplify a form (remove the items that are less important)
    • Add testimonials
    • Social proof
    • Change headlines or sub headlines

    A/B Testing CTA locations

    Once you’ve been able to fine-tune your pages, you can start thinking about testing and changing website flows. This is more complex, as it involves creating an entire secondary flow parallel to the existing one.

    One question you can come across it when you’re testing contact forms is if you should show all fields on one page or split up the fields across multiple pages. Showing too many fields at once might discourage a visitor from submitting all the info.

    Most of the time, it’s best to start small and split the form in 1 to 3 steps. The first step must be relatively simple to motivate visitors to start filling in the form. In an A/B test scenario, you could, for example, have one form flow with 2 steps and one form flow with 3 steps.

    Another thing you could test, besides simple on-page elements and flows, is behavior-based personalization. If you have visitors who return to your website looking for a specific type of product, you could test if showing that product on your homepage increases the chance of them purchasing it.

    How long should an A/B test run?

    I often see people running A/B tests and cutting them off too soon. Depending on the traffic on your website and the number of conversions, you will need to wait a while before the test results are statistically significant. Very often people stop testing after 2 to 3 days. In most situations that is too early, as visitors are probably used to the old website. In this case, when you change something, people need to adapt and you need to give them time to do that.

    Most tools can already estimate how long it would take before you know when you will have significant results.

    Tools for A/B testing

    In the past, A/B testing was usually manually programmed. That takes development time. Today, that is not necessary anymore as there are really good tools for A/B testing on the market. They help you see when a test is statistically significant, for instance, or remember which version of the test a visitor has seen before.

    Some content management systems have a built-in tool to do A/B testing. Some examples:

    There are also some other good tools like:

    A/B testing, when done right, can be a great help in optimizing your website. The best thing, in my opinion, is that you’re taking out the guesswork. Testing gives you an actual view of what your visitors prefer and you can prove changes are (or aren’t) working with real metrics.

    Download the case study: A new Adobe Experience Manager CMS leads to a close partnership with KBC


    ➡ Read this article in German

    Published on    Last updated on 30/11/2016

    #Digital Marketing, #Digital Strategy

    About the author


    Participate in this discussion