The complete guide to User Testing - Chapter 4

A/B testing tools: the top 14 in the game

list of top a/b testing tools

A/B testing tools are all about letting the numbers speak for themselves. But which A/B testing tool is right for you?

UX designers are all familiar with the idea of user or usability testing. But once that website is up and running, does that mean it’s the end of testing for us? The answer is a definite no.

Just because something is online doesn’t mean the work is done. Much to the contrary, now is the time to carry out experiments to see what can be improved. The name of this game is conversion and the things designers can do to increase it. And that is when A/B testing tools become handy.

Start prototyping websites that convert with Justinmind
Download Free

Conversion can mean just about anything – from carrying out payment to a purchase to simply clicking a subscribe button. How does a change in this element impact the number of users who went on to click the button? Does this color make users more or less likely to click it? A/B testing will help you find out: all you need is the right tool.

Why you should care about A/B testing

UX design is many things. It touches on interface design, navigation, interaction, visual design and many more. And so, it can be really challenging to deliver a final design that ticks all these boxes. How do we know what makes a successful design? Which way is up? Was that last change an actual improvement – or was it just a change?

why it matters: a/b testing tools

The thing about UX design is that it can be subjective. In truth, what makes a good UX design is entirely up to the user. If the user dislikes or doesn’t fully understand something, it immediately becomes an issue for the designer – even if that particular area of the prototype is great by designers’ standards.

The people creating and building products need the ability to put themselves in the shoes of users. And while most of us have empathy on some level, truly seeing things from the point of view of another individual is incredibly difficult. It’s quite often that design teams are surprised when they test a particular component or screen to find that users don’t approve of it.

Each user has different standards, preferences and tastes. Finding a way to please most users can be quite like sailing across the pacific with no GPS, not even the stars to help guide you.

That is the main reason why design teams will invest lots of time and effort into usability testing, user testing… and A/B testing tools. These are all the tools at the design team’s disposal to make sure they understand what the user wants and needs. You need to validate everything, to know anything.

A/B testing: how it works and why it’s special

A/B testing is all about comparison. While other forms of testing focus on certain aspects of the design, A/B testing is quite specific: two different versions of the same design, compared.

The advantages of running an A/B test are usually quite clear. The test points out areas for improvement and possible mistakes made in the design. It shows the right direction, with the help of users. Its main goal is to push conversion of the website until it reaches optimal levels.

Lots of different metrics contribute to good conversion. Generally, these are more familiar to marketers than to designers, but in the end it all comes down to the same point: to improve on the design until conversion is where we want it to be.

But how do we go about that? If your website is already online, you want to get as much data as you can – to find out where you are, before you choose a destination.

Once you get a clear picture of the website’s performance, you can start creating hypotheses as to what does well and what could do better. It’s important you have these hypotheses, as they will give direction to your research and testing. You’ll know which areas to focus on, what you want to improve, and what you hope to achieve with the optimization.

Remember: Good A/B testing tools help you harness information regarding how users behave, laying all the data down at your feet. The interpretation of that data, however, is up to you.

Once you have your initial data and your hypothesis, you can select what variable you’ll put to the test. It’s important you have only one variable, which means there can only be one discrepancy between the two versions.

a/b testing tools for conversion optimization

This is for the simple reason that if you make many changes, it will be impossible to know which one is responsible for the changes in user behavior. You want to have total control over the variable, so you have total trust in the data that arises from the test.

All this planning will result in two different versions of the same product. A/B testing tools show you the data on how users reacted and interacted with both of these versions – which you compare, and draw conclusions from.

Top A/B testing tools to rock your website conversion

A/B testing is enjoying more and more popularity over time, due to how effective it can be in pointing out what works with your audience. With its growing popularity comes a growing number of A/B testing tools out there that enjoy all sorts of features – but which one is the one for you?

How we know all this: For this post, we took a close look at reviews from users from each respective tool on G2. For the full list of comments and reviews, do check out their platform.

Check out these amazing A/B testing tools and start making plans for the optimization of your website.

1. VWO

VWO may be the biggest A/B testing tool in the market. It’s grown to be massively popular, particularly among large companies such as Target and eBay – and it’s easy to see why. Mostly aimed at professionals on the business side of digital products, it’s interface is simplicity itself.

popular among a/b testing tools: VWO

VWO enjoys an intuitive interface, with a simple drag and drop editor that should represent a quick learning curve for designers. Furthermore,  the platform makes it easy to draw conclusions with the way it presents data, with its SmartStats feature.

The VWO blog is full of interesting and educational content on optimization. It touches on design theories but also on the business side of optimizing a website, offering something educational for everybody.

Additionally, you can couple their extensive A/B testing tool with other subscription packages that are very powerful in nature. With the VWO Insights package, for example, designers can actually see recordings of users – their exact behavior compared between the two designs.

The set up process for VWO is quite simple. In the tool, you will create your actual test – complete with traffic allocation and goal setting. Once everything’s ready in the tool, you only need to copy and paste a bit of code into the pages in your website that will be tested. Remember to add your account_id in the code, and you should be ready to test some pages!

  • Price point: from $199/month for the testing package.

2. Optimizely

Another giant in the A/B testing tools field. Together with VWO, Optimizely is a favorite among users from all sorts of backgrounds. Their tool is meant to eliminate guessing altogether from web products, and they offer users all they need to do just that.

a/b testing tool: optimizely

The platform is made specifically for marketing and product management teams, giving users all they could possibly need to run tests and discover which way is up, at every step. Optimizely also offers a great alternative for large companies, as their business plan helps users establish authority and roles for other team members.

The potential: By using Optimizely wisely, folks at HP pushed the performance of a page on their website up by 20%! That meant an additional $260K per quarter. You can check the case study for more information.

The set up for Optimizely is pretty straightforward. You go into the tool, where you’ll find one line of code for each project you create in Optimizely. The good thing is that the code already includes your account ID and updates automatically. This means all you need to do is copy and paste that line of code into the <head> tag of your website. You can find more information on the snippet code on their website.

  • Price point: available upon request

3. A/B Tasty

Both our previous A/B testing tools were very well-suited for large enterprises, but A/B Tasty seems to be a good fit for medium and small companies.

According to reviews by users, the platform itself works well – but the real star is their customer service. Putting the client first is the only way things are done over there, and users are thrilled. Sometimes, A/B testing tools need a human touch.

a/b testing tools: a/b tasty

Of course, no A/B testing tool is perfect. Users seem to believe their cookies setting or the global code to set tracking to be unnecessarily difficult, which does represent a few areas for improvement. But even with those recurrent complaints, users would still recommend it to a friend – which says a lot about how much users rely on the platform.

To set up A/B Tasty in your website, you’ll need to go into the tool and make your way to “Tags implementation”, and then to “Generic tag”. There, You’ll find the line of code you need to copy and paste into your <head> tag. The further up in the code you paste it the better, as your website will display the variation more quickly – avoiding that users see the original version for a split second.

A/B tasty asks that users put this code into every page of the website – even the ones that don’t participate in the testing. This allows the tool to be more accurate in calculating metrics such as the bounce rate.

  • Price point: available upon request

4. Convert

Among A/B testing tools, being simple and easy to use is a major selling point. But what about those who have a lot of technical knowledge, and need to be able to get deeply technical in their tests? Well, it seems those users flock over to Convert.

Users can, for example, filter their tests according to JavaScript events just like they would filter according to geographical location. Convert allows you to recreate any page in their platform, excluding the need for you to inject code into the actual product.

a/b testing tool: convert

Users seem eager to recommend Convert due to its technical capabilities, customer support and overall features. Some users, however, do warn that the interface isn’t always clean or completely intuitive – but most tools that offer a more technical side struggle with those issues. All in all, users were happy with the tool!

Start prototyping web and mobile apps with Justinmind
Download Free

To set up Convert in your website, you need to first create a test or a project. Much like other testing tools in this list, you’ll be offered one code snippet per project. Go to your project in the tool, and head to “Project settings”. There, you’ll find a section dedicated to the Convert tracking code.

Once you have the code, you just need to copy and paste it into the page that will be tested. Convert needs the code to go before any other loaded content, so right after the <title> works best.

Price point: starts at $599/month with the annual Essential plan

5. Omniconvert

Among A/B testing tools, Omniconvert seems to offer a good balance between what users need and what they want. The platform itself has very good reviews from users in terms of how complete it is, and the different ways users can apply it to their work.

Like many A/B testing tools on this list, it offers additional features that can be interesting for designers. For example, users can go from email opt-ins, to personalization and surveys in one platform. Users are also quick to praise their customer service for their quick responses and overall ability to solve problems as they come.

a/b testing tool: omniconvert

Something worth mentioning is that the platform works with pre-established templates that users can edit to represent the ideas they want to test. However, some users have claimed that makes their life a little more difficult, as some changes simply cannot be made within the interface.

If coding is not a skill you can implement on these tests, this might not be the tool for you.

Omniconvert makes it easy for users by putting the Tracking code settings right on the dashboard of the tool. From there, you can find the Synchronous code (better suited for A/B testing). Simply copy and paste it into the <head> tag of your website. That’s it!

  • Price point: The price varies according to the views you want to test. The minimum is 100k views, which goes for 324$/month for the Platform plan. Enterprise plans are also available.

6. Freshmarketer

Freshmarketer is among the A/B testing tools that embodies a lot of different marketing-oriented activities – but those can come in handy for designers too. Users swear by the platform’s sales funnel feature, as well as its dashboard design – making it easy to manage all current activities being carried out. For A/B testing tools, the dashboard is crucial!

Another huge positive is the many different integration and extension options available for users, ranging from Google Analytics, Google Chrome to WordPress and Kiss Metrics. We particularly enjoy their dynamic heat maps!

a/b testing tool: freshmarketer

Among the negatives from users, we find comments on a rather steep learning curve if users hope to make the most out of it. Small issues, such as heat maps not being stored within the tool once any changes are made, are also to be found.

Freshmarketer has its tracking code in the Setup page, behind the link “Freshmarketer JS code”. Opt for the Synchronous version of the code and simply copy the whole thing. You have a button on the same page, so it’s easy to send this code to your developers, should you want to.

Simply paste it into the <head> tag of the website. Once that’s done, you won’t need to update the code for new tests. The tool also gives you a verification option, so you can be sure that the code works properly. To verify, just go the Tracking code page, and paste the URL of the webpage in the Verify Installation input.

  • Price point: starts at 49$/month for the Garden plan (for up to 1,000 contacts)

7. Crazy Egg

Crazy Egg might be a familiar name to many readers out there. The tool is popular due to their heat maps and recordings of user behavior on websites – but their testing abilities got them on this list of A/B testing tools.

Crazy Egg offers a simplistic version of A/B testing, which doesn’t get as technical or as deep as some of the tools on this list. That is perfectly fine – sometimes you just don’t need all those features. Sometimes, you need something that gets straight to the point and Crazy Egg does that very well.

a/b testing tool: crazy egg

No to mention that Crazy Egg is integrated with a wide variety of tools… including Justinmind! We love it when combining forces becomes a reality in the UX game. If you’re wondering what integrations we have at your disposal, do check out the full list of Justinmind integrations.

Some users do wish some changes be made, such as the option to subscribe and pay for the service annually, rather than monthly. Another common user observation is that the heat maps experience some issues when it comes to parallax pages – an issue found throughout the heat map tool sector.

Upside: Their pricing varies a lot, which leaves a certain margin for all sorts of professionals to use the tool, no matter what the budget looks like.

To set it up, just go to “Install Crazy Egg” page. You can get there directly from the dashboard. Head over to “I can do it myself” – you’ll be sent to a page with the code. As is the norm with A/B testing, just paste that code into the top part of your webpage’s source code.

Crazy Egg also offers the possibility to check your installation to make sure it’s all working. Just go to the Install Crazy Egg page, where you’ll find the checking option. From there, just add the URL of the page in question to verify. Easy!

  • Price point: starts at 24$/month for the Basic plan (includes 30k tracked pageviews/month)

8. Instapage

Like many A/B testing tools, Instapage is a landing page builder. Luckily for us, it also offers some great A/B testing and heat maps for those landing pages! It wouldn’t be a stretch to imagine building pages with Instapage, and then building them for real using a proper prototyping tool.

Users are quite happy with the level of freedom Instage page gives them when it comes to designing and building the actual pages, as well as the quick ability to create separate tests. Other users are less happy with the interface design, although it remains intuitive throughout.

a/b testing tool: instapage

An interesting comment by a user pointed out an important concern when it comes to A/B testing tools: granting permissions to users. Instapage doesn’t actually have a system of roles and permission for teams in the platform.

Designers who work within larger teams will be painfully aware that having control over who can do what is crucial – different people can make changes that wouldn’t have been approved by the rest of the team. An easy example would be designers making changes that cannot be implemented by developers, or marketers changing things that compromise the work of designers.

Instapage asks users to do something that differs slightly from the rest of the tools in this list. Because they are constantly expanding, they ask that users with a naked domain create a 301 redirect. This is because the tool needs tested webpages to point to their own servers – it avoids broken pages when Instapage adds more servers to their network. Check out their article on setting up a naked domain for more details.

  • Price point: 149$ for their Business pan, billed annually

Marketing tools with integrated A/B testing

These aren’t necessarily A/B testing tools, but rather digital marketing tools. These tools are commonly used by marketers in order to not just do things like send out emails, but also to carry out A/B tests to keep improving on the conversion.

These tools are a great resource for running simple but constant tests. These are likely to work well with more advanced tests such as multivariate tests – but they can be a good ally for slow but constant improvement.

Email marketing

9. Mailchimp

Of course. The biggest email marketing platform in the sector offers all its users A/B testing for their email campaigns. The idea remains the same: you create variations of your email, and Mailchimp sends different combinations to a small representative group of contacts.

mailchimp as an a/b testing tool

The cool thing is that you don’t need to do anything when it comes to setting up the tool. Another huge plus is that you can simply establish which criteria Mailchimp will use to identify the winner (example: opening rate) – the tool will then automatically send the winning version to the rest of your contacts.

  • Price point: starts at 9.99$/month (free version doesn’t include A/B testing)

10. Campaign Monitor

Much like Mailchimp, Campaign Monitor is all about creating awesome emails that convert users. Aside from the classic features of email marketing tools such as having templates or the possibility to customize the email per contact, the tool also offers an A/B testing feature.

campaign monitor as a testing tool

Since you won’t be testing webpages, there is no set up necessary. You can just create emails, design variations and test them against each other. The A/B testing feature is considered a core feature, so it is included in all their plans.

  • Price point: starts at 9$/month for up to 500 contacts

Landing page builders

11. Unbounce

While many UX designers may be hesitant to use a simple drag and drop landing page builder for their tests, we shouldn’t take them for granted. Unbounce is a wonderful example of a landing page builder that adds value to our toolbox, by offering us a quick and easy route to new pages.

unbounce as a/b testing tool

Unbounce may not be our sole way of creating new pages, but it does open the door for quick new pages that live short lives. The tool itself focuses on building landing pages, popups and email marketing – and true to the standards of the sector, it offers A/B testing as a classic feature. Running tests on the fly just got so much easier!

  • Price point: starts at 79$/month for the essential plan

12. Leadpages

Leadpages goes beyond just a landing page builder, going into the realm of creating whole websites without touching any code. While that feature may be better suited for non-UX professionals, the tool does offer a lot of flexibility when it comes to creating landing pages.

Much like Unbounce, Leadpages can be a tool for quick landing pages or fast conversion studies. The tool offers split testing as well as automatic guidance on the general usability of your landing pages.

leadpages as testing tool

It is worth mentioning that Leadpages doesn’t include split testing in their most basic plan (Standard plan). That means users who seek split testing will need to choose their second cheapest plan.

  • Price point: starts at 79$/month for the PRO plan (billed monthly)

13. Getresponse

Another marketing tool in this list that offers landing pages, email marketing and even a webinar solution that uses email marketing to deliver webinars. Like Leadpages, Getresponse considers A/B testing to be a classic feature – which means it’s included even in the basic plan!

getresponse as a/b testing tool

Getresponse includes unlimited landing pages in all their plans, as well as unlimited lead funnels. Their price varies according to how many contacts are in your list (if you intend on using the tool for email marketing) – the lowest price applies to lists of up to 1,000 contacts.

  • Price point: starts at 15$/month

14. Wishpond

Another cool landing page builder with A/B testing features. We love that Wishpond manages to check all the classic boxes in the sector, like a drag and drop interface, while also offering more rare features – like their built in popup maker.

testing tool - wishpond

Unlike Getresponse, Wishpond only offers split testing to users with the “Everything you need” plan – as opposed to their cheapest plan, “Starting out”.

  • Price point: starts at 99$/monthly for the “Everything you need” plan, with annual billing

The wrap up: A/B testing tools are a must

A/B testing tools have a lot to offer designers. They go straight into the heart of digital business: conversion. Whereas for marketers, conversion is a prime concern from the start, designers are all about offering the best experience – which is why A/B testing tools are so valuable.

They help designers understand a business side of UX that may not necessarily come naturally to them. A/B testing tools should make the testing itself easy on you, while offering the data in the best way possible.

The information you get from these tools needs to be reported in a way that you can digest, allowing you and the team to draw the right conclusions from the tests. A/B testing tools need to have the documentation and representation of statistics down, otherwise their whole purpose may be compromised.

PROTOTYPE · COMMUNICATE · VALIDATE
ALL-IN-ONE PROTOTYPING TOOL FOR WEB AND MOBILE APPS
Rebeca Costa
In-house writer, usability enthusiast and patron of all sleep-deprived designers