Are you doing A/B testing?

Its easy, its cheap and it could increase the conversion rate on your website significantly. Why then are we not all doing A/B testing?

I seem to keep coming across articles and applications relating to A/B testing recently. For example somebody in the Boagworld Forum mentioned, a brilliant site for reviewing other people’s tests and the benefits they provided.

I was also reading an interview with Tim Ferris in which he talked about the benefits of A/B testing. He went on to talk about how to avoid being overwhelmed with too much data.

“Dont measure too many things,” he says. “People often become overwhelmed with a deluge of data because they’re looking at 1,500 variables. And that can be paralyzing because you end up sitting there looking at your analytics program all day long.”

Good advice.

All of this made me feel kind of guilty because although I know A/B testing is incredibly beneficial I rarely do it on anything other than client websites. This is particularly stupid when you consider that there are some excellent free tools available for A/B testing. Take for example Google website optimiser. It is entirely free and relatively simple to set up if you know how to copy and paste code into HTML.

Google Website Optimizer

Even if you don’t know how to edit HTML there is still nothing to stop you doing some A/B testing. There is a tool called visual website optimiser makes the process even easier and provides an WYSIWYG environment for creating your alternate versions.

visual website optimiser

Why is it then that so many of us do not bother A/B testing websites? I would be interested to hear your thoughts in the comments.

  • is another resource for A/B testing. It’s another visual service, they have tiered pricing and it looks pretty affordable.

    Note: Not an endorsement, I haven’t actually used them myself, but I have heard good things.

  • My main reason is that A/B testing does very little to build up internal design expertise. All it can tell you is WHAT, not WHY: how a design change affected a number in a particular scenario. It doesn’t help you make better decisions in future.

    Learning about users in depth, designing thoughtfully and running face-to-face testing can help you truly understand the domain and userbase, meaning better designs in future. As always, a mix of approaches is best.

  • I don’t know, I think the what v why debate is overblown. If it was possible to develop internal design expertise that resulted in high converting designs every time, then there wouldn’t be any need to A/B test — we’d just knock out super converting designs and be done with it. Plus, the reason why A converts better than B is often hard to fathom — again, it’s testing that matters, not so much the ‘why’ of a given winner.

    Why we don’t A/B test is a mixture of a lack of familiarity, a bit of pride (no need to test, my design MUST be the best!), and a lack of ownership over conversion rates. Most of the time we don’t measure what we do in terms of performance metrics, let alone try and improve it. Easier to fly blind, get sign off, get paid, and move on to the next project, I guess.

    So, it may seem round-about, but until we take performance and web analytics seriously — i.e., have a solid foundation for measuring what we do — I don’t think we’ll be that inclined to do much A/B testing. I intend on doing my bit to change that ;)

  • I agree to a large extent with Luke Stevens above – lots of people build websites with no real goal other than to have a ‘web presence’. If you don’t know what you’re trying to achieve (i.e. what your conversion criterion is) then I guess it doesn’t occur to you to try to measure it or improve it.

    Where I am most puzzled at the lack of A/B testing is among small business owners that use websites to sell their products. In these cases there is a clear conversion criterion (customer adds something to their shopping cart) and a strong motive to increase the conversion rate! Yet no-one seems to be using A/B testing.

    My own theory is that while WordPress, Ebay, Etsy, Shopify etc. has made it very easy for a small online business to get up and running, A/B testing tools still require a fair degree of technical know-how. Even stuff like Visual Website Optimizer requires you to insert Javascript into your web pages. There are plenty of small online business owners who would have no idea how to do this!

    I think that the key to getting people to start using A/B testing (and I admit I am strongly biased; I run an A/B image testing webapp) is to (1) make it easier to get started and (2) concentrate on the benefits of optimizing a single aspect of their site at at time. I concentrate on talking about optimizing product images, since I think this is an easy and intuitive way to get started. In other words, once you show people the benefit of A/B testing with something easy then they will motivate themselves to learn how to use more complicated tools like GWO.

  • I would be curious if anyone has any experience with this WordPress A/B testing plugin:

  • I’ve always been confused as to what elements of the page we should be tesing and in what way? For example, should we change a button but leave all other elements the same?

    The whole topic just really confuses me.

  • A/B testing is really good site and its tools are useful. Thanks for link.

  • It’s amazing to see the ways people use your site. Most of the time it’s exactly the opposite of how you intended.  A/B testing is a perfect way to narrow the margin.

  • It’s amazing to see the ways people use your site. Most of the time it’s exactly the opposite of how you intended.  A/B testing is a perfect way to narrow the margin.

  • It’s amazing to see the ways people use your site. Most of the time it’s exactly the opposite of how you intended.  A/B testing is a perfect way to narrow the margin.

  • Yep A/B testing rocks! I am the Search Director for a large digital agency (not saying which one as who really cares!?) and we definitely see positive ROI from A/B and multi variate testing. On my own personal site i have put together an article on top tips for testing you might find vaguely interesting. 
    Keep up the great podcast guys!

  • verango

    Another A/B testing tool is They are also a visual tester. I dont see any pricing but it looks promising