Why A/B Testing is Suspect – Read This Before Signing Up

why you should question a/b testing

If you’ve been in the market for digital marketing services, you’ve probably come across a few A/B Testing (aka: Conversion Optimization) sales pitches. These two dynamite-drop-in terms seems to be all the rage for those offering internet marketing services lately; mainly by those who have no clue what they are doing.

Why is this? Because A/B Testing is:

  1. Easy to explain and understand
  2. Makes you sound like an expert regardless if you are or not

Let me give you a quick rundown about A/B Testing to help you decide if it is right for your business or not.

What is A/B Testing

A/B Testing is when you compare one version of a webpage to another*. More specifically, how Version A converts** website visitors compared to Version B. You keep the winner, and, as a result, optimize your conversions.

The difference between Version A and Version B may or may not vary much. Variances could be anything from rephrasing a headline to changing the color of a button to a completely different page design. All depends on what you want to test.

Why A/B Testing is Easy to Sell

It is this easy… “By changing a single word on your website we increased your conversions by 20% in just 1 week!”

Version A

Free Evaluation

Sales 20%

Version B

Free Consultation

Sales 0%

You get it. I get it. We all get it.

You’d have to be a genius to pull something like this off, right?! And, of course, you’ll be willing to shell out more cash because they just made you 20% more!

I see this almost exact same pitch on Linkedin and Facebook posts all the time. I’ll bet people buy it. You should do your homework before doing so… here’s why.

Why A/B Testing is Suspect

Here is a example as to why A/B Testing may have little to do with conversions and/or sales.

Date Visitors Calls Sales Revenue
May 23 958 143 40 $12,272
May 30 1,048 138 34 $9,256
Jun 6 868 144 26 $7,356
Note: This is actual client data from 2017

Few comments:

  • All dates are Tuesdays, for a week-to-week comparison.
  • In this case, the website’s goal is to drive calls that lead to sales.
  • Jun 6 had the fewest visitors, the most calls, the lowest revenue
  • Notice the ~$5,000 difference in revenue in just two weeks time

So what changed A/B Testing wise? Nothing.

In this case, the issue was consumer behavior; one of several unpredictable variables that can throw a serious wrench into your A/B testing data. You could end up making change after change that did nothing but waste your time, energy, and money.

Does A/B Testing Ever Work

Yes it can work… but we only recommend trying A/B testing for high traffic / transaction sites.

If you’re a small business and/or do not have many visitors or transactions on your site, you should focus your conversion-optimization efforts elsewhere – and/or – try to increase your website’s overall traffic.

And certainly do not fall for a sales pitch like the one above.

*The term A/B testing may also be used to compare different ad sets, mobile applications, etc. For the purposes of this article, we are only refering to webpage A/B testing.

**A conversion is when a website visitor executes the goal of the web page. This could be anything from filling out a contact form, signing up for a newsletter, downloading an app, or making a purchase on an ecommerce site.