Why A/B Testing Sucks

I’ll admit, I jumped on the A/B testing bandwagon early last year.  I mean it was hard not to.  Every session of every conference I went to included one if not both of the following lines…

“Raise your hand if you are testing”

“I don’t know, just test it”

Everybody exclaimed “Just jump in and get started.  Everybody is doin’ it maaan.”  So, I started testing and things were going great.  I was making incremental improvements and seeing a positive difference in bounce/conversion rates.   For the purpose of my testing, a conversion was to get people onto a product detail page, not necessarily an actual order as it might take like 10 years to get to statistical significance (mistake numero uno).  I was testing edits to my home page to help facilitate this goal.  Change the color of this.  Move that  box up there.  Tweak the copy here.  Great stuff!  I became obsessed with numbers (mistake numero dos).  As soon as I started work in the morning I’d check the race to see if the new version was winning.  I had a purpose.  A clear goal.  My life was meaningful and I couldn’t get enough of it.  Before the test had concluded I was already trying to think of what I was going to change next.

After a few months of this I was able to show the boss man, look, we’re getting better, see this graph.  Proof positive.  Pat on the back.  A good ol’ ‘atta boy.  Keep up the good work.  I guess I was beginning to feel some “testing fatigue” at this point and I took a break for a week or so.  During this time, I took a step back and looked at my site and said “man, this looks like shit”.  I promptly put on my a few of my persona (good ol’ marketing) hats and and come away utterly confused.  How could my website be worse than when I started.  I have statistical proof that this is better!?

Mistake One – Test what you’re really after, not what you think is a proxy for it just because it will get you an answer quicker.  My goal in life had become trying to get people to my product detail page and it failed me.  To increase the conversion rate I had shortened my funnel taking visitors right from prominent links in the top left of my home page to the product detail page.  No category page, no explanation of who we are or what we do, no confidence builders, nothing in-between.   When my visitors reached the detail page, they were confused and actually bought less.

Mistake Two – Don’t waste time getting bogged down in the minutia of testing just for the gratification of 1% better conversion.  If you’re website site is processing thousands of orders a day, this tip is not for you as 1% might be meaningful and you probably have the man power to devote to this.  I on the other hand, do not process that many orders and am really doing this thing solo.  If I spend two hours a day checking my horse in the race, thinking up new tests and reflecting on statistics … that’s two less hours of link building, content creation, etc.  In my case, small tests are not worth it as they usually produce small increases in conversion.

So, after reflecting on my website post “A/B Day,” I went back to marketing, design and code basics.  I put on my persona hats and asked, what do I want to know, right now.  The result was a 150% increase in revenue and countless hours that I can devote to other, more productive endeavors.

Now I know a lot of you testing fan boys out there are going to flame me saying that I did it wrong and you’ll show graphs and throw out huge percentage gains but I don’t care.  Take your testing and cram it.  I’ll trust my marketing intuition over some cold statistics any day.