A/B Split Testing Demystified

Are there any Seth Godin readers out there? All Marketers Are Liars, Purple Cow, The Big Red Fez do any of those best selling titles ring a bell? Anyway the guy is tuned in and he can move some serious traffic around the web when he wants to. Last week I was both thrilled and discouraged after reading his Where are the tweakers? post.

Target: squidoo.com

Sure he had totally validated my Quickly Redesign offering and the expense of my recent MEC Certification Course but he had also mobilized an army of freelance Web designers under the banner of Tweaker.

Skip to the "Before & After"

What is a Tweaker

A tweaker is someone that stays up for weeks while ingesting large quantities of methamphetamines… no wait a tweaker is someone that makes small incremental changes to a Web page in an effort to increase conversions; he is an A/B slit tester.

The tweaker concept is very powerful, at its core it is all about optimizing the traffic you already have, it’s about turning a landing page that had been converting at 1% into a page that converts at 2% by getting creative with the headline. Yes that’s double your conversions with no additional traffic; I have seen it happen.

Dr. Flint McGlaughlin describes it this way, "Testing is marketing."

The Inspiration

Since Seth Godin was the inspiration for this post let’s use his site squidoo.com to illustrate.

The following is fiction: The other day Seth came to me and he said, "Jake, I get crazy traffic to my Sign Up page but my conversions to Step Two are abysmal. Can you help?"

After spending some time looking over the page I called Seth back, "Seth, I looked over your page and didn’t see any low hanging fruit. If you are serious about doing this I think we need to get radical with the redesign, test it for a week or two and then fine tune it from there." Seth gave me the go ahead, time to formulate my test plan.

Building the Test Plan

Every good test plan starts with a question. Seth’s question gives us a great jumping off point, "How can I get more users through to the second step of my Sign Up process." With this question in mind let’s look at the page and find the largest group of variables available for our test.

The page doesn’t consist of many variable clusters, which is what you want for a sign up page. Beyond the header which we are not going to touch there is the value proposition which is weak, we can fix that later because the real problem here is the length and complexity of the form itself.

figure 1

After speaking with Seth he agreed that we could pick up First and Last Name during the next step, he also agreed that we don’t need to subject our users to retyping their email address and that however techy cool the security phase is over kill. After all we make them validate the address after sign up anyway. (side note: I actually aborted my first Squidoo Sign Up because I couldn’t get past an illegible security word.)

In just a few minutes we removed huge friction points from this vital first step of the sign up process and are setting this page up to be much better at getting our foot in the door with our users. Even if they drop out after this step we now have an address to ping and bringing form droppers back into the fold is something we can test later.

The Treatment

With the go ahead on the structural changes our next step is to refine our test question. We need to change our "How" to a "Which."

"Which landing page had a higher rate of conversion?"

After we have nailed down our test question we need to formulate test treatments. The treatments are the one or many new landing pages that we are going to create for the test; this is where the A/B comes in. Each treatment page will be optimized in its layout and messaging to convert our user from Step 1 to Step 2.

figure 1

The Numbers

Now that we have our treatment pages we need to run a bunch of fancy math. This math will take current daily traffic volume and tell us how long we will need to run each treatment for. According to the numbers we need to run each page for a week in order to get reliable results.

The Happiness

Two weeks later we have run our tests and the results look good, Seth was really fired up that we were able to turn his 0.5% conversion rate into a more respectable 1%.

At 1% Seth is now converting 1,500 more users a day without increased traffic. After we were done hugging I told him that although a .5% jump is huge, we can do better. We need to go back now and test the individual elements, the headline, the button text, the value proposition etc. "We can get you even more conversions Seth." Then a small tear of joy began to run down his face.

That’s all for now, view the completed redesign

If you are interested in a Usability Redesign for your site read more about Suimple’s Quickly Redesign Service.

We currently do not offer comment functionality, if you have something to say about this usability review email contact[at]suimple.com and I will add your comments to the piece.