A/B split testing demystified | close
The following is fiction: The other day Seth Godin came to me and he said, "Jake, I get crazy traffic to my Sign Up page but my conversions to Step Two are abysmal. Can you help?" After spending some time looking over the page I called Seth back, "Seth, I looked over your page and didn't see any low hanging fruit. If you are serious about doing this I think we need to get radical with the redesign, test it for a week or two and then fine tune it from there." Seth gave me the go ahead, time to formulate my test plan.
The Test | close
After speaking with Seth he agreed that we could pick up First and Last Name during the next step, he also agreed that we don't need to subject our users to retyping their email address and that however techy cool the security phase is over kill. After all we make them validate the address after sign up anyway. (side note: I actually aborted my first Squidoo Sign Up because I couldn’t get past an illegible security word.)
In just a few minutes we removed huge friction points from this vital first step of the sign up process and are setting this page up to be much better at getting our foot in the door with our users. Even if they drop out after this step we now have an address to ping and bringing form droppers back into the fold is something we can test later.
<< back | next>> The Treatment
The Treatment | close
With the go ahead on the structural changes our next step is to refine our test question. We need to change our "How" to a "Which."
"Which landing page had a higher rate of conversion?"
After we have nailed down our test question we need to formulate test treatments. The treatments are the one or many new landing pages that we are going to create for the test; this is where the A/B comes in. Each treatment page will be optimized in its layout and messaging to convert our user from
<< back | next>> The Numbers
The Numbers | close
Now that we have our treatment pages we just need to run a bunch of fancy math. This math will take current daily traffic volume and tell us how long we will need to run each treatment for. According to the numbers we need to run each page for a week in order to get reliable results.<< back | next>> The Happiness
The Happiness | close
Two weeks later we have run our tests and the results look good, Seth was really fired up that we were able to turn his 0.5% conversion rate into a more respectable 1%. At 1% he is now converting 1,500 more users a day without increased traffic. After we were done hugging I told him that although a .5% jump is huge, we can do better. We need to go back now and test the individual elements, the headline, the button text, the value proposition etc. "We can get you even more conversions Seth." Then a small tear of joy began to run down his face.
If you are interested in a Usability Redesign for your site read more about Suimple's Quickly Redesign Service.
