The ABCs of Split Testing

By now you've probably heard of A/B split testing — or showing your customers different versions of the same pages to see what works — and like most time-strapped e-tailers, aren't sure exactly what's involved and how to tackle it.

When done correctly, A/B can help you increase your conversion rates, address problem pages and challenge your assumptions about how to optimize your site. While such testing can be complex to conduct, if you have command of the basics and stay focused on what you need to do for your small business, you can glean valuable information without too much effort.

Jim Miotke, president of BetterPhoto.com, says split testing is giving him the tangible results he needs to make important business decisions. "We took our best guess at what people would like, what would be a good design, but in retrospect, looking back on it, it's like coding in the dark," said Miotke who launched his site in 1996. "Then the idea came to us that we should do some testing. We had stats and analytics, and it told us some basics, but we didn't know the 'why,' so we looked into A/B split testing. My first attempts were abysmal failures."

Fortunately, he didn't give up, and now he's seeing results. He realized one particular page was performing poorly, so he set up an A/B split test, with four new versions of the same page. "I ran the test and in a short amount of time I had enough data to realize the one page I liked didn't do well at all," said Miotke. "But one version out-performed the old page by eight to two, meaning the old page got two orders, but the audience that saw this other version resulted in eight sales. You can see of all the people who went to a page, how many bought something, and that's what you want to see, the bottom line."

In another case, he wanted to see what image worked best on a page. "One particular picture and layout got 15 orders compared to 9 on the other page, that counts for a 68 percent conversion lift," said Miotke. "So based on that, we are designing our other pages to fit with that, with what our visitors selected as what worked."

Matthew Roche, CEO of Offermatica, a company that provides testing software used by Miotke and other online businesses, said the key to testing is to pick the appropriate format and to be sure you conduct it over a significant period of time.

"In terms of traffic, it's not just how many people purchased, it's how consistently they do so," said Roche. "The problem most small businesses have is measuring significance of results, meaning you need a big enough sample size so you can be sure the outcome would be the same if you ran it on any other day in the future."

With software such as Offermatica's, e-tailers can see whether or not the test has been run long enough to merit predictable results. Offermatica's Web-based application lives outside of your IT infrastructure, making it generally easy to get up and running in a couple of hours, according to several clients. You load your content into Offermatica, add a small amount of Javascript to the pages to be optimized on your Web site, and your test campaign is ready.

The Offermatica server stores the content options and rules for your test campaign and delivers that content to pre-defined regions on your site that are created by the insertion of a small amount of Javascript on your HTML page. When a shopper visits an Offermatica-enabled page, his or her browser communicates with the Offermatica server to find out whether a special offer is to be displayed. If the visitor is determined to be a member of a targeted test group, he or she will get the content designated for that group for the duration of the test period.

Drew Sanocki, of DesignPublic.com, said he began doing testing because in his business, designer furniture, it's competitive and "we needed to see how to differentiate ourselves through design and content at the site."

He reiterated the need to let tests run long enough to get the best results and recommends using a subscription, Web-based solution. "Multivariate testing is hard to implement yourself, there's a lot of analytics involved," said Sanocki. "So an all-in-one solution like Offermatica is good if you're small, like us, we've got six employees so we're just finding the time to be aggressive with it."

In terms of pricing, Offermatica reviews small businesses on a case-by-case basis, said Miotke and Sanocki, and offer affordable monthly plans, though they declined to give details.

There are different types of testing, the simplest being A/B testing, then there is multivariate testing, and finally the Taguchi method, a more complex form of multivariate testing. While there are limitations to the simpler A/B-testing format, Roche said it is usually adequate for most small businesses, describing the Taguchi method as probably being "overkill" for small Web shops. Still, it's good to know the pros and cons of all the approaches. Bleow Roche outlines the logistics and benefits of each.

A/B Split Testing: Simple test of one element of a page against another to see which is more effective.
AB split testing allows you to randomly divide your visitors into two groups and show each group a different version of a page to determine which version leads to higher conversion, average order value, application completion or other target. These visitors are then tracked and a report is generated that describes the impact of the A or B page version on this outcome.

A common use of the A/B Split Test is to evaluate the impact of a new page layout on the likelihood of generating a sale. Two versions of the page are created. Often companies test a new page design versus the existing design. Traffic is randomly split between the two pages using custom programming, by splitting the application servers or using ASP tools like Offermatica. The visitors are identified as belonging to the A or B test group and are watched through their visit, and occasionally across multiple visits to see if they are more likely to purchase based on which page they saw.

A/B Split Testing is a simple approach to letting your customers "vote" on changes through their behavior. It also suffers from significant limitations. First, many changes have either a negative or not measurable effect. Basically, not every element on a page influences a purchase, making it necessary to run several tests in a row to find the element that matters.

Unfortunately, it can often take 10,000 visits and 50-200 orders over a minimum of two weeks to achieve confidence in an outcome, so running enough tests one after the other can take a long time. If a significant event happens during that period, wuch as Valentine's Day or tax time, it can compromise the result.

The second issue is that an A/B Split Test of a new page treatment against the existing may confirm a positive impact, but it cannot tell you what elements of the new design actually made the contribution. Imagine a situation where a new copy treatment on a home page was very effective, but the new navigation was actually worse. The overall test of the page may show a slight improvement, but hides the fact that you could have done even better with just the copy and not the new navigation.

Multivariate Testing: Helps you define what elements matter and which combination is the strongest.
One way to improve your chances of finding a winner is to test three or four or more versions instead of just testing old versus new and pick the best performer from among the larger group. This approach improves your chances of finding a winning version, but also increases your content development burden. A better way is to test elements on the page in different combinations of "recipes." This approach is called multiple variable testing or multivariate testing.

This approach allows you to test the elements on a page that you believe impact sales. When planned and executed carefully, multiple variable testing virtually guarantees a positive change over your existing page and offers insights into how to market to your customers and prospects elsewhere on your site.

A multivariable test on a product-landing page might test the product image, the headline and the product description copy. The goal is to create the most compelling page possible so that visitors to this page, often paid for through search or banner advertising, convert to customers at the highest possible rate. Two or more alternatives of the picture, description and headline are created and a page is composed for every combination of these elements in each of their versions. If there are three elements with two alternatives, this requires eight combinations or "recipes."

By splitting the traffic randomly and showing each visitor only one version, we can determine the optimal recipe. The advantage of Multivariable Testing over AB split testing is that you can nearly always find a recipe that the outperforms existing page. The problem with this method is that if you have more than three elements or more than two alternatives, the number of combinations becomes so large that it takes too many visitors to run a conclusive test.

The Taguchi Method: Determines the "best" configuration of elements using the smallest possible number of visitors.
If you have four elements in a multivariate test including the product picture, headline, copy, navigation and a promotion, and you have four alternatives for each, you need to run 64 recipes. It still takes 40-200 conversions for each recipe to achieve a conclusive test and the volume of traffic required is too great for most applications. Because of this limitation, experimental design methods have been created to test a small, indicative subset of recipes and estimate the theoretical best recipe even if it was explicitly tested. This approach is called fractional factorial testing and can be done using a number of methods including the Taguchi Method.

The Taguchi Method was developed 50 years ago and has been used with great success to optimize automobile and other product manufacturing. More recently, The Taguchi Method was applied to direct mail and Web applications. The Taguchi Method takes a number of elements on a page with one or more alternatives for each element and dictates exact combinations that will allow you to estimate the positive or negative effect of each element/alternative.

There are three beneficial aspects to this approach. First, by creating a "best page" using the best performing alternatives for each element, significant improvement can be achieved. Second, the length of the test cycle and the number of visitors required is surprisingly small. And finally, since the "recipes" are created using modular element/alternatives, using a solution like Offermatica, Taguchi tests can be designed and executed in a surprisingly small amount of time.

Taguchi tests have been run on e-mail, PPC ads and landing pages with great success. Where an A/B split test might create a 5-10 percent improvement, a Taguchi test cycle will regularly return 25-45 percent improvement and has been known to improve results by 100 percent.

A test cycle includes two weeks of testing a large number of elements in just two alternatives to identify which elements increase the likelihood of converting a visitor to a customer, a second test where the high-impact elements are tested with a greater number of alternatives and a final test of the "best recipe" against the original page. The test cycle takes from a couple of days to a month depending on traffic and variance and can be designed and run without significant quantitative marketing or statistics experience.

Michelle Megna is managing editor of ECommerce-Guide.com.

Do you have a comment or question about this article or other e-commerce topics in general? Speak out in the SmallBusinessComputing.com E-Commerce Forum. Join the discussion today!