AB split testing allows you to randomly divide your visitors into two groups and show each group a different version of a page to determine which version leads to higher conversion, average order value, application completion or other target. These visitors are then tracked and a report is generated that describes the impact of the A or B page version on this outcome. A common use of the A/B Split Test is to evaluate the impact of a new page layout on the likelihood of generating a sale. Two versions of the page are created. Often companies test a new page design versus the existing design. Traffic is randomly split between the two pages using custom programming, by splitting the application servers or using ASP tools like Offermatica. The visitors are identified as belonging to the A or B test group and are watched through their visit, and occasionally across multiple visits to see if they are more likely to purchase based on which page they saw. A/B Split Testing is a simple approach to letting your customers "vote" on changes through their behavior. It also suffers from significant limitations. First, many changes have either a negative or not measurable effect. Basically, not every element on a page influences a purchase, making it necessary to run several tests in a row to find the element that matters. Unfortunately, it can often take 10,000 visits and 50-200 orders over a minimum of two weeks to achieve confidence in an outcome, so running enough tests one after the other can take a long time. If a significant event happens during that period, wuch as Valentine's Day or tax time, it can compromise the result. The second issue is that an A/B Split Test of a new page treatment against the existing may confirm a positive impact, but it cannot tell you what elements of the new design actually made the contribution. Imagine a situation where a new copy treatment on a home page was very effective, but the new navigation was actually worse. The overall test of the page may show a slight improvement, but hides the fact that you could have done even better with just the copy and not the new navigation. Multivariate Testing: Helps you define what elements matter and which combination is the strongest.
One way to improve your chances of finding a winner is to test three or four or more versions instead of just testing old versus new and pick the best performer from among the larger group. This approach improves your chances of finding a winning version, but also increases your content development burden. A better way is to test elements on the page in different combinations of "recipes." This approach is called multiple variable testing or multivariate testing. This approach allows you to test the elements on a page that you believe impact sales. When planned and executed carefully, multiple variable testing virtually guarantees a positive change over your existing page and offers insights into how to market to your customers and prospects elsewhere on your site. A multivariable test on a product-landing page might test the product image, the headline and the product description copy. The goal is to create the most compelling page possible so that visitors to this page, often paid for through search or banner advertising, convert to customers at the highest possible rate. Two or more alternatives of the picture, description and headline are created and a page is composed for every combination of these elements in each of their versions. If there are three elements with two alternatives, this requires eight combinations or "recipes." By splitting the traffic randomly and showing each visitor only one version, we can determine the optimal recipe. The advantage of Multivariable Testing over AB split testing is that you can nearly always find a recipe that the outperforms existing page. The problem with this method is that if you have more than three elements or more than two alternatives, the number of combinations becomes so large that it takes too many visitors to run a conclusive test. The Taguchi Method: Determines the "best" configuration of elements using the smallest possible number of visitors.
If you have four elements in a multivariate test including the product picture, headline, copy, navigation and a promotion, and you have four alternatives for each, you need to run 64 recipes. It still takes 40-200 conversions for each recipe to achieve a conclusive test and the volume of traffic required is too great for most applications. Because of this limitation, experimental design methods have been created to test a small, indicative subset of recipes and estimate the theoretical best recipe even if it was explicitly tested. This approach is called fractional factorial testing and can be done using a number of methods including the Taguchi Method. The Taguchi Method was developed 50 years ago and has been used with great success to optimize automobile and other product manufacturing. More recently, The Taguchi Method was applied to direct mail and Web applications. The Taguchi Method takes a number of elements on a page with one or more alternatives for each element and dictates exact combinations that will allow you to estimate the positive or negative effect of each element/alternative. There are three beneficial aspects to this approach. First, by creating a "best page" using the best performing alternatives for each element, significant improvement can be achieved. Second, the length of the test cycle and the number of visitors required is surprisingly small. And finally, since the "recipes" are created using modular element/alternatives, using a solution like Offermatica, Taguchi tests can be designed and executed in a surprisingly small amount of time. Taguchi tests have been run on e-mail, PPC ads and landing pages with great success. Where an A/B split test might create a 5-10 percent improvement, a Taguchi test cycle will regularly return 25-45 percent improvement and has been known to improve results by 100 percent. A test cycle includes two weeks of testing a large number of elements in just two alternatives to identify which elements increase the likelihood of converting a visitor to a customer, a second test where the high-impact elements are tested with a greater number of alternatives and a final test of the "best recipe" against the original page. The test cycle takes from a couple of days to a month depending on traffic and variance and can be designed and run without significant quantitative marketing or statistics experience. Michelle Megna is managing editor of ECommerce-Guide.com.
|Do you have a comment or question about this article or other e-commerce topics in general? Speak out in the SmallBusinessComputing.com E-Commerce Forum. Join the discussion today!|
Webinar: Should You "Switch-Off" Your ACD? Hear Why CareFirst BCBS is Switching to a SIP-based Contact Center Event Date and Times: Wednesday May 29, 2 PM Eastern / 11 AM Pacific Wednesday...
Business-Grade Video Conferencing for SMBs: Enhance Your Employees' Productivity Today Small and mid-size organizations (SMBs) are encountering a transition in their businesses, and they must react to the new way of working to stay...
DDoS Mitigation: Best Practices for a Rapidly Changing Threat Landscape Given the extraordinary and rapid changes in the DDoS terrain, traditional DDoS mitigation tactics are no longer sufficient to protect an...
Data Protection Made Simple Increasing demands placed on IT, along with tightening budgets has prompted IT leaders to seek out alternative technologies and improved methods of...