Split Testing… Are we there yet?

When my daughter was in a car seat as an infant and we drove our car through town in the Summer, other drivers would roll up their windows because of the piercing noise of her screams. Well-fed, changed, and gently coddled in the softest blanket, her protests mocked it all.

I took to keeping a set of squishy earplugs in the ashtray for tricky city traffic. In a three hour drive to visit grandparents, we’d endure 2:50 of this. One upside was our confidence we’d never hit a deer because of the high-pitched ultrasonic waves sent ahead of the vehicle.

Then, in the last 10 minutes, drenched in sweat, she fell into blissful sleep. Cheerful and welcoming, my parents would probably wonder why we had odd twitches and complained of ringing in our ears for hours. “Why is she so sweaty?” they’d say, “don’t you guys turn on the A/C when you’re driving?”

The bold segue I’ll make here is that of constraints. In split testing, we must constrain variables while we test others. We must be rigid in the control groupings and how we use results. To many clients (and designers) this is just like being strapped down for three hours.

While protests may not come in the form of blood-curdling squalls, they come in other ways. It’s so hard to deploy split testing without there being a complete and total understanding of the protocols and methodologies involved. And that is incredibly difficult for a consultant to achieve.

Thank goodness for PPC. With PPC, marketing consultants like myself can design landing page variations and produce some evidence that we actually can improve the performance of the site. That evidence then serves to fuel discussions of wider tests. Without PPC, split testing simply wouldn’t happen in my world.

The other day, someone said something to me that drove home the difficulty in using split testing in a smaller project situation. “You mean you’re going to design the same page more than once?” they said. While experience will help us get closer with an initial test group, we simply cannot hit it first pass. It’s sad because many of the companies who would be best served by split testing are the ones most resistant to it.

After this, there is another level of misunderstanding. That split testing is a design-time task. “Once you’ve got the split testing done…” started an email I got a couple of weeks ago. I am a believer that as long as the split tests are resulting in a business gain (profits, branding, long term growth) minus the costs of my doing the tests, you should continue doing it. Split testing is like printing money when done right. You don’t stop the presses.

I’m busy developing materials to address some of these issues, and at the same time working on my certification in such testing. For this reason, I’m also working on a 12 week split testing plan for clients - using Google Calendar for the first time! This will let clients know what’s happening in the testing cycle.

I’m going to be looking at some new software for split testing use in the next 4-6 weeks and putting into action on some sites I’m operating.

I’m reviewing the following

www.offermatica.com
www.memetrics.com
www.vertster.com
www.sitespect.com

…and some other scripts out there as well as some “by hand” methodologies I might use.

Those with opinions I welcome your comments!

del.icio.us Digg it ma.gnolia Netscape reddit StumbleUpon Yahoo MyWeb

One Response to “Split Testing… Are we there yet?”

1. Apr
   21
   2006
   At 10:53 am SiteSpect A/B Split and Multivariate Testing Says:

   Nice post! Kudos for intuiting that testing, or more appropriately content optimization, is an ongoing process.

   Just as in manufacturing, where process/QA engineers are constantly producing/measuring/fine-tuning, web site optimization solutions like SiteSpect enable you to run open-ended tests that dynamically optimize the controllable variables for the best response (i.e. conversion behavior).

   As you’ve stated, using PPC settings is a handy way to dip your toe into the split testing pool. However, when you want to test many page/site factors, it doesn’t alleviate the content management burden of producing a large number of pages. For example, a landing page with 3 factors (headline, image, copy) and 3 versions of each requires 3^3 or 27 unique pages. Who has time to produce/manage 27 versions of a landing page? Then multiply that by the number of separate PPC groups/campaigns. No thanks!

   The software and services that you’ve mentioned alleviate the content management problem and let you focus on the real challenge, which is setting aside the time for analytical thinking to iteratively test and learn which page factors are influential, and then refining them for optimal response over time.

   Please keep us posted as you progress through your certification!

Leave a Reply