Lately, everywhere you go analytics industry folks are talking about AB Testing. That’s a good sign, since it means the industry is focusing on an overlooked leverage point in their web analytics investment.
But as so often happens, achieving full buzzword compliance has become the goal rather than the means; what lies behind the words is often lost. In this case, “AB testing” – the buzzword – has become a euphemism for plain old “testing”, which, like ordering liver on a first date, may be good for you, but is certainly not sexy. But throw some “AB” in front of “testing” and your dour liver is magically transformed into paté de foie gras.
This is a bit disturbing, especially when you hear people sprinkling the “AB” condiment to add flavor to anything from a focus group (“Hey, did you AB Test the response to the new company logo?”) to the mundane (“Suzie’s lamp is out, can you AB Test the light bulb?”) to the painfully comical (“Honey, let’s AB test the Lord of the Rings Director’s Cut with the Wide-Screen edition!”).
Mixed in there, perhaps lost among the cacophony of buzzword hype, are the ingredients to some real AB testing and with it a future vision of how to achieve its true objective.
What is AB Testing?
AB Testing is based on a simple principle that we’re all familiar with:
- compare and contrast alternatives;
- based upon measurement, act accordingly.
Let’s say we want to determine whether Nolan Ryan is a better baseball player than Homer Simpson? How should we proceed? First, we might set a metric for what we mean by a “better” baseball player. We can measure evidence in concrete ways, noting the two subjects’ different batting averages or RBIs or the like. What we’re searching for is the right metric—a formula that would lead us to a correct decision. Such a formula is more precisely termed a “fitness function.”….
______________________________________________________________________________
This is the begining of a 14 page paper John wrote about AB testing. We’re still not sure how we will be distributing it but if you are interested in reading it then email Jeff Eisenberg ( jeff at bryaneisenberg.com ) and he’ll send you the PDF.