Desk of Brad


The Myth of A/B testing

“The journey is the reward.”

– Peter M. Senge 

Let us now praise famous data scientists 

Most websites are terrible at selling. As both a customer and as a consultant, I know it. You know it, too. The problem is two-fold. First, insulated by the law of large numbers, online marketers have gotten away with being terrible salespeople because of a failed thinking that if they churn through enough people, eventually some portion of them (usually a minute portion) will convert to customers. It’s a volume business. It’s also a totally expensive, inefficient, and ineffective way to market. 

The shotgun approach flowed out of the ancient art of direct mail and telemarketing in which scads of letters and carefully scripted phone calls would blanket the countryside in hopes of converting a few recipients into paying customers. In that world, a conversion rate of one percent—just one customer for every 100 letters or calls—would be thought a resounding success. Do you know any floor salesperson who would consider themselves successful if they converted just one of every 100 customers who came in their door? Exactly. Neither do I. 

Nonetheless, the direct marketing industry has thrived. To them, the concept of sophisticated data science was encapsulated in a technique known as A/B testing. A certain portion of the letters or call scripts were ever-so-slightly modified. They might have used different headlines, or different inducements on the envelopes to get you to open them, or the offer might have been tweaked. The marketers then waited to see which letters or calls performed better. In this world, a difference of a few hundredths of a percent was read as if oracles from heaven that one way of saying things was better at selling the product than the other. The phrasing that performed better in this simple, side-by-side comparison became the standard against which future words were tested. 
In the advertising world, this reality is reflected in the traditional frequency and reach data models that define the success of marketing campaigns by how many and how often people experience an advertisement, not how many of them convert to customers. These models were born in an era where the majority of people had fewer than ten channels on their televisions and a handful of brands from which to choose. Firms focused on brand marketing simply because channels and technology did not provide bandwidth for anything else. 

Accordingly, companies spend increasing sums on traditional marketing initiatives, only to experience diminishing returns on investment—they spent more and got less. Traditional campaigns struggled to connect with consumers, and, when they succeeded, very little real information was communicated.  Though the least-informed customer can pinpoint the problem, the self-appointed experts cannot articulate a cause. Let me tell you right now, the era of A/B testing was built on a myth. It is an incredibly unwieldy way to market. As I said above: it’s expensive, inefficient, and ineffective. I believe it’s ineffective because it treats audiences as monolithic. There’s no subtlety in the messaging, no ability to adapt to who the prospect is or what they are saying with unspoken language. On a sales floor or at a high-intensity sales pitch, the salesperson always reads the audience and adapts on the fly to the vibe of the customer. It is how selling gets done. I’ve done it a million times. (I’m doing it right now.) 

At Tahzoo, we believe that the era of A/B testing is over and that true data science and data-driven marketing is not only possible but critical to business success in today’s global marketplace. The maturation of both big data and content management technologies have reached such a level of sophistication that a new era of online selling is now entirely within reach. You’re witnessing our philosophy—the “Why” of Tahzoo—in action as we press our clients to get better at data science, to improve their technology infrastructures to prepare for personalized messaging in which the data we know about the prospect influences the content they see. You’re also seeing through acquisitions of cool technology such as Adnovate. 

So, forget A/B testing. The past is dead. The future is within reach. Our mission at Tahzoo is nothing short of changing the world of marketing through data and technology. Those disciplines have rung the death knell on that old era and marketers and marketing firms need to acknowledge they are no longer the sole gatekeepers of purchase-critical information. To effectively communicate—not merely to present—relevant and personalized content to drive consumer decision-making means that the floodgate to personalized content is opened, and firms must embrace it and invite consumers into a real and lasting conversation. When this happens—when marketing is truly aligned with customer needs—only then will customers be both engaged and educated. And that’s when great things can happen.