Let’s See What Happens. 

I loved science experiments as a kid.  My dad was an engineer, and my mom a teacher, so exploratory learning is in my DNA.  Often, I see young marketers starting out in their careers take a “test it and see what happens” approach.  Translated, this typically means a trial and error approach – or what I like to call “the spaghetti on the wall crawl.”  Just like in science class, experimentation requires a set of testing protocols in order to discover truth.   There’s nothing wrong with error, as long as you learn from it.  However, you’ll learn a lot faster if you have an Experiments Framework.

Common Marketing Tests. 

The most common tests found in marketing are A/B or multivariate tests.  These split tests typically help discover which variable best performs and where it outperforms.  Other common tests are User, Usability, Content, Incremental, and Intermittent.   All can be beneficial depending upon the desired outcome one seeks.  However, not approaching these common tests within a set of measurable parameters can lead to skewed results.

The Benefit of Testing Protocols.

One of the better guides I’ve come across was written a decade ago by Eric T. Anderson,  Hartmarx Professor of Marketing at Northwestern’s Kellogg School of Management, and Duncan Simester, NTU Professor of Management Science at MIT’s Sloan School of Management.  While we’ve seen quite a bit of progress since the penning of their guide, the principles they outline for business experiments is still on target.  In “A Step-by-Step Guide to Smart Business Experiments,” the authors lay out a scientific approach and provide their protocols for running experiments.  I will tell you that I have saved countless hours by following their approach.  A more in-depth review can be found on Harvard Business Review (HBR), but here’s a quick outline to get you thinking.

Seven Rules for Running Experiments

1. Focus on individuals and think short term. The most accurate experiments involve actions to individual customers, rather than segments or geographies… [note the observations of their responses].
2. Keep it simple. Look for experiments that are easy to execute using existing resources and staff… Much of what companies learn from mammoth experiments can be gleaned from smaller tests that involve fewer variables, saving resources for follow-up tests.
3. Start with a proof-of-concept test. In academic experiments, researchers change one variable at a time so that they know what caused the outcome. In a business setting, it’s important to first establish proof of concept, then change as many variables in whatever combination you believe is most likely to get the result you want.
4. When the results come in, slice the data. When customers are randomly assigned to cohort and control groups… you may effectively have multiple experiments to analyze…  Most actions affect some customers more than others. So when the data arrive, look for subgroups within your control and cohort groups. If you examine only aggregate data, you may incorrectly conclude that there no effects on any customers.
5. Try out-of-the-box thinking. A common mistake companies make is running experiments that only incrementally adjust current policies. However, it may be more profitable to experiment with completely different approaches.  If you never engage in “what-if” thinking, your experiments are unlikely to yield breakthrough improvements.
6. Measure everything that matters. Viewing results in context is critical whenever actions in one channel affect other channels or when short-term actions can lead to long-run outcomes.  Start with actions that have only short-run outcomes.
7. Look for natural experiments. The key to identifying and analyzing natural experiments is to find cohort and control groups that were created by some outside factor, not specifically gathered for an experiment. Geographic segmentation is one common approach for natural experiments, but it will not always be a distinguishing characteristic.

Remember:  The goal of any test is not to conduct perfect experiments but to discover those truths that help you make better business decisions.