A/B testing and the presidential campaign

Why what works so well for an online form won't work when it

comes to choosing a president?

A/B testing is a great thing.  It lets you compare alternatives and make better decisions.  In this short post we want to demonstrate the benefits of a/b testing by comparing testing for online forms with using a/b testing for choosing the best candidate in a political campaign.   As you can imagine, it won’t be so simple for the latter.

A/B testing background

A/B testing is a simple way to compare two things in order to find out which one of them works better.  In most scenarios, once you figure out which one is your “winner” you stick with it, and get rid of the other.  The treatment for the winner as well as the loser is probably where the similarity between running an a/b test for an online page and a presidential candidate starts and unfortunately ends.

Making only one change

When running an a/b test for an online form, it's important to make one change at a time.  Do not change both your key message and your background image,  otherwise you will not know which change made the difference.  If you are using a tool that supports multi variant testing (a/b/c/d test)  you can test more versions in one run.  But still you need to make sure that you understand what variants you are testing. e.g. if you want to test two variants such as message and background, four versions will cover all the options you need to test.    In a presidential campaign it's very different.   Imagine running a test for “version” Trump vs. “version” Clinton…  However, even if you could run such a test, you would be testing two completely different versions. You won’t have the luxury of controlling internal variants such as candidate’s recklessness, smartness, loudness, maturity, record, experience, etc.   

Concurrency, running the test simultaneously

One of the important elements in an a/b test is that you should try to minimize and control all aspects of the test. I.e.    You don’t want to test version A for one month and version B in the next month since there may be some other factors (like seasonality) which would impact the results.  So ideally, when testing an online form you want to serve one version of your form to one user, and another version of the form to the next user and so on.   Now think about presidency:  Can you imagine trying out Trump for one day, Clinton the next day, back to Trump and so on?  Obviously that would not work well.

How to measure success and conversion rates

Measuring success for online forms is easy:  their goal is to convert, so all you need is to count the number of visitors who submit version A out of all visitors to that form.  Do the same for version B, compare the rates and you have your winner.   Its also very easy to present the results of such a test in a graph or a dashboard.  Many times, a short glimpse is all you’ll need in order to see if you have a clear winner.    But what about candidates for presidency?  How do you measure success?  E.g. a president who acts to enforce gun controls would be considered ‘success’ for some and ‘failure’ for others.  Same for a president who passes a health care reform or who sends US troop overseas.   So even if you could run an a/b test among presidents, it would be much more difficult to decide on a winner.

To sum up,  while a/b testing is a great practice for testing landing pages and online forms it does not land itself to use for selecting your candidate for presidency.   More seriously, if you have a decision making situation in which you have several options or “versions”, check yourself :

• Do you need only one winner and should get rid of other less effective options
• Can you set a clear goal to help you compare and decide on a winner

• Can you run a short-time test to compare the options while controlling all external influences

If you answered yes to the above questions, you should consider trying out a/b testing!