It’s one of the most common questions we hear from PickFu users: How do I test more than two options?

There’s two main approaches to doing just that, but before we delve in, let’s step back a second and address something fundamental: Why does PickFu only test two options in the first place?

Good question. It’s a concept called a pairwise comparison.

Pairwise comparisons

A pairwise comparison is pretty much what it sounds like: comparing things in pairs in order to determine preferences. This method has many applications, including studying voter preferences and attitudes, product testing, and psychology, where it’s sometimes referred to as a paired comparison.

The pairwise comparison is popular for its economy – only one sample population is needed, and the simplicity of choosing between two things causes smaller variances. The result is absolute; your audience prefers one entity, or it prefers the other.

Sometimes criticized because it’s “all or nothing,” pairwise comparisons don’t measure intensity of feeling. For instance, if you poll whether people prefer apples or oranges, you will have a clear result, but you may not fully understand how much your audience likes apples or oranges, or how they feel about any given trait of each fruit (acidity, texture, etc).

To ameliorate this shortcoming, sometimes you’ll see questions phrased with “on a scale of 1 to 5…” or ask whether you strongly agree with something, somewhat agree, strongly disagree, etc. These kinds of questions require respondents to judge their answers on a continuum, providing a range of choices. Though these scaled response questions can provide value in your research, the added complexity can also cause confusion among respondents.

One reason is that ranges are difficult to define. If asked to rank your five favorite fruits, for example, your top choice is probably pretty clear. But once you get down to your third or fourth favorite, the lines get blurrier. And if your survey respondents are confused, then your poll results will likely reflect this lack of clarity.

That’s why, in the quick-and-dirty style of PickFu instant polling, we prefer the simplicity of a pairwise comparison. And because respondents supplement their vote with a text explanation, they can qualify their answers if they feel they need to explain something further.

Want more good news? Even in pairs, it is possible to test more than two options. Here’s how:

Method 1: Elimination tournament

Have you ever filled out a March Madness bracket? The elimination tournament is very much like that. What you’ll do is set up a head-to-head match between options, eliminate the losers and test the winners of those surveys against each other.

Below is a graph to illustrate how to test four options in an elimination tournament, also called a knockout tournament. On the far left and far right, you’ll insert your options. Then, working inwards, you’ll test them against each other (1 vs. 2 and 3 vs. 4).

After the first round of polls, two winners will remain. We’re calling them Winners 1 and 2. Run one final poll head-to-head, and you’ve arrived at your ultimate winner.

To make things handy, you can download a blank eight-option tournament bracket here. We’ve also created a blank bracket to test four options. If your number of options isn’t four or eight, you can use this site to create your tournament bracket (select “Blind Draw,” not “Seeded.”) In that scenario, an option or more will have a bye, in which it will not compete in a given round.

The elimination tournament means once an option has been beaten out by another option, it’s out of contention to win. That method may seem overly cutthroat, especially if any given result is statistically insignificant. Statistical significance measures how likely results are due to meaningful audience preferences instead of random voting variations. At its most basic, statistical significance answers the question, what’s the likelihood that if I ran this poll again, I would see similar results?

PickFu will tell you whether your results are statistically significant or statistically insignificant. You may want to consider adding more respondents to achieve statistical significance, or running the poll anew to compare results.

Another consideration in a knockout-style tournament is it’s possible for a more polarizing choice to achieve the highest number of wins. Put another way, all wins are not created equal. Let’s revisit our favorite fruit example.

In this theoretical example, bananas come out as the overall winner, but if you add up the raw score, apples received the most positive votes (115 to 106). Therefore, you might intuit that even though bananas won the tournament, people have the most favorable attitude towards apples.

Seeding (no pun intended) is also important. Seeding is determining where in a tournament each option sits. It’s almost like a lane assignment in swimming – the stronger swimmers are seeded in the middle of the pool. What if apples had faced off against bananas in the first round, rather than in the semifinal? One of them would have been eliminated. Therefore, an element of luck (or in a testing scenario, voting variances) may come into play. Going back to sports, March Madness basketball seeds are determined by each team’s regular season performance. But the excitement of the elimination tournament is that a “Cinderella team,” or unexpected winner, can prevail over a powerhouse. So when you’re seeding your testing choices, your own biases in how seeds are determined may affect the ultimate outcome.

Does this mean an elimination tournament won’t work? Not necessarily. Like any testing scenario, there are strengths and weaknesses. Eliminations are popular because a smaller number of polls are required. The outcome is a definite winner (see: bananas) but not necessarily the fairest setup (see: apples).
*Bias full disclosure: I totally hate bananas.

Method 2: Round-robin

A round-robin tournament is an “all-play-all”-style competition. Here, each option is tested against every other option in the set. Let’s say you’re a softball team in a league. In a round-robin scenario, your team would play every other team in the league. Every possible matchup would be covered.

There are some key differences between a PickFu round-robin and a softball one. If you’re on a softball team, you can only play one game at a time. So a softball champion is determined over a whole season of play. Therefore, an elimination tournament usually takes less time than a round-robin would to crown a winner. In PickFu, however, since you’re not limited by the availability of players, you can run all your matchups at once. This means a PickFu round-robin takes less time than an elimination.

Another key difference is in setup. In a softball team scenario, you could play with either an even or odd number of teams. If you played with an odd number, one team would have a bye and would not play during any given round. But remember – in PickFu, all your tests can happen at once, so byes are unnecessary.

Setting up a round-robin is easy with a little organization. This site provides blank schedules for four to eighteen different options (or “teams”). Simply assign each of your options a number and set up your PickFu polls accordingly, ignoring the rounds and any byes.

Our four-option fruit example from above would look like this in a round-robin:

Once all the matchups have been completed, you have two options to determine the winner. In the first option, tally the wins and losses of each of your options. The option with the most wins is the ultimate winner. If two options have the same number of wins, the winner of their head-to-head matchup can be deemed the final winner, or you can set up a final tiebreaker round where they are tested against one another again. In a final tiebreaker, consider increasing the number of respondents and remember statistical significance. The second scoring option is to add up the raw score (number of votes) of each option, and see which one scored the highest.

Round-robins are generally considered to be fairer than eliminations. There is no seeding, so each option has an equal chance against all other opposites. The main drawback of a sports round-robin is usually the time it takes to complete, but this does not apply in PickFu if you run all your round-robin polls at once. In the fruit example, all six rounds could be completed simultaneously.

There’s also more data to work with. You’ll see how each choice performs against other choices, and can read the comments of each head-to-head matchup. Ranking is also easier in a round-robin, so if you’re interested in seeing second-, third-, and fourth-place finishers, a round-robin will accomplish this more easily than an elimination tournament.

Which method should I choose?

If you’re choosing between an elimination tournament or a round-robin, consider these pros and cons:

Elimination tournament

Pros:
Requires fewer polls than round-robin
Definitively eliminates options by half with each round – good for a large number of options

Cons:
More susceptible to “luck,” or irregularities in any single matchup, so it’s less fair than round-robin
Has the potential to eliminate an option that might otherwise be a strong contender

Round-robin

Pros:
Gives each option a more equal chance because it’s all-play-all
Can be completed all at once rather than in subsequent rounds
Gives rank order of options (first-, second-, third-place finishers, etc), and more comments than elimination

Cons:
Requires a higher number of polls than elimination

In the end, it really depends on what you’re testing. If the question you’re looking to answer is a gut check, a knockout tournament might suit your needs. I previously wrote about an author who used an elimination tournament to test three different book covers. If it’s a weightier decision, either because you can’t gauge the relative strength of the options against each other, or because you want to make sure each option gets a fair shake, a round-robin is probably your best bet. We at PickFu used a round-robin when we were testing our logo.

One final tip: keep the wording of your question consistent across all polls, regardless of the method you choose, so that all respondents are answering the same question.

Update, June 2016: Testing more than two options is now available auto-magically! Simply add as many options as you need and the round-robin happens without any planning. Still, it’s nice to see what’s going on behind the curtain, isn’t it?

Ready to give it a try? Set up your PickFu tournament now!