Creating unbiased poll questions will help you achieve unbiased results.

One of the biggest benefits of polling is accessing an audience of people who have no familiarity with your product, logo, book, business name, or whatever it is you’re testing. They approach the question without bias… but as the poll creator, do you?

This guide will help you recognize whether biases may be influencing your results. Avoid these common mistakes and poll respondents will answer more openly and honestly.

Contents

• Contents
• Polling Question Mistake #1: Leading words
• Polling Question Mistake #2: Making assumptions
• Polling Question Mistake #3: The double-barreled question
• How to create good unbiased survey questions with PickFu
• Summary: Unbiased questions

Polling Question Mistake #1: Leading words

Your question may include a positive or negative bias — words that consciously or unconsciously lead the respondents toward a certain kind of answer.

Examples of biased survey questions: leading words

• How much did you enjoy this YouTube video? (positive bias – implies that the viewer enjoyed it, and leads respondents to answer more favorably)
• Should responsible parents vaccinate their children? (puts respondents on the defensive by insinuating that parents who do not vaccinate are irresponsible)
• A recent poll found 80% of Americans disagreed with this government policy. How unhappy are you about this policy? (negative bias – not only is the question phrased negatively, but it also includes a statistic that shows many unfavorable views, leading the respondent to feel as though he or she should feel that way, too.)

How to fix leading words

Remove leading words and phrases and structure the questions as objectively as you can:

• On a scale of 1-5, with 1 being the worst and 5 being the best, please rate this YouTube video.
• Do you think children’s vaccinations should be required?
• Please rate your level of agreement with this government policy.

Remember, biased questions lead to biased results. And if you’re using polling to make important business decisions, you want your feedback to be as objective as it can be.

Polling Question Mistake #2: Making assumptions

Sometimes questions include an opinion. For example, “Do you agree that dog owners should be able to walk their dogs off-leash in the park?” This assumes that dog owners want to walk their dogs without leashes, which may or may not be true. But by including this assumption, you are leading more people to respond in agreement.

Examples of biased survey questions: Making Assumptions

• Is your favorite color blue? (assumes the respondent’s favorite color is blue)
• As long as nobody minds, is it okay to smoke indoors? (assumes that nobody minds)
• Where do you like to party? (assumes respondents like to party)

How to fix assumptions in poll questions

Structure your questions around facts, not opinions. Make sure you’re not painting respondents into a corner so that they can only answer one way. In the first example, it would be better to ask, “Do you think dog owners should be allowed to walk their dogs off-leash in the park?” Similarly, these questions eliminate assumptions:

• What color do you like the best?
• How do you feel about smoking indoors?
• What do you like to do in the evenings?

Polling Question Mistake #3: The double-barreled question

Sometimes questions ask about two disparate things but only allow for a single answer. These double-barreled questions do not make good survey questions because respondents will often only concentrate on the one topic that means the most to them.

Examples of biased survey questions: double-barreled questions

• How do you feel about our two new flavors, French Vanilla and Hazelnut Cream?
• How have teachers and students responded to the new dress code?
• Which title and subtitle do you prefer?

How to fix double-barreled questions in polls

Double-barreled questions should be separated into two distinct questions and polled separately from one another. When questions are worded similarly in a multi-question poll, it’s also good to distinguish them with underlined words or italics, as below:

• How do you feel about our new flavor French Vanilla?
• How do you feel about our new flavor Hazelnut Cream?
• How have teachers responded to the new dress code?
• How have students responded to the new dress code?
• Which title do you prefer?
• Which subtitle do you prefer?

How to create good unbiased survey questions with PickFu

The same rules apply to preference testing on PickFu as in general surveying: avoid leading words, don’t make assumptions, and only ask about one thing at a time.

For example, if you’re testing mobile app icons, the simplest unbiased question would be, “Which app icon do you prefer?” To embellish the question with anything else may introduce bias. Sometimes pollsters want to know whether their icons read a certain way, so they’ll ask something like, “Which icon seems more kid-friendly?” While not biased, per se, this question asks respondents to judge your icons differently than they might otherwise.

Because PickFu respondents include comments, one approach to avoid any hint of bias is to ask the simple question, “Which app icon do you prefer?” and then read the comments to see if kid-friendliness is something the respondents took away by themselves, rather than being led to see kid-friendliness or not.

Another approach is to include factual statements (unbiased, of course!) to explain what you’re testing, and then ask the respondent’s preference. For example, “Our app is a game aimed at children under 10. Which icon do you prefer?”

Another tip is to customize your audience by behavioral segmentation, meaning based on their habits. In this app example, you could ask an audience of parents who also own an iOS device and spend money in the App Store.

These approaches may all yield various results, so some experimentation might be necessary to see what’s most applicable in your case.

Keep it consistent

Consistency is key in testing creative options. If you’re testing book covers, make sure that each design option includes the same information. If you tested two cover designs and each cover had a different title, for example, you wouldn’t know whether respondents preferred the design layout or the title on that design layout. Remember, only test one thing at a time.

Below is an example of an author who tested two covers. However, the two titles are different, only one option included a subtitle, and only one option included the “superfitdads” logo. Any of these variances may have skewed the results.

This poll would be less biased if each cover included the same title, subtitle, and author attribution. Then, the only test parameter would have been the cover design.

Here’s a good survey question example. This author is testing two cover designs. The title, subtitle, author attribution, and even graphic layout is the same. The only difference is the “Action Plan” stamp. By only testing one thing at one time, this author knows the stamp helps his cover.

Summary: Unbiased questions

In both general polling and preference testing, it is important to keep these main ideas in mind as you design your survey:

1. Avoid leading words that may sway the responses positively or negatively. Phrase your question objectively.

2. Ensure your questions are fact-based, not opinion-based. Do not make assumptions about your audience.

3. Test only one thing at a time. Do not use double-barreled questions. When creating comparisons between two creative options, include only one test parameter per question, such as layout, title, or color palette.

4. You may need to experiment with the level of specificity in your question. Generally, the simplest form of the question will be the least biased. However, there are instances where you will need to direct the question a certain way rather than leave it open-ended. Just be sure to do it according to the three tips above.