9 Common Mistakes Can Ruin Survey Results
A survey is a popular tool to collect information, opinions, and feedback from respondents. For example, before launching a new product, companies can conduct a survey to do market research. Or, after a course, teachers can use it to collect students’ feedback. Based on the feedback he received, he can improve the instruction materials. That’s why a qualitative survey can benefit so much. On the other hand, a bad one can destroy the data. Thus, this blog will reveal to you 9 common mistakes that can ruin survey results.
- Leading questions
- Loaded questions
- Double-barreled questions
- Uncleared scales
- Unbalanced listings
- Complicated languages
- Too long survey
- Forcing people to answer every question
- Untargeted distribution
Image from Freepik
Let’s get started!
#1: Leading Questions
Let’s learn about the first mistake that can ruin survey results. It is adding leading words in questions. It can change how people respond. For example:
Bad survey question: Do you feel our company’s working environment is friendly?
This question includes the leading word “friendly”. You are introducing your bias that the company’s working environment is friendly. It appears likely that you want respondents to agree. However, some of them may feel it’s not friendly at all. When facing this question, it’s difficult to say anything other than yes. They can randomly answer the question. Of course, their untruthful answer will result in inaccurate data. You can ask this way instead:
Good survey question: How do you rate our company’s working environment on a scale of 1-5?
If you already learned from this mistake, now, let’s recheck your survey question list, remove all words containing your opinion and preference. Try to keep the language as neutral as possible. You can do a test on your friends, family, or colleagues. Ask them if they feel compelled to answer any question. If yes, fix it.
#2: Loaded Questions
Another unintended mistake you may make is asking loaded questions. Similar to leading questions, the loaded ones can cause confusion, misleadingness, and of course, influence respondents’ answers. Leading questions try to drive respondents to one side of the argument. Meanwhile, the loaded ones include an unverified assumption. To understand what is the loaded question, take a look at the example below:
- Do you appreciate the DINK (Double Income No Kids) trend?
This is a loaded question due to containing a presupposition that DINK is a trend. No matter respondents say either yes or no, they inadvertently agree that the presupposition is true.
To avoid this survey mistake, stay away from assuming anything in your questionnaire. Alternatively, break the original one into small parts. For example, ask them about the assumption first:
- Is DINK trending now?
If they say yes, move them to the second question. In case they say no, let them skip the second one.
- Do you appreciate this trend?
#3: Double-barreled Questions
Don’t ask two things in one question. Avoid this mistake if you don’t want to ruin survey results. For example, take a look at this question:
- How do you scale the food and service in our restaurant?
“Food” and “service” are two different fields. How about the case of delicious meals but poor service? How do they scale them at once? The advice is to ask two separated questions instead of one to have accurate feedback:
- How do you scale the food in our restaurant?
- How do you scale the service in our restaurant?
#4: Unclear Scales
Scales need to be clear and consistent throughout the whole survey. For example, if you use a scale of 1-5 to measure customers’ satisfaction, you should give a detailed explanation from the beginning. For example, 1 stands for “extremely unsatisfied”, 5 is “extremely satisfied”. Besides, remain them in the same order until the end of the survey. Otherwise, respondents may be unaware of changes and give inaccurate answers.
Another point you need to notice is to avoid mutually non-exclusive response categories:
In the above example, how should they select if they are at the age of 18, 35, or 60?
#5: Unbalanced Listings
Sounds simple, but you can ruin survey results if you make this mistake. Unbalanced listings can drive respondents to one side of topics. To make it balanced, add equally weighted negative, positive and neutral to the list. Remember to arrange them from negative to positive, or vice versa. Randomized order is not highly appreciated in this case.
#6: Complicated Languages
Using complicated languages is also considered a mistake that can ruin survey results. Try to avoid acronyms, technical terms, or jargon to make your survey easy to understand. If you have to use them in some cases, you can provide their definition or example.
Take a look at the example below:
Bad survey question: Have you ever experienced Linux?
Good survey question: Have you ever experienced Linux (an operating system like Windows or macOS)?
Regarding operating systems, people have commonly known Windows or macOS – two popular ones. Meanwhile, not everyone has heard about Linux. Thus, giving more details about Linux helps respondents understand the question clearly.
#7: Too Long Surveys
Don’t make the survey too long. Put yourself into respondents’ shoes. If you were them, would you be willing to spend so much time taking a survey? Too long surveys may bother them. They may try to finish it as quickly as possible by randomly selecting answer options. The carelessness in their answer will affect the data quality.
To avoid this survey mistake, make sure that your surveys only need some minutes to complete. Recommended number of questions is around 10-15. Try to make them short, precise, and qualitative. Besides, abusing open-ended questions (for example, essay questions) is also a reason to make your survey long. That’s because this question type needs more time and effort to answer. Don’t write more than 2 open-ended questions in one survey.
Regarding answer options, people may make mistakes by composing either too many or too few choices. The survey’s purpose is to gather information. Thus, the former can make respondents confused and cause difficulties in data analysis. Meanwhile, the latter seems to force respondents to make a choice that’s not representative. For example:
If you ask students who are homeschooling this bad example question, both options are not their correct answers. Therefore, in this case, you should add the “Other” option and give them a space for writing details.
#8: Forcing Respondents to Answer Every Question
Remember that respondents spend their own time fulfilling your survey. In most cases, they get nothing, and their lives are unchanged whether they take the survey or not. Therefore, try not to force them to answer all questions. You can make them uncomfortable if you ask some sensitive or personal questions. According to SurveyMonkey.com, when asking people across the U.S:
- 77% of people said they wouldn’t provide contact information like the phone number or email in a survey.
- 55% said they wouldn’t give their full name.
- 33% said they wouldn’t mention how much money they make.
- 27% said they wouldn’t provide identifying geographic information, like zip code.
- 7% said they wouldn’t provide information on race or ethnicity.
To avoid this survey mistake, you should explain why you want to ask these questions and promise not to leak their personal information, for example. Another way is adding options like “No idea”, “I don’t know”. Alternatively, you can leave some questions optional to answer. Making it possible to skip some questions is better than letting respondents quit the whole survey. SurveyMonkey also shows that 27% of respondents will give up answering if they can not skip just a question. Thus, you can receive less data as you want. Furthermore, 25% will go back and provide a random answer only to continue. For this reason, the data quality can become worse.
#9: Untargeted Distribution
Last but not least, distributing a survey to the incorrect target audiences is a typical mistake that can ruin survey results. As you know, the more people take the survey, the more accurate the data is. However, distributing it randomly to reach the desired number of responses can bring counterproductive effects. For example, your company is about to launch a new cosmetic product. The target customer is young women who are under 30. Sending out questionnaires to those who are age at the age of 40 or 50 can get wrong research results. Experts said that signs of skin aging started appearing from the age of 30. Thus, women under and over 30 have different skin problems. They have different needs in cosmetic products. Therefore, try to send it out to relevant people to have the data as accurate and representative as possible.
Final Thought
Above are 9 common mistakes that can ruin your survey results. Be conscious to avoid them.
Nowadays, with advances in technology, you can bring surveys online. It benefits you so much: cost-efficiency, time-saving, accessibility, etc. You don’t need to print out hundreds of papers, but distribute them with only a link. Furthermore, when the coronavirus pandemic is ongoing, people have to keep a safe distance from others. Thus, conducting online surveys is an optimal solution. If you are looking for a tool to get it done, ActivePresenter – a powerful eLearning authoring tool is here to help. With this app, you can:
- Create an online survey using multiple ready-to-use question types: yes-no, multiple-choice, rating scale, etc.
- Export projects to HTML5, SCORM, or xAPI packages, then share the link.
- Receive the data via email or Google Sheet.
Let’s download ActivePresenter and try. Contact us if you need any assistance.
See more: