Response bias refers to a set of factors that lead respondents to answer a question incorrectly. This generally happens when participants either lack the knowledge to answer correctly or they do not want to answer a question correctly. Response bias can lead survey takers to believe that participants think a certain way when, in reality, they answered the question with inaccurate information.
Response bias can be very dangerous as it can lead organizations to derive inconclusive or wrong insights from survey results.
Often, response bias is caused by an inability or lack of desire to answer questions correctly. This can be caused by a lack of knowledge or experience, unfamiliarity with a product or service, inability to recall information, lack of question context, respondent fatigue, or unwillingness to respond honestly. It can also happen when questions are not written correctly, such as when there are not enough options provided in the answers list.
An example of response bias would be when survey respondents are asked to compare your product with other similar products in the market. Respondents who lack knowledge about your product or similar products would be forced to answer the question randomly to continue with the rest of the survey.
With this in mind, let’s analyze these problems and discuss various ways to avoid response bias. Below are a list of points to keep in mind while designing your survey.
1. Be careful while framing your survey questionnaire
Keep your questions short and clear
Although framing straightforward questions may sound simple enough, most surveys fail in this area. Keep your questions short and clear; half the battle of avoiding response bias is won by framing the right survey questions. Respondents are less likely to answer if a question is too long or if they do not understand the question.
Avoid leading questions
One of the common errors people make while framing questions is to ask leading or hypothetical questions. At times it’s better to avoid these completely forthright questions.
Instead of asking a leading question such as “Are you satisfied with our product/ service?”, you can ask respondents about the quality of service provided and give them a variety of options:
- I enjoy using this product/service.
- This product/service meets my needs.
- This product/service is below my expectations.
Avoid or break down difficult concepts
Avoid introducing difficult concepts without assessing the audience’s knowledge of your topic. If you must ask about a difficult concept, break it down into multiple connected questions to offer better clarity to the respondents. Limiting the answers for these connected questions to a simple yes or no would make these questions even clearer.
For example, if you are conducting a survey on a complicated topic such as the healthcare compliance policy, you should carefully assess your audience’s knowledge of healthcare policies before sending out the survey. You should also be careful to explain any acronyms or jargon that appear in the survey questions.
Use interval questions
One simple way to make your surveys more effective is by changing Yes/No and multiple choice questions to interval questions. Interval questions usually enhance the quality of analysis on the survey results. Frame a question and provide answer options such as “Strongly disagree”, “Disagree”, “Neutral”, “Agree” and “Strongly agree”. Coding these options into an interval scale of 1-5 can help you get more accurate and effective answers.
Keep the time period short and relevant
You can also reduce the response bias by keeping the time frame or sample time period short. For example, you’re more likely to get an accurate response to a question about the the previous month than a question about the the previous year. This is because respondents are more likely to recall recent events.
2. Provide a simple, exhaustive set of answer options
One important aspect in reducing response bias is ensuring that the answer choices are short and concise. A well-framed question asks for a precise response and does not include any additional questions embedded within it. If words like “either”, “neither” or “nor” end up in your question, there’s a good chance your question is not precise.
For example, if a travel department were to ask, “Were you satisfied with your travel experience and the knowledge of the guide who helped you?”, a simple “No” answer could have more than one meaning. The travel department would be unable to identify if the respondent was unhappy with the guide’s knowledge of the place of travel. This imprecise question would lead to response bias.
In addition, response bias can result from an incomplete set of answers. For example, if a travel department were to ask “Why did were you unsatisfied with your travel experience”, an answer set consisting of only “High price”, “Too short”, and “Bad guide” would be incomplete. Including options such as “None of the above,” “Other,” or “I don’t know” in your set of answers ensures that survey questions are collectively exhaustive.
Furthermore, if answer options are long and unfamiliar, respondents may face difficulties in evaluating all the options. One technique to frame questions in a precise manner is utilizing a matrix-style question type. A matrix question with multiple answer choices would be extremely useful. After the respondents answer the questions, you will have numerous responses to further develop your service or product.
3. Use precise, simple language
Using simple, precise language is an important step in creating questions that yield precise answers. It is best to avoid jargon, company acronyms, and obscure terms. These terms can confuse respondents and make them respond in an incorrect or misleading manner. Keep the following points in mind to avoid the usage of imprecise and complex language:
- Try to avoid uncommon words and complex sentences. Only include abbreviations if you are absolutely certain that every survey respondent will understand them, or spell out abbreviations when they first appear in the survey.
- Avoid complex and confusing questions that can lead to confusion among participants. Complex questions lead to response bias in the form of unclear answers and unfinished surveys.
- Survey respondents may hail from diverse backgrounds. Make sure your survey questions are not open to misinterpretation, especially if any of your respondents are not native speakers. Words have different meanings for different respondents, and topics relating to location, time and distance can be subjective.
4. Structure your survey appropriately
One of the most overlooked forms of response bias comes from poorly designed survey structures. A survey structure generally refers to the order in which the survey questions are asked. Specifically, survey structure can refer to the number of questions per page, the survey logic, or the survey length.
Every part of survey structure can contribute to response bias and drop-outs. As with the other forms of response bias, you can structure your survey by thinking about how respondents will react to each aspect of your survey.
For example, putting the most prejudiced or personal questions at the end of your survey instead of the beginning can decrease your number of drop-outs. If a seemingly prejudiced question or a personal question appears early in the survey questionnaire list, it can offend the respondents and make them abandon the survey.
Also, respondents are biased toward the first answers they encounter in a survey question — i.e. they might choose the first option when another option would be more appropriate. Thus, randomizing the answer choices so that the answers appear in a different order to different participants can reduce response bias.
Another useful technique to add logic and structure your survey is by using a branching functionality to make sure that survey questions are relevant. For example, if you can ask the respondents what products they use. Then, based on their relevant product or service, you can branch them to a page that displays only the relevant questions pertaining to those products or services. Understanding and structuring the survey information in this manner can help one construct well-structured surveys.
5. Personalize the survey by keeping your target audience in mind
Your target audience will be more likely to respond if the survey is personalized and relevant. Personalizing surveys based on products, categories, or dispositions can decrease the response bias by enhancing the customer’s response rate.
Use segmentation techniques to target your audience based on demographics. This can make surveys more meaningful and help you derive more useful customer insights. Since respondents may not necessarily be expert users of your product or service, it’s always useful to map respondents’ capabilities to personalized survey questions. Otherwise the respondents may fail to understand certain technical jargon related to your product.
Another way to dig deeper into understanding the target audience would be to devise generalized questions about satisfaction. A good way to do this would be to assess if any of the respondents state that they were very dissatisfied with the product or service. If they respond positively then you can branch them to another page of questions to gain more knowledge about their dissatisfaction.
If the topic of the survey is irrelevant to the respondents, it’s often helpful to add an opt-out option (different from a neutral choice) that can filter between those who either cannot answer and those who do not wish to answer. For example, an opt-out option page can be done by framing a question that directly asks if the product or service is not relevant to the respondents. If the respondents answer positively, you can direct these respondents to the end-of-survey page.
6. Continuously track the metrics to be measured
In most situations, we tend to focus on metrics in the final stages of the survey lifecycle. But metrics are effective if we keep track of them right from the beginning stages of the survey design. For example, you can initially test the questionnaire internally within a sample user group. Once the responses are recorded, you can look at the response metrics to predict which questions or survey structure might cause response bias and take corrective actions before releasing the survey to a bigger sample of respondents.
Below are some of the metrics that can be considered to measure the response bias.
- Average score for a survey question over time: This metric refers to the variation in the number of responses for a survey question.
- Topic monitoring: This metric enables you to quickly gather information about text responses without having to go through every response in the survey.
- Page drop-off rate:
One cannot expect the respondents to answer questions with 100% accuracy or without any bias. But following pre-defined survey guidelines and a robust survey structure can help you minimize response bias.
1 Comment
Very informative post…thanks!