5 Survey Errors That Will Hurt Your Data Quality

Data is undoubtedly valuable. It offers unprecedented insights into past trends and future predictions that few other non-numerical sources can match, making it an invaluable resource in policy making and social, political and economic understanding. 

However, data can only be useful if it is accurate and reliable. There are some major obstacles that can stand in the way of credible data collection and analysis. These obstacles can occur at various stages in the process, from survey development to execution to response analysis.

This article will briefly address the five most common survey errors that arise during data collection, plus tips on how to spot and avoid each one.

Survey Error #1: Bad Survey Questions

Did you know that the way you ask your survey questions can affect your end result?

Everything from the general survey structure all the way down to the way questions in a survey are worded can influence the people taking the survey, affecting the overall credibility of the data collected.

Using neutral wording to avoid pushing readers towards a certain opinion, avoiding loaded questions that keep people from answering in their own authentic way, and refraining from posing absolute questions that require definitive and likely unrealistic answers are all valuable strategies to employ when crafting good survey questions. It’s also helpful to avoid any area of ambiguity, be brief, cut out any unnecessary jargon and offer well-defined and realistic answer choices.

Survey questions are the backbone of survey-based data collection, so it’s important to have well-thought survey questions if you want the data collected to be reliable.

Read more: How to optimize your survey by choosing the right survey question types

Survey Error #2: Researcher Bias

Stemming from bad survey questions is “researcher bias,” which is something that can creep up in surveys and flaw an objective outcome.

Experienced survey designers will likely use common words and simple vocabulary to keep survey questions from suggesting something different, or more complex, than what is intended.

No matter what the subject, reliable data depends on a common and universal understanding of the questions and terms at hand.

It’s also important to be aware of researcher confirmation bias that might occur on the back end of a survey, when researchers are evaluating data and looking for information to support a point they believe exists or want to prove. While humans will be inherently biased to an extent, crafting surveys that are as free from bias as possible can help to ensure the most objective outcome possible. One of the best ways to ensure the validity of a survey before releasing it is to pilot the survey and incorporate any feedback. 

Read more: Is researcher bias distorting your survey results?

Survey Error #3: Sampling Problems

Developing a valid survey is the first crucial step in data collection. However, once that has been done, it’s equally important to make sure the survey is being given to the correct audience.

Surveys are useless unless they properly account for their respondents.

There are a variety of sampling techniques that can be used to make sure that those taking the survey will provide the best results. The more rigorous and intentional your sampling is, the more valuable and usable your results will be.

It is important though to be aware of sampling errors that can occur; things like selection bias that can occur during the sampling process can lead to a survey even more flawed than one not impacted by sampling errors.

Read more: 6 sampling techniques to help you choose a representative subset of the population

Survey Error #4: Bad Data Collection

The people tasked with administering surveys have an important role, but they too can become a liability to data quality if they are collecting data improperly.

Their errors may be a result of surveyor bias or they can be caused by uncaught errors from earlier on in the data collection process, such as bad survey questions or improper sampling.

Fortunately, there are a number of ways to minimize the errors that arise from data collection:

• Solution 1: Train surveyors on the questions, and explain the importance of the survey and of data quality. The more that surveyors are engaged in the overall vision, the more likely they’ll collect good data.

• Solution 2: Add basic checks to each survey. How long did the survey take? Where did the survey happen? Creating these checks will allow you to easily flag any survey that’s out of the norm and consider removing or recollecting that data point that may be contaminating the greater whole. 

• Solution 3: Analyze and flag data as it comes in. Analyzing data collected in real time allows for an immediate correction of flagged or flawed data points, because surveyors can be immediately sent back to recollect that data.  

Read more: How to build a great field data collection plan

Survey Error #5: Response Bias

Response bias refers to a set of factors that lead respondents to answer a question incorrectly. This generally happens when participants either lack the knowledge to answer correctly or they do not want to answer a question correctly.

Response bias may be caused by poorly worded questions, which confuse respondents and prevent them from understanding a question’s intent; pressure to answer a question with a socially acceptable answer; a tendency to agree with surveyors; participants’ desire to give the answer that they think surveyors want to hear; fatigue from a long survey; and more. 

Response bias can lead to incorrect conclusions, since surveyors may think that participants think a certain way when, in reality, participants answered the questions with inaccurate information.

While it’s impossible to completely remove response bias from a survey since there is no way to guarantee that everyone being surveyed has the same level of knowledge or desire to answer honestly, there are things that can be done to minimize response bias as much as possible. Being mindful of survey questions, structure and language, along with being intentional about the target audience and continuously tracking the metrics of a given survey are all ways that response bias can be reduced.

Read more: Are you following these 6 steps to avoid response bias in your survey?

Decisions made from data are only as valuable and accurate as the data that lead to them. Data is one of the most relevant and useful aspects of any policy-related undertaking, so ensuring the data is truthful and unbiased is crucial to sound data analysis. Being aware of things like researcher bias, sampling problems and response bias can help to craft more sound surveys and ensure that they produce more reliable data.