11 Things Everyone Should Know About Piloting a Survey

Collecting primary data at any scale is challenging. Though data quality can be difficult to measure, it is crucial to ensure that you are not wasting time on poor quality data. Creating a good survey is one of the best ways to ensure data quality. A bad survey will only lead to bad data, and thus bad inferences.

How can you ensure that your survey will collect relevant, accurate, useful data? Pilot your survey. In a pilot, you can test out all aspects of your survey — question flow, order, language, etc. — before you use the survey to collect real data. Piloting helps you identify and fix issues that would have led to poor quality data. A pilot is like putting your survey through a simulator to understand what is right and wrong.

Piloting can be a time and energy-intensive process. However, it can also be fun, since piloting leads to unparalleled levels of learning!

The goal of piloting

Piloting should help you answer the following questions:

• Am I catering to my audience?
• Will my data collectors be able to seamlessly collect data using this survey?
• Does my survey format cater to my needs? Is it capturing too much or too little?
• Is my survey collecting the data in the correct format?

In addition, piloting can help you understand implementation hassles. By testing out your survey in advance, you can predict any problems that might arise during the actual roll-out. For example, a pilot can help you learn how surveyors should be traveling from place to place, whether you should inform respondents in advance, what your surveyors should carry with them, and more.

11 things to remember as you pilot

Piloting can be full of sleepless nights and stressful days! Here are 11 tenets of piloting to help the process go smoother. We developed these tenets from our experiences rolling out all kinds of survey — large and small, long and short, quantitative and qualitative.

1. Ensure that you have done enough secondary research

You don’t have to learn everything about your survey in the field. A pilot will be more successful if you first check how others have conducted similar surveys. A good best practice is to read at least 10 similar surveys. These can be found in research papers, company websites, and ebooks. Learning from existing data collection materials will give you a great head start.

2. Take feedback from your organization

Present your draft survey to lots of people for feedback. Some of the basic issues or changes can be identified by people within your organization. You can also speak to people who have done similar exercises before or reach out to a few experts in other offices.

3. Choose a representative sample for piloting

During your pilot, you should survey a sample of your final audience. Don’t choose this sample randomly. Choose each person in your pilot deliberately.

Make sure to consider the following factors for each of the participants in your pilot:

• Age
• Gender
• Education status
• Income status
• Caste (in the Indian context)
• Geography

Build a rigorous sample for your survey using these 6 sampling techniques.

4. Pilot the survey in the correct medium

Ensure that you pilot the survey in the medium it will roll out in. Do not do any of the following:

• Pilot a paper survey if you will actually collect data on a mobile app.
• Pilot an English survey if you will actually collect data with a Hindi survey.
• Pilot a small survey of 10 questions if you will collect data with a long survey of 100 questions.

5. Integrate the pilot with your training

Usually, organizations conduct surveyor training before data collection starts. Use the pilot as part of this training. The pilot will give the surveyors a chance to test out the survey in the field, ask questions about the survey, and figure out if they have any issues.

6. Question the survey questions

The most important part of the format is to understand what questions are you asking and what information you are getting.

One of the problems to look for during your pilot is compound questions — a question that asks two or more questions at the same time. For example, “How many children do you have and which one is the youngest?” is a compound question. It actually is asking two separate questions: “How many children do you have?” and “Name the child which is the youngest.” Be sure that each question on the survey only asks for one piece of information.

In addition, the pilot will show which phrases in your survey are widely understood and which ones are vague terms or jargon. Be sure to eliminate any vague words or jargon from your survey, no matter how obvious they might seem to you.

7. Examine the data being collected

Don’t just focus on the questions themselves; be sure to also look at the data that is being collected. Sometimes, issues in the collected data can show problems in your survey. Use your data to make a data-driven decision on what’s working and what’s not working.

8. Pilot all aspects of the survey

Pilot the question flow, the order of the questions, question types, and even the help text to ensure that the survey is clear to both your surveyors and respondents. To pilot question flow and order, you should observe how comfortable surveyors and respondents are as they go through the survey. Check whether the surveyors or respondents get confused or give the same information twice. If this happens, it might be helpful to change the order of the questions or add help text.

9. Collect feedback from everyone

Since piloting is an iterative process, it is important to ensure that you include viewpoints from all stakeholders. Get feedback from supervisors, surveyors, observers, respondents, and any other people in the pilot, and make sure everyone contributes during feedback sessions. In addition, you can observe different people to compile your own feedback.

Consider and balance all these viewpoints carefully. It is important to make sure that you don’t get biased by any single opinion.

10. Never re-pilot the same version of a survey

At the end of each day, incorporate all the feedback into your survey. Then you can use this updated survey for the next day of piloting. By immediately updating your survey, you won’t waste time on the same feedback.

11. Be thoughtful about what changes to make

You don’t need to include every piloting insight. Sometimes, you will get bad feedback if a surveyor misunderstands something or the sample respondents are not chosen correctly. It is important to remain critical and make sure you are only incorporating the right feedback — something that will drastically improve the survey for everyone (rather than just one or two surveyors) and will not make the survey more difficult. A good practice is to make sure that a significant number of people (more than 60%) agree to the survey changes.

A typical pilot day

Morning session: Make sure that your surveyors know the purpose of the survey and what you are testing before they collect data. For example, tell them to think about which questions to keep and which to remove, which questions should be asked first, how the questions can be worded better, whether help text should be added, and how the respondents reacted, how interesting the survey was, etc. If you tell your auditors what you are looking for, you will get better responses during the evening session.

Day monitoring: While the data collectors are collecting data, you shouldn’t sit back and relax. Shadow them to understand and observe the entire exercise. Don’t correct them when they go wrong; just record the mistake.Be sure to follow diverse people and watch several surveys end to end (if the survey is a reasonable length).

Evening session: Once everyone is back from the field, gather everyone’s viewpoints on the survey. It’s ideal to go over the entire survey, question by question.

Night session: Assemble the day’s feedback, analyze what feedback to include, and update the survey accordingly.

That’s how you pilot like a boss! Happy collecting 🙂