Creating an effective survey design
Editor’s note: Steve Wigmore is the senior director, modern surveys at Kantar. This is an edited version of an article that originally appeared under the title “7 dos and don’ts of effective survey design.”
The human attention span is at an all-time low. Distracted by text messages and push notifications, the average person only manages to concentrate on a single task for eight seconds – a 33% decrease since 2000.
This is bad news for market researchers. If you rely on surveys to gauge consumer opinion, your data could be spoiled by responder dropout, straightlining and other biases related to our hyperactive brains.
Luckily, you can collect reliable insights by being proactive about data quality during questionnaire development. It’s time well spent to polish your survey upfront for accurate insights, rather than reactively managing quality concerns post-fielding.
It’s good to understand the dos and don’ts of effective survey design.
1. Don’t ignore mobile users. Do design mobile-friendly surveys.
Modern consumers are busy. They are constantly racing against a ticking clock, only pausing during fragmented bits of leisure called “time confetti.” Most people spend these brief moments scrolling social media or answering e-mails.
As a market researcher, you must use screen-agnostic questionnaires to engage respondents during these brief moments. By optimizing your surveys for mobile devices you can reach 30% to 40% more survey takers and garner results that are more reflective of your target audience.
2. Don’t design long questionnaires. Do implement short surveys.
As previously noted, the average attention span is waning. We are constantly faced with a barrage of phone calls, news articles and enticing cat videos. Because of this, surveys should be kept short.
Generally, longer surveys correlate with higher instances of responder dropout. Research conducted by Kantar, for example, suggests that a survey over 25 minutes loses more than three times as many respondents as one under five minutes.
We like to aim for 10 to 12 minutes but as always, consider your research and the overall panelist experience.
3. Don’t be repetitive. Do prioritize efficiency.
If you are struggling to trim a lengthy survey, start by looking for questions that prompt the same or similar responses. For example, if your questionnaire asks respondents to rank their favorite ice cream flavors in one question, there is no need for them to later select the most delicious flavors from a list.
Redundant questions can be confusing and frustrating for survey takers, triggering survey dropout. Market researchers can collect more valid and reliable data by eliminating all redundancies. You can do this intuitively by reviewing each question from the perspective of a respondent.
4. Don’t overcomplicate questions. Do be clear concise.
Nearly 20% of workers will stop reading an e-mail after a paragraph. Keep this in mind when you are developing questionnaires. Most respondents are likely to abandon a survey when faced with wordy questions or will select answers at random to speed things along.
To improve your questionnaire design, remove unnecessary text.
Consider this verbose question: “Think of the last time you purchased an article of clothing (such as a blouse, jacket, sweater or pair of pants). Your purchase may have been conducted online or in person. Now, please rate on a scale of 1-10 (where 1 is strongly pleased and 10 is strongly displeased) how satisfied you were with your purchase.”
Instead, you could simply ask, “Thinking about the last time you bought clothing, please rank how satisfied you were with your purchase.”
By narrowing your language you can garner more accurate responses. You can also reduce respondent fatigue.
5. Don’t ask random questions. Do keep the purpose of the survey in mind.
Survey designers are often so fixated on question order and wording that they give little thought to question type. However, the types of questions you ask can significantly impact the survey experience and, therefore, data accuracy. For example, you may use slider or grid questions to present multiple options – say, brand names – to a user at once. Or, if you need to present responders with mutually exclusive answer choices, you may opt for a multiple-choice question. Meanwhile, if you are hoping to garner qualitative insight from survey takers, you may opt for an open-ended question. By thinking critically about the type of information you want to collect, you can select the most appropriate type of question. This, in return, improves data quality.
6. Don’t ignore participants’ needs. Do adopt an empathetic questionnaire design.
There is an undeniable science to designing effective and efficient surveys. However, you must also remember that the people answering your questions are, well, people. They are emotional and unpredictable beings who could potentially feel judged by the questions you ask.
Questions concerning salary, education status or habits (e.g., “How often do you eat fast food?”) can feel interrogative and overly personal. As a result, respondents are liable to bend the truth, which leads to poor data quality.
Instead, consider adopting an empathetic approach to survey design. An empathetic approach is one that encourages you – the market researcher – to think from the perspective of a respondent. In doing so, you can help the survey taker feel more comfortable telling the truth. In practice, empathetic survey design may involve reframing potentially loaded questions or using humor to diffuse a socially charged prompt. For example, Kantar found that 27% percent of people who were shown a meme in the question “Do you recycle?” admitted to never recycling. Comparatively, only 1% of respondents who weren’t shown the meme admitted to not recycling.
7. Don’t forget about quality assurance. Do conduct data quality checks on your surveys.
Today’s survey respondents feel more time-pressed than ever before. Because of this, they are likely to rush through questionnaires or offer nearly identical responses to each question. This can skew the statistical accuracy of your survey.
As such, quality checks are considered a survey design best practice. Fortunately, these quality checks don’t need to be complicated. You may, for instance, choose to incorporate a red herring into your questionnaire. A red herring is a question that offers a bogus option among a set of valid ones. As an example, you may ask, “What is your favorite dessert?” The answer choices may include ice cream, cake, pie, pudding and furniture. Obviously, the latter isn’t a toothsome treat but rather a trick for catching respondents who aren’t paying attention. Well-crafted surveys are just one essential element for collecting honest and reliable marketing research data.