Think like a respondent

Editor's note: Sarah Faulkner is principal and founder of Faulkner Strategic Consulting, a Cold Spring, Ky., research company.

We’ve probably all seen examples of awful questionnaires or discussion guides. If you want more, there are plenty online – check out @MRXshame on Twitter for some hilarious ones. The crux of the issue is that we, as marketing researchers – client or vendor side – have all been guilty of designing surveys that we would never want to complete ourselves. We conveniently forget or ignore how tedious those grid questions are, how annoying it is to answer the same question worded slightly differently multiple times, how impossible it is to remember something you bought six months ago and how your attention span starts to wane after 10 or 15 minutes.

There are significant consequences of bad questionnaire-writing, in the form of bad data and unreliable results, from straightlining or random responses from the people who do finish your survey to chronically underrepresenting certain groups from people who drop out. For example, Quirk’s published a compelling study in February 2016 (“The impact of survey duration on completion rates among Millennial respondents”) which found that there’s a major dropout inflection point among Millennial respondents after 15 minutes.

Education and training are critical to master good research design and quality but even if we have a good foundation, we can still lose touch with the people who respond to and participate in our research. To that end, I want to offer a few simple suggestions that we can all start applying today to help us create research we would actually want to participate in ourselves.

Be someone else’s respondent. Sign up for some online quantitative research panels or apps as a respondent. Try to cover a range of different types; for example: a traditional online panel like e-Surveys, a shopper-focused app like Field Agent, a customer experience-oriented app like SurveyMini and a gamified research app like The Pryz Manor from Upfront Analytics. Always be honest – if they’re screening out marketing researchers, you can’t participate. But for the ones you can complete, you’ll get great ideas for what works, what to avoid and how to make survey research more engaging. Nothing builds respondent empathy faster than taking a poorly-designed survey!

One recent trend I’ve observed from being a respondent is the addition of an “are you paying attention” question to quantitative screeners, presumably to screen out bots or respondents who randomly click on responses. One version has a multiple-choice question with one clear right answer among other clearly incorrect responses. Another clever approach has a very complex-looking question with lots of response options but if you read the entire question closely, it tells you to ignore the preceding instructions and just pick answer x.

I would not recommend signing up for qualitative panels though. They should all have industry screen-outs and even if they don’t, the chance of messing up someone’s research in a qualitative setting with small base sizes is just too high. However, if attending a researcher colleague’s focus groups as a backroom observer is an option, you can still learn a lot by observing different styles of qualitative research that you didn’t personally design.

Eat your own cooking. Take your own survey. No, you might not be the target consumer, but you are a human. If filling out that complex matrix question drives you nuts – and you wrote it! – imagine how someone who doesn’t care nearly as much about your category/business will feel.

It’s a good idea to follow this discipline as you’re writing a questionnaire and after it’s complete. If you need to, create a persona for yourself based on a target consumer when answering the questions. Assuming your questionnaire will be administered online, be sure to also go through an online test version of the survey as well. On a recent study, doing this caused me to reevaluate one of the screening criteria as I tried to evaluate the concept with fresh eyes and answer the associated questions as a respondent might.

Imagine sitting in that focus group or interview you’re designing as a participant. Is the entire discussion guide just questions being fired from the moderator? That doesn’t sound like a group I’d like to participate in! Where are the opportunities for the participants to engage in actual conversation instead of just responding to a list of questions? Even better, how can you build in creative exercises and activities that add an element of fun or even competition to keep participants’ energy and excitement high?

Be sure to look with alien eyes at those creative exercises you planned. Are the instructions clear? If you didn’t know what you know about your product/brand/category, would it make sense to you? How long will it really take to find all those images or complete that storytelling exercise? This is especially important if you’re designing a new activity or applying an existing exercise to a new category or consumer group for the first time.

Phone a friend. Request peer-reviews of your questionnaires and discussion guides. If you’re on the client side, exchange surveys with colleagues for feedback, especially those outside your business unit/category if possible. On the supply side, you can also get feedback from co-workers but just be mindful of confidentiality if you go outside the client team. To do that, you can use an in-market ad or package instead of the test one as stimuli and, if necessary, remove any proprietary client questions.

If you’re an independent consultant or don’t have ready access to colleagues for any other reason, strike a deal with a few trusted professional contacts to do a “feedback exchange” for questionnaires, guides, etc., where you review each other’s materials on a regular basis. The confidentiality caution applies here too – either put a confidentiality agreement in place or strip out client-identifying details and confidential stimuli.

The purpose of this type of feedback is to get another expert researcher’s opinion and to overcome any knowledge bias you may have. For example, if you’ve worked in the health care industry long enough, you’ve almost certainly picked up terminology and classifications from the professional or provider side that may not be transparent or make sense to consumers in survey or discussion guide questions. Having someone with research expertise outside your own industry can help make sure questions and response options are worded in layperson language.

Straight from the horse’s mouth. There’s probably no better way to understand the real survey-taking experience than pre-testing it with consumers (i.e., not professional researchers). There’s a range of ways to approach this – from very quick and informal all the way to an additional phase of research, depending on existing knowledge, business risk and budget.

The most informal way to do this is to find people who fit the most basic criteria (e.g., pet owners, restaurant-goers, detergent buyers, vacation planners, etc.) in your workplace or among friends and family and go through your screener, questionnaire or discussion guide with them. In this context, it’s most effective to administer it like a face-to-face interview where you read the questions out loud and mark their answers. You’ll get some instant feedback as you go (e.g., facial expressions, questions about the questions, etc.) and you can also ask for direct input too. Explain at the start that your objective is to make the questions as clear and easy to understand as possible and ask for their honest feedback.

On the other end of the spectrum, if you’re planning a large-scale research project (multiple legs or geographies, a very high investment or large potential business impact), doing a small qualitative phase up front to develop the questionnaire or guide can pay huge dividends. This also applies if you’re going to be researching a category/industry that’s relatively new to you and you don’t necessarily know all the right consumer language, response options, category attributes, etc.

I recently worked with a client on a research project for a product type that falls in their general industry but not one with which they have specific experience. We were planning a quantitative study to both gather some fundamental category understanding as well as get some direction on possible benefit spaces. I proposed a two-phase approach where we would start with a few local in-home consumer interviews to provide a base level of knowledge and then design the quantitative questionnaire from there. As a result, the client got some great insight from those interviews that we used to hone the benefit spaces, I designed a better quantitative study and I also provided some of the qualitative understanding to the final quantitative report to help explain the findings and shape the recommendations.

Make our research better

I hope this has given you some ideas about how to design research with the participant or respondent in mind, putting yourself in their shoes. Whether you’re learning to think like a respondent by actually being one or getting feedback from a professional researcher or layperson, we can all use these insights to make our research better – a little clearer, less complex, more engaging and, ultimately, higher-quality.