Editor's note: Susanna Warnock is assistant professor of business, Piedmont College, Walker School of Business, Demorest, Ga. J. Sumner Gantz is an MBA student, Piedmont College, Walker School of Business.
Judging by the number of survivalist reality shows – Naked and Afraid, Survivor, Ultimate Survival Alaska, etc. – there is no shortage of 20-40-year-olds willing to be seen on national television struggling to survive – literally!
Now try to find a 20-40-year-old to participate in a survey and you might be better off asking them to strip down and starve themselves for three weeks on television. The survey research industry has gotten a bad reputation of endlessly torturing participants with bad questionnaires – a reputation that is leading to declining response rates and the threat of non-response bias.
Gamification has been discussed as a possible solution to the challenge of reluctant respondents. The idea is simple: make the task of completing surveys less onerous by incorporating more game-like elements to make providing information feel less like an invasive exam and more like a fun and entertaining way to pass time.
There is no standard template for gamified surveys or even a set of accepted gamification techniques from which to draw. Some companies tout proprietary software that is designed with avatars and storylines to mimic a videogaming experience while collecting data. Other researchers have focused on updating the functional features of traditional survey questions (slider scales, card sorts and so forth). Still others have experimented with changes in the wording of questions (incorporating projective techniques, for instance) or with providing overarching narratives to the survey experience (a press conference in which the questions are asked by hypothetical reporters, for instance). The anecdotal evidence from this wide range of gamification in surveys suggests that the techniques have promise in engaging respondents, increasing completion rates, response rates and even in generating a higher quality of data.
For most independent research companies the gamification techniques that require sophisticated software (to create video games, for instance) are inaccessible. But if simple changes in the wording of questions and aesthetics can increase response rates, the method is worth pursuing.
We partnered with New South Research (NSR), an independent marketing research company based in Birmingham, Ala., in order to test a few of these gamification techniques. (We are grateful to Rick Wilson at Paradigm Sample for partnering with us for an early test of gamification techniques. The results of this early test allowed us to refine the techniques and led to this study.) NSR has maintained a panel of research participants for over 25 years and this panel is actively used to recruit focus group participants and conduct a limited number of surveys. In order to test gamification, we launched a survey concerning restaurants in Birmingham using NSR’s panel of research participants. Half the prospective respondents were sent an e-mail linking them to a standard-format survey and half were sent an e-mail linking them to a slightly gamified survey.
Based on the previous success of gamification, we expected those who opened the slightly gamified survey would be more likely to complete the survey than those who opened the standard survey. Surprisingly, we found quite the opposite. Among those who opened the survey, 49.3 percent completed the standard survey while only 39.0 percent completed the gamified survey.
The pattern holds true, regardless of gender or age. Both males and females were statistically significantly more likely to stay with the standard survey than the gamified survey. Likewise, in every age group, more respondents completed the standard survey than the gamified survey (Table 1).
If the idea of gamification is to reduce the burden of survey-taking by creating a more entertaining experience, the simple techniques we tested failed. The respondents completing the gamified survey were statistically no more or less likely to say they enjoyed the experience or that they would recommend the survey to a friend or family member (Table 2).
The survey we tested included several different gamification techniques and, before we threw out the baby with the bath water, we wanted to know if any of the specific techniques had success. Among the techniques tested were:
Use of images: Images were used in three different ways including simply to improve the aesthetics, to replace words with symbols on rating scales and replace words with logos on a select question.
Competitive elements: Explicit time limits were given on two questions (although no timer was used to disqualify respondents) and an implied sense of urgency was given on a third question.
Projection techniques: Respondents were asked to place themselves in a scenario in order to answer one question.
Feedback: The respondents were given a description of their personality after identifying the importance of several attributes.
Rules: Respondents were asked for a specific number of answers in an open-ended question.
Personalization: Respondents were asked to provide personal advice to the survey writer.
Mixed results
When evaluating the completion rates of each question type, we found mixed results (Table 3). Consistent with our overall findings, the standard format had statistically significantly higher completion rates in several questions that included time limits, rules, projection techniques and personalization.
The standard version...
“Thinking about food culture, what are five adjectives that you would use to describe the food culture in Birmingham?”
...had a higher completion rate than the gamified version:
“Ready for another game? Name five adjectives – no more and no less – that describe the food culture in Birmingham. You have 15 seconds starting when you click ‘Continue’ below.”
The standard version...
“In terms of the restaurant choices, what is important for a city to offer?”
...had a higher completion rate than the gamified version:
“Foodie TV has announced a competition to select the best destination for food. What can your city do to win?”
The standard version...
“Overall would you recommend Birmingham as a destination for an excellent food culture?”
...had a higher completion rate than the gamified version:
“If I could go to any city to eat dinner tonight, would you recommend Birmingham?”
However, it should be noted that the results of other questions that tested some of these same techniques (time limits and personalization, in particular) showed no statistically significant difference in the completion rate between the standard and gamified formats.
The gamified format had statistically significantly higher completion rates in several questions that included an implied urgency and feedback.
The gamified version (with different fonts and colors)...
“On the next few screens, you’ll see a description of a city. Is it…Birmingham? Atlanta? Jackson? Chattanooga? Charlotte? Montgomery? Mobile? Nashville? SPEED COUNTS! What cities do the following descriptions sound like?”
...had a higher completion rate than the standard version:
“Below are several features. For each feature, what city do you feel best fits the description?”
The gamified version...
“Wanna find out what your restaurant preferences say about you? We’ll show you several features of a city’s restaurants. Tell us how important each feature is to you and we’ll tell you what kind of person you are.”
...had a higher completion rate than the standard version:
“Below is a list of features of a city. For each feature, please indicate how important it is to you for a city to have this feature.”
Not a panacea
The conclusion? There may be merit in using gamification techniques to curb the continuing decline in completion rates but it is not a panacea. Some techniques seem to have a negative impact on completion rates – explicit time limits and rules, for instance. However, other techniques show promise – feedback mechanisms, in particular. In whatever way the challenge of engaging reluctant respondents is addressed, the problem is one that deserves our attention and creative solutions.