By the Numbers: The impact of gamifying survey questionnaires to increase spontaneous awareness

Editor's note: Pete Cape is global knowledge director in the London office of Shelton, Conn., research firm SSI.

Much has been written about the potential for gamification in online questionnaires to improve the quality of results. But the focus has usually been on gaming elements which use graphics, virtual reality, avatars and the like. Is it possible to harness the power of gaming psychology within a more traditional questionnaire design – by using different wording alone? Does a gamified survey question perform better than one using standard wording? Specifically, can gamification like this increase spontaneous brand awareness? This was the test we conducted with members of our firm’s U.K. online panel.

In face-to-face and telephone surveys the interviewer is instructed to probe at the end of a spontaneous awareness question by saying “Any others?” This instructs the respondent that their task is not yet over and they should search their memory for more answers.

Transferring the same question text to an online setting loses this prompt. While we generally get more answers in an online setting anyway (since the respondent has more time to think) how do we know that we have exhausted the respondent’s recall?

By turning the question into a game we put in place rules that stimulate the respondent to keep going until they can name no more. Does the gamified survey question perform better than the standard one? We fielded a matched sample of U.K. respondents from SSI’s online panel to test this.

Shown in the box in Figure 1 is the gamified survey design. Notice how we allow 10 boxes per respondent, not one. Not only does this make coding easier but it also allows respondents to assess their performance quickly and see how much further they need to go to compete the game.

Consider the question. We are very clear that this is a game. Since the rest of the survey was not 100 percent gamified it is necessary to do this to avoid having the question appear to be somewhat strange. In the question we set the rules and the objective (60 seconds to write down all) and the criteria for winning (getting 10). Before the respondent has time to think about how silly this might actually be, or to decide not to play our game, we are off: “Your time starts now!” We do not actually run a timer or close off the question after 60 seconds since this would add unnecessary pressure (and potentially reduce our data set). Remember, this is not actually a game.

Note also we give the respondent feedback after the question. This is motivational on two counts. First it is feedback as to competence – we are telling the respondent they did it well. Whether they did so or not, on an absolute basis, does not matter; they will never know and it is still motivational. Secondly it hints that someone is taking notice of what they are doing, as they do it. The fact that it is a computer does not really matter.

So does it work? In a standard version of the survey, respondents name, on average, 3.9 mobile phone brands. In the gamified version this goes up to 5.9. So we certainly get more brands being mentioned – a 50 percent increase in fact.

 But what extra brands? If it were simply the case that the respondent dredges up some obscure brand, then the gamification may have little point – the top brands would remain top and have the same awareness. But they do not come up with multiple obscure brands: A total of 59 different brands were recalled in the gamified survey, 51 in the standard version. It is recall of the top brands that increases (see chart).

One can almost imagine the respondent to the standard survey giving their four answers and feeling that enough work has been done. With no interviewer to probe “Any others?” they move on. They have satisfied the task, just not optimized their performance. Almost all the additional brand mentions are among the best-known brands. The ranking does not substantially change.

The differences in the levels of awareness are substantially different, however, and any researcher contemplating changing the question text of a tracker should be prepared to deal with the step-change in the data that will occur. That would be the cost. And the benefit? That would be that the data is better. It is more of the truth and therefore will be more explanatory.

It is worth noting as a final point that respondents in our study were willing to “play” the game. The level of “don’t know any brands” is exactly the same between the two versions of the survey.

The conclusion is that gamification, simply using text, clearly works to extract more brand information from the average respondent than a traditional open question. When considering gamifying a survey to improve the quality of results, graphics, video and virtual reality are not the only potential tools. Gamified wording alone can make a powerful difference.