Editor’s note: Alex Wheatley is research innovator at research firm Lightspeed GMI, London. This is an edited version of a post that originally appeared here under the title, “Survey confession: the impact of survey design on honesty.”
There are many questions that are difficult to answer:
- “Have you considered an affair?”
- “How many vegetables do you eat a day?”
- “How often do you go to the gym?”
- “Have you lied to your boss?”
When it comes to a question like, “How much do you drink?” it can be hard enough to be honest with ourselves let alone a researcher! Fortunately, the anonymity and context of online research puts researchers in a unique position to secure honest answers to sensitive questions. This is no easy feat. When we ask a question there are many hurdles we must overcome to reach an honest answer.
Some of these hurdles are unique to online research. If we ask someone, “Which of these [blank] do you own?” there’s an array of biasing factors that can impact our responses:
- Acquiescence bias creates over-claim simply because we like to say yes.
- You may have owned an item in the past or plan to buy one, that’s basically a yes … right?
- You could be curious to see what questions come up if you click yes on something you like.
- You’ve got something close.
- Oh no – the dreaded miss-click.
- I’ll lie, I’m sure my answers will still be interesting for them.
Moreover, in any context there are a plethora of barriers to the truth that aren’t unique to survey research and go beyond simple questions of ownership. Some questions are embarrassing to answer, some are tricky to put a number on and others can be hard to remember.
In a survey these issues can be further exacerbated by privacy concerns and the desire to get through the questions quickly. Whatever the issue, the result is the same – researchers cannot take an honest answer for granted.
Lightspeed GMI conducted a multi-nation project to look at the best means of securing honest answers. By experimenting with different methodologies and observing in the data the amount of confessions the team elicited, we were able to evaluate which tactics produced the most honest responses.
The findings divided the issue into three main categories: questions we’re scared to tell the truth about; questions we lie to ourselves about; and questions we want to lie about. Each category presents its own set of issues.
1. Questions we’re scared to tell the truth about.
When you ask someone a sensitive question that they might be scared or embarrassed to answer, the determining factor in whether they feel comfortable giving you a truthful answer is your relationship with them. In a survey you only have one opportunity to build a rapport that puts your respondent at ease, and that’s the introduction.
Our research found that the key to getting people to answer sensitive questions is to engage them emotively and firmly with a powerful introduction. The most successful approach for us was one which broke down our key messages of trust framed within the challenge: “Can you answer our questions as if no one was listening?” and followed by the pledge, “I promise to tell the truth!”
While we’re not suggesting every survey should have a pledge, this approach was much more effective than a traditional introduction because it didn’t make respondents question why we were asking a sensitive question.
2. Questions we lie to ourselves about.
The self-deception at play when you ask respondents to make an observation about their behavior is not something that a strong rapport can solve. In fact, our research found the barriers to truth here are not necessarily a matter of honesty or deception at all.
The biggest obstacle respondent face when trying to make an accurate self-observation is simply calculating the value. We get the most truthful data when we break down the task to make it simple for the respondent to evaluate, by giving a time frame rather than burdening the respondent with ambiguous task of averaging.
3. Questions we want to lie about.
When asking questions which might tempt respondents to actively lie, we go beyond the truth-invoking powers of a strong rapport. Instead, we need to ask the question without them realizing we have asked it.
We received the most truthful responses to our tablet ownership question when we disguised the question by asking it less directly. Asking instead if they wanted a tablet first; then if the respondent said yes, asking if this was to replace an old one; and finally, if they said no, asking if this was because they already had one. Our research showed that while we should ask our question in a less direct form, we must remember to still ask it directly. Avoid the temptation to use a question format that does not force an answer as component-based selection biases strongly influence data.