Editor's note: Philip Derham is director of Australia-based Derham Marketing Research.
Online surveys should be easy to do. You do them when you want and where you want to do them. They are not dinnertime interruptions, nor are they foot-in-the-door interviewers, calling when you would rather be doing something else.
But a recent analysis of 14,111 recent Derham Marketing Research online survey participants across Australia found that while four in five participants who started their online surveys completed them, one in five dropped out partway through. Completion incentives were provided, the surveys averaged 15 minutes, were written to be as involving and interesting – given the topics – and all had skips, so only relevant questions were asked. All surveys were optimized for completion on personal computers, laptops, tablets and smartphones.
The online survey software records what devices people use to start their online surveys, so we first looked at the devices used, to see if those provided indications of why people dropped out of the online surveys. The rise in use of smartphones in Australia, as elsewhere, and of their utility, was a particular interest.
Most participants used desktops or laptops (79 percent) to start the surveys. Few used tablets (9 percent) or smartphones (12 percent) but the starting and completion rates differed markedly by device, as Figure 1 shows. Smartphone users were far more likely to start and stop than were desktop or laptop participants. Tablet users fell between the two.
Seeing the results led us to ask why smartphone users started but dropped more than desktop or laptop online survey starters, particularly as the surveys were designed expressly for smartphone use as well.
The time the Australian respondents took to do the surveys was a possible reason for the differences.
On average, across all devices, the surveys took 15 minutes each to complete (dropouts ignored). This all-devices average time was the same average time as those who used desktops to complete the survey. Those who used tablets took a little longer to complete their survey – 16 minutes or 7 percent longer. Those who used smartphones took 18 minutes on average to complete their surveys – 17 percent longer than average, and a noticeable difference from the all-devices average, indicating online surveys are harder to complete on a smartphone than on other devices.
Seeing this data, some suggested smartphone screens were too small; or were difficult to use in online surveys; or smartphones were used in time-pressure situations; or that surveys on smartphones were interrupted to take phone calls or to read incoming SMS; that PCs, laptops or tablets were easier to use; or could be used at work. But these hypotheses were not knowledge.
Test several hypotheses
We were able to test several online survey hypotheses in an Australian pilot survey ostensibly about chocolate bars, vacations and supermarkets. At the end of the survey, we asked about the survey itself and about preferences for devices that can be used to complete online surveys.
The sample was drawn from an online panel provider’s database and the sample and responses were from a predominantly female (72 percent) and middle-aged (30 to 59 years – 58 percent) audience. These results are thus indicative and further more extensive research across Australia is planned.
Having identified that smartphone users were more likely to drop out of online surveys than were users of other devices, the pilot survey sought to understand attitudes to device use, as these may influence online survey completions.
As Figure 2 shows, smartphones were seen to be the most difficult devices to use to complete online surveys and desktops and laptops were seen to be easiest.
And to emphasize the negative attitudes to using smartphones for online surveys, two-thirds of the participants said that if they were to be invited to do an online survey on a smartphone, they would be likely to start and not finish or (almost worse!) look but not start. In comparison, only one in 10 desktop, laptop or tablet users would look and not start or look, start and not finish the online survey (Figure 3). Desktop, laptop and tablet users were more likely to expect that they would start and complete an online survey in one go.
Some sample comments included:
“I only look at my e-mails and look at surveys between sessions, when waiting for clients to arrive. If I can’t do it in 5 minutes, or the client arrives, I stop.” -- Robin, counselor
“Do it on the way to work in the train and often lose contact when we go in the loop and don’t bother starting again.” -- Simon, public servant
“Do you know how small my Galaxy Ace screen is and how big fingers are?” -- Russell, builder
“Always do surveys at work – one way at getting back at the boss!” -- Anonymous, occupation not stated
“Like anything, is so much easier using the right tools – and my desktop and keyboard are the right tools for any Internet activity.” -- Ted, retired
Less engaged
Key takeaways from the original data analysis and from the Australian pilot study are that smartphone users are less engaged with online survey invitations, take longer to complete online surveys and, if they start, are more likely to drop out partway through. And what was identified by desk research with past survey findings is borne out by the pilot survey participants’ attitudes to completing surveys on smartphones.
Given the ubiquity of smartphones across Australia, the gradual diminution of desktop PC use and the increasing interchangeability of laptops and tablets, these results suggest there is a need to understand how to better engage with smartphone users in online surveys.