One size does not fit all

In conjunction with this issue’s focus on mobile research, Quirk’s interviewed our go-to research industry expert Tim Macer, managing director of U.K. consulting firm meaning ltd., to get his views on the strengths and limitations of mobile self-completion research, the impact of the iPad, tips on designing mobile surveys and how our relationships with our smartphones may work to a researcher’s advantage.

What types of issues do potential mobile research practitioners and buyers need to mindful of?

Mobile research represents an additional touchpoint with respondents, which I think we need to look at as supplementing the other methods. It’s inappropriate to think of it as being a replacement technology, which is effectively what online research has been. And perhaps online research has been pushed too far in that direction.

The opportunity we have at the moment is that devices are much more user-friendly and much more prevalent so you aren’t necessarily going to find that you are just researching among an elite group of early-adopters. Although, while we are a little bit beyond the early-adoption stage of mobile technology, sampling is still definitely an issue with mobile self-completion research. That’s not to say it’s an insurmountable issue - it depends on your sampling frame and who it is that you’re expecting to contact. It’s likely that it’s going to be people who have more disposable income and who are more highly-educated and so on adopting this kind of technology.

What are the similarities and differences, both in practice and in theory, between SMS-based mobile surveys and Web-based mobile surveys?

A bias exists among mobile respondents, and it’s very dependent on what kind of mobile interviewing you’re going to be doing [SMS vs. Web]. The lowest common denominator is to use SMS text messaging because it’s available on everybody’s mobile phone. Mobile phone penetration has reached the point where it is at saturation in most developed economies and is much better than Internet penetration in many developed economies as well, so it’s a fairly ubiquitous device if you’re able to use it for SMS. However, SMS surveys are very limited in what you can do. That’s not to invalidate it, but you are really restricted, some people would say, to five or six questions. Other people would say a survey with more than about three questions is probably too long for SMS. That’s a huge limitation. You have the advantage of coverage and reach but are very limited in the length of the survey and the way in which you can ask those questions.

Once you move on to using Web-type surveys [delivered to a smartphone that has a Web browser in it] that really opens things up. There are still limitations but the limitations are much more manageable for many surveys. The difficulty you have is the penetration and the reach. It has gone beyond the early-adopter stage but it’s still typically a minority of people who actually have access to mobile Web. That’s very similar to where we were with Web surveys about 10 years ago, in that it’s very much a minority that has access to these kinds of surveys.

What can mobile research accomplish that traditional or traditional online research cannot?

One special capability is the intimacy aspect of it and the extent to which it is a personal device. The moment-of-truth thing is quite interesting as well because it means that you can actually ask fairly sophisticated questions of people at the point where they actually experience something or have really closely experienced it. Mobile research presents the opportunity to get much closer to respondents at the moment of truth - the point they are actually engaging with services or interacting with products - than you can, say, with an online survey, where you’re still relying on the recall of the respondent at a point later in time.

A good example of this is the diary-type survey where people are expected to fill in advertising, TV viewing, radio listening or other media or product consumption that they’ve observed throughout the day. This is notoriously misleading, as people will omit things later on - they just forget things or forget the context. If you’re trying to ask people not just about the beverage that they consumed but the context of the arena at that time - these are things that are very difficult to recall accurately. If they’re able to complete the surveys as they’re going along, that’s very powerful and you’re likely to get much more realistic and accurate data.

Another advantage with the mobile solution is that you have real-time communication happening. The data is coming back as people are entering it. These are some of the same advantages that you have with Web surveys - the speed of turnaround.

What are some ways researchers can overcome the sampling limitations of mobile research?

Some researchers provide people with the devices instead of relying on people using their own technology that they happen to have. If the sampling, coverage and reach are sufficiently a problem, it can be that you supplement the people that do have the devices required by recruiting other people you lend those devices to. It is not expensive to equip the people with the devices that are needed to conduct the surveys, especially if you’re doing something like a diary survey. It may take a little bit longer to get the devices out to people, but you get the data back very quickly. That’s an area where you can overcome the difficulties of sampling because your sample frame isn’t of people who happen to have a particular device or an Internet-enabled contract on their phones. The device part of it actually becomes a fairly inexpensive part of the whole process. After all, the respondents are doing the data entry for you and that is a huge cost savings - that’s where some of the economies come.

When designing mobile surveys, what do researchers need to remember?

The big danger - and this is the big trap with mobile - is that it cannot work in the way that people have been forcing what we call “traditional online surveys” to work. That is, with very long interviews. It just won’t work. Part of the reason is because it’s likely that people completing surveys may be on the move, may only have a limited amount of time and are likely to be more discriminating about the amount of time they devote to it. Nobody is going to take a 30-minute interview on a mobile device. Even if you’re using a Web-style presentation, it still needs to be much shorter than what’s traditionally accepted. That is a lesson the industry has failed to learn. This is something that was known and understood about Web surveys, and many people have said that traditional online research was an opportunity to ask fairly short questions and to value the respondents’ time. Over the years we’ve seen that largely ignored.

There’s no question it’s the length of interviews and respondents’ remembered experience of being invited to take part in these surveys that are very long and often very boring that has done the most damage to response rates. There is a danger that that then repeats itself in this new medium.

I suspect that this goes back to the relationship between the research company and their clients in that there is tremendous pressure from research buyers to max out on all surveys they do and to try to put everything in there. Nobody seems to be practicing the less-is-more principle.

There are also issues of the small format of the screen itself, which immediately puts limitations on the kind of surveys possible. That has all sorts of implications for survey design. For instance, you can’t have very long lists, or if you do have long lists you have to be aware that the participant has to scroll up and down those lists and is not going to be able to see all of the items. That starts to have an effect on the kinds of responses you’ll get. If you just take a survey that you’re already doing by one channel, such as online, and you just make that into a small format to go on a mobile device you are likely to get different responses because the items may not be visible or not visible in the same way. You have to design the surveys to the format.

Finally, people shouldn’t underestimate the extent to which respondents are willing to give verbatim text or answer open-ended questions on mobile devices. Because people are used to using the medium for text messaging and e-mailing, these days it isn’t really a problem to have open-ended questions on mobile devices.

How does a respondent’s “relationship” with his/her mobile device factor into conducting a mobile study?

It’s technology that a lot of respondents are quite used to using to respond in an emotional way. They’re using the devices for communication, for SMS and of course phone calls, but the point is that people associate mobile phones and mobile smartphones - with communication about personal matters and using them in ways to communicate about emotional activities. [Discussing personal matters via mobile survey] is not such an alien step for respondents to take because it is a personal device. The respondent is in control of it, even more so than they might be with a desktop machine which is shared. They also might be completing a survey at work on equipment that doesn’t actually belong to them. We’re talking about a device that is actually in their pocket or in their handbag, and I think that gives it an advantage.

I think there seems to be some evidence to support an emotional/honest response. People who have been doing mobile research have found that respondents are very willing to be quite candid and willing to explore subjects which are quite intimate or personal.

How does a product such as an iPad fit into the mobile research scheme?

The iPad is a completely different channel and a device that just hasn’t existed up until now. Is it a rather large smartphone or is it actually a desktop or a laptop being shrunk down? It is effectively a laptop that you can use in a social/domestic context in the way that laptops don’t particularly sit all that naturally. Although it’s obvious an iPad is a mobile device in that it doesn’t tether you to a desk or a particular location, it doesn’t have all of the advantages that the smartphones and the small-format mobile devices have. It’s not something people are going to use on the move. You’re not going to be taking the iPad in the street with you. It’s difficult even to use them outside where there’s sunlight; you’re not likely to be able to see what’s on the screen, which you usually can with a smartphone. It loses some of those advantages but perhaps has advantages as being something people will use a lot and spend a lot of time in front of.

I suspect that the iPad probably has more in common with the existing desktop and laptop partly because of the size and format of it. What it opens up is the possibility for doing more sophisticated surveys than you can hope to do on a smartphone. You can’t show any complex media or stimulus material on smartphones or if you can it’s very limited. With the iPad and the other devices that will come along to imitate it, you can show stimulus materials because it is all about media. The iPad is much more conducive to surveys in the way that people are used to designing them.

How can we expect developments such as the iPad to impact mobile research?

What people are learning from mobile research is that most people are participating as if they were doing a Web survey - and that is at home. So in that context, the iPad could work extremely well as a self-completion interviewing platform.

The context for the iPad is slightly different because perhaps the iPad will come to be seen as a device on which you consume content rather than being one on which you create content. The input through the on-screen keyboard is not quite as natural as it is on a PC or a laptop with a conventional keyboard. But perhaps that’s a little bit overstated. The virtual keyboard certainly hasn’t held people back using smartphones and SMS and effectively using an interface with only nine or 10 keys to type vast amounts of material. I suspect that that isn’t going to be such a limiting factor.

Is mobile research as “mobile” as some would like to think?

It’s worth bearing in mind that - most of the time - most people who participate in surveys using smartphones are doing surveys at home anyway. It’s a myth to believe that when you’re interviewing people using mobile devices that they’re all out there in the line for the fast-food restaurant or in transit or even in the workplace. In the majority of cases, people will participate in a mobile survey when they’re at home. I don’t know that that’s a big disadvantage; it actually shows that they have control and are taking control of the survey invitation and not feeling that they have to participate as soon as they are invited. Instead, they participate at a time that’s convenient for them, and that has to be a good thing.

Do you feel like researchers are getting too excited about mobile research? Or is the excitement justified?

Some researchers are getting excited about the technology and seeing it as a way that they can differentiate what they’re doing from other people, but what I actually see is a lot of researchers being rather cautious about mobile interviewing, due to coverage issues and concerns about the extent to which you can do lengthy surveys. A lot of people seem to be dismissing mobile research as being inappropriate and not really having any potential at all and that misses the point. I don’t see a lot of researchers getting terribly excited about mobile research, and I think they could perhaps be a bit more excited because there are some situations where it can really improve engagement among respondents.

Are there other potential problems or problem issues that researchers should be aware of or try to avoid?

A lot of survey tools used at the moment don’t really contain support for mobile devices, so you may have to use a specialized tool to do the mobile interviewing. If you do use one of the few tools that support mobile as a channel you have to pay attention to what that survey will look like.

Researchers have to be much more in touch with the process of getting the survey ready and fielding it. Some researchers are very hands-on and take a great deal of interest in what the survey looks like when it is delivered to the respondent, but that’s probably the exception. Most researchers turn the design and actual deployment over to somebody else, and then they’ll test it and look at it. But they’re not that critical about how things are arranged on screen. Researchers have to become more critical because the screen real estate is at such a premium.

Web browsers are inconsistent on different phones and use different technologies for delivery. Within the phones themselves, once you deploy the survey to the smartphone, you can either deliver it through the Web interface that the smartphone has on it or you can deliver it through a captive application. Some survey software packages just support Web browsers whereas others are building their own applets that will run the survey in a captive sense on the device. Of course, that gives much more control over how it appears.

The captive application then depends on getting the respondents to agree to downloading that application to their smartphone. They’re not necessarily going to be reluctant to do that, and if you’re reaching them through a panel they’ll probably be reasonably willing to do that. Additionally, one of the interesting little things that seems to be emerging is the number of people who are already doing Web surveys on handheld devices. If you deploy a Web survey it’s likely that you’ll find maybe 1 percent or 2 percent of participants possibly already taking it on a smartphone or an iPhone. That number is likely to grow. People have to think quite hard about how to treat that.

How can mobile research complement/supplement existing methods?

As a complementary research method, it’s also worth remembering that a lot of people who have mobile devices will also have access to the Internet. So with some studies, you can play to the strength of both of those channels by doing a hybrid survey, where some of the initial parts of the survey you would do using a conventional desktop or laptop machine and then you would go to the mobile part.

What kind of respondent feedback can mobile researchers expect?

It could be a halo effect because taking mobile surveys is something new and different, but researchers are reporting that if respondents are recruited into mobile research they tend to respond very favorably when asked about it afterward. Some panel providers have been keeping information about whether respondents do have mobile phones; whether they’re Web-enabled mobile phones; and whether they’re willing to be interviewed on their mobile phones. The panel providers aren’t necessarily seeing a huge demand yet from clients to use the panels in that way, but when they do surveys like this the respondents really rather like them. That indicates that as a method of engagement it has a lot going for it. But again, that could just be a transitory experience with the novelty of it, which could wear off in time.