Strength in numbers
Editor’s note: Annie Pettit is vice president, research standards in the Toronto office of Research Now and the chief research officer of Toronto research firm Conversition Strategies.
Can I help you? It’s a common phrase among friends, clients and colleagues, but it’s less often used among research methodologies. The debate about whether focus group research, social media research or survey research is the better methodology rages on in the Internet space because the method we trained in and use on a regular basis is usually the method we love the most. But, we need to remember that our favorite method is not the only method capable of solving problems. Our favorite approach is simply one of many complementary options in our ever-expanding research toolbox. This article will demonstrate how taking advantage of each method’s unique strengths, whether qual or quant or somewhere in between, can improve the overall success of a research project.
Social media research helps surveys - example 1
Let’s consider a slightly embellished research objective. We would like to learn everything we can about the experience of purchasing and consuming coffee. We want to know who buys, what they buy, where they buy, when they buy and why they buy coffee.
For most people, surveys would be the starting point for this research project, given they are unparalleled for generating census-representative measurements with statistically-determined margins of error (if you are a magician capable of generating a probability sample). Plus, surveys are great for measuring quantities and frequencies for very specific researcher-determined questions.
But, the sheer number of questions required to address all of our research objectives would necessitate a survey of unreasonable length. A simple question like “Where do you purchase coffee?” would have to account for hundreds or thousands of coffee shops, fast-food outlets and casual restaurants in a horribly long grid question. To prevent respondent fatigue and ensure they come away from the survey with a positive experience, it is the researcher’s job to keep questionnaires short and include only the most relevant items. Doing so, however, is not always easy and can result in much discussion among the researchers and clients.
Without causing any responder fatigue, social media research can be used to help reduce the number of answer options by identifying the items that are most relevant to consumers. For this project, we gathered a random sample of 100,000 online consumer-generated coffee conversations from blogs, microblogs, forums and more. Then, we prepared qualitative word clouds (Figures 1 and 2) in both of the retailer categories we were interested in: coffee shops and fast-food restaurants.
Figure 1
Figure 2
We learned that consumers participating in social media are more likely to mention Starbucks, Second Cup, Dunkin’ Donuts and Coffee Time. And at a minimum, those retailers needed to be included in our answer options. In addition, we decided to also include McDonald’s and Burger King as they were the most popular options within our second important category. This data helped us to narrow thousands of possible choices to just 10 choices, a number that was less onerous for our responders.
Social media research helps surveys - example 2
With the survey launched and data from 1,000 U.S. and 1,000 Canadian participants collected, we generated census-representative measurements that could be cut by any combination of variables from age, gender, income, household size, region and more. According to our results, the percentage of coffee purchasers who had purchased a beverage in the past 30 days at McDonald’s was about 63 percent in the U.S. and about 52 percent in Canada, making it the most popular place to purchase a coffee on-the-go. Given that McDonald’s is not technically a coffee shop and might not have been considered in the original list of coffee shops, it’s a good thing the social media research convinced us to include it.
Unfortunately, though, the second-most popular place to purchase coffee fell into the dreaded “Other” box. Is that “Other” box so popular because many tiny shops combined together, or is it because we missed one of the major players? Here again, social media research came to the rescue. Instead of generating a word cloud within each of the food-service categories, we expanded the options to include any retailer - any place where a purchase could be made.
The word cloud in Figure 3 reveals two interesting options that were not included among the survey options: gas stations and bookstores. Though they aren’t food retailers, our experience in everyday life tells us that these are legitimate responses. In fact, the text message-based research we conducted (more on that in a moment), which was intended to measure real-life experiences, also generated many gas-station comments. The correspondence between the mobile research and the social media research caused us to consider two options: 1) gas stations and bookstores don’t incorporate branded coffee shops when perhaps they should or 2) gas stations and bookstores do incorporate branded coffee shops but they need to do a better job branding them. Either way, something needs to be done.
Figure 3
Social media research helps text-messaging research
One of the great advantages of SMS text diaries is that researchers and brand managers can experience a day in the life of people actually using their products. Where are they when they use it? What are they doing at the time they use it? Who are they with? What do they see? Research participants don’t have to try to remember one experience from weeks or months ago or try to aggregate a number of disparate experiences together because they are directly engaged in the activity while they participate in the research. In this case, we asked several hundred research participants to send us a text message every time they purchased a beverage and tell us about their purchase experience. Over a 24-hour period, we received hundreds of individually-crafted, very short text messages.
One of the problems with qualitative data like this, particularly when dealing with vast quantities of it, is that several people are required to code the results. And, as every qualitative researcher knows, when multiple coders work on a single project, there are always concerns about inter-rater reliability, even when coders use the same code book. Coding may become more accurate as coders become more experienced but it may also become less consistent due to fatigue.
An advantage of social media processes, however, is that their automated systems can be applied to any set of verbatim data whether social media data, survey verbatims or text messages. Automated content analysis systems can identify the topics mentioned by the research participants and even recognize alternate phrasings and incorrect spellings. For instance, in this text-message research, many people indicated that they added milk to their coffee. But, some people said they added “2%,” or they had a “skinny” coffee, or they had “mlik” in their coffee. The automated system coded each slang and spelling-challenged word as “milk,” ensuring consistency across all of the records.
In addition, the coding system that was applied to the SMS text data was, of course, applied to the social media data. Even though the text message data and the social media data offer very different styles of language, both used an identical code book, ensuring perfect reliability between the two methods and across hundreds of thousands of data points.
Social media research helps surveys
Research participants answer surveys for a variety of reasons. Some people are genuinely interested in the research process and want to help improve and develop products while others like to participate in research to receive incentives. There are two basic incentives that always rise to the top when participants are asked what they would like to receive. First, they desire some kind of financial reward, whether money or points. Second, and nearly tied in popularity with financial incentives, are information incentives. Even though research participants help to generate knowledge, this knowledge is rarely, if ever, shared back to them. Research participants simply want to know what the results were and perhaps whether something happened as a result of their participation.
However, survey researchers often have trouble sharing research results with responders because the research results are proprietary. One option is to add extra questions to the survey with the intent of sharing those results with responders. But this adds length, something we are desperately trying to avoid. Fortunately, social media research doesn’t have this problem. The unimaginable quantity of data points makes it simple to generate results that can be shared with responders without fearing the inappropriate dissemination of information.
As part of this study, we prepared a set of research results based on the social media research specifically for our survey participants and sent it to them as a thank-you. Short, sweet and appreciated.
Can support each other
These are just a few of the ways that research methods can support each other but there are many more. For example, much research is conducted at one point in time when the research is specifically required. Since social media research can access time-stamped data points from months and years ago, custom survey projects can be complemented with pre- and post- data.
In addition, unique findings from the survey or mobile research can be further delved into with social media research. For instance, where survey responders indicate that they enjoy eating sweets with their coffee, social media research can help identify whether donuts, cookies or biscotti are the sweet of choice and which flavor of each treat generates the highest degree of satisfaction.
Or, when surveys must halt at the “need to know” questions to stay under the 20-minute mark, social media research can help discover the “want to know” information and discover insights that simply couldn’t fit into the 20-minute limit. For instance, which celebrities do your consumers most identify with? Which brands of clothes and shoes do they appreciate? Which vehicles and entertainment systems attract them the most? In other words, what consumer components should be gathered together to create a marketing campaign that is directly relevant to your consumers?
Understand the nuances
Clearly, there is no single research method that can meet the goals of every research project. But, every method has clear advantages and disadvantages. Our role as researchers is to understand the nuances of every method and take advantage of them wherever possible. Where census-rep numbers are required, we must use surveys. Where live data is required, we must use mobile phones. And, where massive quantities of variables are required, social media research can save the day. We must remember this: Think of the research objectives first and the methods will follow.
