As I’ve previously written, the advent of our annual salary survey lets me sample - albeit briefly - some of the highs and lows of data analysis, specifically coding and processing responses to open-ended questions. For this year’s version of the survey, we included the same two open-ended questions with the client-side and vendor-side salary surveys and they generated massive text files, which have been alternately tedious and a ton of fun to analyze.

One of the questions asked respondents to declare what they viewed as the biggest challenges facing marketing research in the next few years. While a number of factors emerged, the two mentioned most frequently were the twin plagues of DIY research and social media data.

DIY tools like Survey Monkey, many respondents said, are doubly damaging because they put research capabilities in the hands of people who may not know how to use them and they also cheapen the perceived value and importance of the internal research function. This forces the research department to have to, in the words of one respondent, “explain our value-add in a Zoomerang world.”

Some other comments:

“We need to educate research consumers on the value of doing research the ‘right’ way. Just because data collection tools have become so readily available (e.g., Survey Monkey) doesn’t mean using them assures good research. In my mind, it’s like saying that we are all qualified CPAs because we know how to use Excel!”

“The rise of Zoomerang and Survey Monkey is destroying the science of marketing research. Anybody can create a survey but they are not likely to get proper results if they haven’t studied methodology and design.”

“We need to address the onslaught of ‘cheap and fast’ research that is often convenient but leads to completely incorrect assumptions. We must demonstrate the return on investment of thoughtful and considered research.”

Of two minds

Respondents were generally of two minds on the topic of social media data. While they recognize that it’s here to stay and acknowledge that it has a role as a listening device, many seem overwhelmed by the thought of having to make sense of it and are also alarmed that internal clients view it as a (potentially free) replacement for ad hoc research projects.

As one Quirk’s reader put it, social media makes some marketers feel, “that they know their customers sufficiently without marketing research. These days, members of the general public make their voices heard through all sorts of media, including social ones, and some may feel that those voices are representative. In essence, they are saying, ‘We’ve given the public sufficient attention through listening to what they’re saying in letters, blogs, tweets, etc.,’ but not really recognizing that simply giving time to ‘the public’ does not equate to understanding the feelings and motivations of their customers.”

Further, said another, “there’s a trend toward mining social data as a substitute for conducting research, which may be useful in some regards, but could also end up ‘listening to all the gossip in the neighborhood’ rather than providing actionable data.”

Some researchers expressed feelings of being damned if they do, damned if they don’t when it comes to social media and other new data-gathering techniques. By ceding ground to or giving too much legitimacy to social media data, researchers could be rendered irrelevant in the eyes of those hungry for “insights.”

But despite anger over “the perception that social media is the answer to EVERYTHING!” these data sources can’t be ignored, especially when internal calls for mining them grow louder by the day. And the researcher who ignores them or denigrates them runs the risk of being seen as out of touch (at best) or unhelpful (at worst): “By not keeping up with newest Web technology, social media, competitive intelligence, we will allow others to supplant what should be a market research function.”

Dearth of analytical skills

Other oft-cited problems included the economy and the impact of tightened budgets on research departments; declining respondent cooperation rates; the dearth of analytical skills among new or incoming researchers; and several expressions of concern about our ability to capture consumers’ attention in a fast-paced, tech-crazed world of smartphones, tweeting, texting and Facebooking. (Interestingly, there were also multiple mentions of NPS as a contributor to the dumbing-down of research by reducing everything to one score, as if that one score were sufficient enough to tell the entire story.)

While a host of external factors like these were mentioned, some respondents pointed a finger back at themselves and their peers:

“Too many researchers see/envision the value of research in and of itself. Too few can make a clear business case for their research. In particular, non-commercial (govt. and academics) give research a terribly bad name/image (i.e., eggheads who are theoretical twits, suck off other peoples’ money/resources and do little other than study ‘stuff’ that matters little if at all). I’m quite serious about the foregoing comments.”

“Staying relevant. Researchers need to adapt their presentations to actually drive action on the insights obtained. The ones that get it get rehired and/or promoted. The ones that don’t get stuck in the same job for 15 years with no advancement, always wondering why.”

“We need to focus on the quality of research. With the number of companies, different technologies, new methodologies, overseas analytics, we need to be very diligent with the data. Make sure our findings don’t end up under the heading of ‘lies, damned lies and statistics.’”

‘This Titanic’

Then again, in the view of one particularly gloomy respondent, perhaps all of the above discussions are moot, as we are all doomed in the long run: “Well, number one, the whole economy is going to collapse again, worse than this last time because the lessons of this last time have not been acted upon. Seriously. I think the global threats way outweigh any little contests for advantage between us as marketing researchers and other passengers on this Titanic.”