Create a rising tide

Editor’s note: Paul Kirch is vice president, business development, at Common Knowledge Research Services, Dallas.

The last time I checked, I work in the marketing research industry. For some reason, I frequently hear that I’m selling a commodity. Crude oil, coffee, grain, hogs, iron ore and sugar are the items I most associate with the term “commodity.”   Yet I have heard many a professional in our industry throw that term around to describe what we have dedicated our careers to promoting.

Obviously, research is valued and useful. But most areas of primary market research suffer from the commodity perception, especially data collection. Price has become such a driving factor on who wins a job that it is increasingly difficult for firms that truly are quality-driven to be competitive, unless they implement cost-cutting and/or corner-cutting procedures.

My career in research started back in 1990. I spent my first 11 years at the Gallup Organization, which was a quality-oriented firm. It wasn’t uncommon for the Gallup name alone to bring business in the door. Today it is harder to rely simply on name and reputation. Those things can still open doors but in most instances clients want to see if there’s another supplier out there who might save them a few dollars.

Since 2001, my focus has been on selling online research. I’ve noticed a lot of negative press recently about online panel management and development. Many industry observers are questioning the integrity of quantitative research conducted online. (Of course, similar things were said about telephone research years ago.) But the issues extend far beyond online research.

As we focus on one methodology’s problems, our industry tends to overshadow the issues confronting other techniques. All forms of data collection are facing falling respondent cooperation rates, yet Internet-based research is currently under the microscope and it is an easy target due to various issues, many related to sampling. These issues have brought price even closer to the forefront of many buying decisions, thus supporting the commodity perception.

As someone who has always represented quality and truly believes in what he sells, I find myself having to defend this methodology, which has been proven to be an amazingly powerful and flexible tool. Powerful and flexible, yes, but dangerous in the wrong hands.

Quite frankly, it has become cheap to open a business if your focus is online research. Also, many non-researchers have professed to be experts at things such as sampling and survey design. After all, if it looks like a survey then it must be a survey, right? By definition, it may meet the criteria of a survey, but that doesn’t mean the end result is quality data. Many of the do-it-yourself tools these individuals are using don’t allow for a significant amount of programming logic. Even if available, improper use of logic can bias or skew the data.

It’s up to the user to understand how to implement these properly. Without understanding the true methodological issues driving the design, these surveys can provide results that give the wrong answer or point to answers that lack real value. Further, they can leave the respondents frustrated by poorly designed, non-engaging surveys. Do these seem like issues exclusive to online research?

Down to sampling

In my opinion, it comes down to sampling, which is the cornerstone of creating quality research, no matter the methodology. If you’re not talking to the right respondents, how can you get the right answers? This is true of any form of marketing and opinion research.

I attended a conference of the American Association for Public Opinion Research (AAPOR) in 2001 at which the academic researchers had a field day with the topic of online research panels. Some felt the issues with sampling bias were leading to a methodology which could never produce quality data.

In the early days of Internet panels, there were certainly huge concerns. After all, Internet penetration was not what it is today and panels consisted of more affluent individuals with higher education levels. Today, the penetration of online has increased dramatically and the representation is becoming less and less of an issue for most consumer and B2B studies. Very low-income households and elderly people are still not represented as much as we all would like, but otherwise the representation is not where the problem lies, assuming the study is appropriate for the methodology.

From the pure academic viewpoint, you are dealing with a sample source that is made up of individuals who are online, thus not representing those who do not use the Internet. Of course, of these online users, panels consist only of people willing to participate in an Internet panel, providing further bias. Recently, I asked veteran researcher Theo Downes-Le Guin, principal of Doxus LLC, what he believes is the No. 1 issue facing the sampling side of our business. “Panels are not going away, so I see the appropriate growth and management of panels as still our greatest challenge,” he said. “Setting aside every other concern, the make-or-break issue is the coverage level that panels offer [the proportion of the population represented by panels]. By the most generous definitions, 90 percent of the consumer population is still not in panels and for professional or B2B panels, it’s often worse. When we can’t capture 90 percent of our population - and the 10 percent or less that we are capturing is not selected systematically - a storm is brewing.”

Looking at the other side of the coin, I have a good friend who sells telephone research. He is very vocal about his methodology being the best way to conduct a truly random sample. Is it? Telephone sampling today can only be random among those individuals who use landline telephones and who are willing to cooperate. According to a survey conducted by the National Center for Health Statistics, 29.1 percent of people ages 25 to 29 and 25.2 percent of the survey respondents ages 18 to 24 have abandoned residential phones and rely solely on cell phones. The report concluded that 12.8 percent of all households are now cell phone-only households, having eliminated their residential phone service altogether.1

Also, with cooperation rates at an all-time low (and going lower) across the industry, the truly random nature of any methodology can’t help but be questioned (sure you can dial random, but you have to sling a lot of mud to make it stick). We have limited pools of respondents to work with and we’re doing more research than ever. In fact, of the respondent pools online, many belong to multiple panels and are participating in a significant amount of the research conducted.

Ignore the issue

The sampling issue is one that a small handful of industry firms are looking at proactively. Unfortunately, others are focused on letting those companies fix the problem we’re all facing, while they ignore the issue and focus on profits. Perhaps this is why some view our industry as quite reactionary. I think we need to band together and come up with long-term solutions that we can all work toward.

I have been stressing for the last three years that we need to educate the public on the importance of participation. Of course, the cost of an advertising and marketing campaign to do this is enormous. A few firms have offered to make a contribution to help fund such a campaign, but most have ignored the call. What else can we do to inform the public?

If you’re a supplier today, inevitably, you’ve lost business on the issue of cost. Our clients all want the best quality, as long as it’s the cheapest. Of course, this isn’t the golden rule, but for many companies it has been their method of purchasing research. Does this sound like a commodity to you? Often, price-based buying decisions lead to quality issues - quality of the data, respondents, programming or overall experience.

I am seeing a shift by some companies who are starting to realize that the lowest bid might just be the one to avoid (especially if it’s too low). Of course, the lowest bid might be the best, but many research buyers are spending more to work with vendors they trust or to evaluate the ones they intend to use. They are starting to replace the word “price” with “value.”

When clients bring up price, bring up value. If you can legitimately differentiate yourself and show your client how they benefit from working with you, price becomes less relevant, and you can win on value. Of course, your client needs to have trust in you and you better be able to stand behind your word. If not, then go back to selling on price. In my opinion, without trust, you have no value.

Rude awakening

Those making business decisions solely on price are in for a rude awakening. In fact, we’re long overdue to see an increase in the cost of conducting primary research. Why? Let’s look at the oil industry. When there is a surplus of crude oil, demand can rise without impacting oil prices for a period of time. When these supplies diminish, and demand continues to increase, the consumer cost goes up exponentially. Pressure to find new sources for oil lead to exploration and drilling costs, along with many other costs that we, as consumers, have to absorb.

It’s ironic that I’m using a commodity as my example comparison, but this really is similar to what the industry has faced over the last couple of years. We have a limited number of respondents participating in research and, even though demand is high, operating costs have not really been impacted. We’re facing a respondent shortage that, not unlike a crude oil shortage, requires efforts to find new and better sources for our survey participants. Unfortunately, this is not simply an online or offline issue. Instead, this will impact everyone in our industry.

To address the issue of quality, more and more researchers have looked at the individuals responding to research studies. Identifying straightliners, speeders and satisficers has become a standard practice by a handful of industry professionals. Finding and removing those respondents they view as answering inaccurately or even fraudulently has made an impact in several ways. As more and more companies adopt this practice, the result is:

  • Cleaner, more accurate data, from legitimate respondents.
  • Panel lists and sample lists are being cleaned when these offenders are identified.
  • The sample pool of respondents is shrinking even more as many fraudulent respondents are removed.
  • Legitimate respondents are doing more and more of the research - many burn out and become satisficers.
  • The internal cost of managing projects is increasing (more time examining data to examine respondent quality).
  • The cost of recruiting and panel management is also increasing.

As you can see from the list above, there are both positives and negatives. I do believe the practice of addressing fraudulent respondents needs to be more widely adopted. However, the downside is we’re overburdening the good respondents. All forms of sampling are facing this. In fact, during the Conference Board’s 2006 marketing research conference in Chicago, Marc Dresner, executive editor for RFL Communications Inc., gave a presentation entitled “Stop Feeding the Elephant.” Dresner cited a quote from a confidential source - whom he referred to only as “a senior executive at a major research agency who has been in this industry for more than 30 years” - who said, “When response and cooperation dipped below 50 percent more than 20 years ago, we all said, ‘This is serious. We have to make sure it doesn’t get any worse.’ When they fell to 40 percent, we said, ‘We won’t have a profession if this keeps up!’ When they dropped under 30 percent, we just stopped counting.”

Inspire and motivate

As our industry continues to face the respondent issues, push the average length of interviews higher and design non-engaging surveys, we are in danger of producing unreliable results.   Yes, there is still great research being done, but the few standing alone in that camp need to inspire and motivate the others or we will suffer as an industry.

I don’t have all of the answers. I do believe there is hope and that there are solutions. However, it’s important that we work together as an industry to give them a chance for success. Below are a few steps we can all take.

  • Survey design

We need to keep respondents engaged. I’d love to see 15-minute survey limits, but many quantitative projects are pushing in excess of 25 minutes. Shorter surveys can dramatically improve the quality of responses for any methodology. Online offers powerful, interactive tools, video and other methods to make the experience fun for the respondents. Are you taking advantage of these tools? Fun, brief or just engaging studies reduce the likelihood of fraudulent activity. In addition, most surveys are designed with the researcher in mind and not written from the perspective of the respondent. Engage the respondents and we all win.

  • Vendor selection

All research firms are not alike. Do your suppliers have measures in place to track fraudulent respondents? If they are a sample supplier, are they removing these from their panel or at least from your study? If your vendor(s) can’t differentiate themselves from the competition, find ones that can, and ask yourself: are the differentiators valuable to you and the industry? Shop value, not price.

  • Get involved

Organizations such as the Interactive Marketing Research Organization (IMRO), the Marketing Research Association (MRA), the Council of American Survey Research Organizations (CASRO), the Council for Marketing and Opinion Research (CMOR), the AAPOR and ESOMAR all provide educational opportunities where you can learn what industry experts are saying about the direction of our industry. In addition, there are volunteer opportunities where you can make a positive impact, gain exposure and give back to the industry.

CMOR is currently working on initiatives to address respondent cooperation and data quality. I encourage you to contact Patrick Glaser of CMOR at pglaser@cmor.org to find out how you can contribute or make a difference. In addition, visit www.youropinioncounts.org, a program dedicated to ensuring the public voice by supporting the mutually beneficial relationship between the public and opinion research.

  • Educate

Educate your clients, colleagues, vendors and others on your best practices and methods for improving our craft. Not only will you be making a positive impact, you will establish more credibility for you and your firm. How are you making a difference?

  • Contribute

The Marketing Research Association recently announced the Education Endowment Fund (EEF). Providing scholarships for students who wish to enter our industry, the EEF provides an avenue to provide exposure for our industry and secure qualified applicants for the next generation(s) of research. By contributing to this fund, we’re all helping secure the future of our industry. Visit www.mra-net.org to learn more.

  • Lobby

When the Do Not Call list was threatening the future of telephone-based research, there was one saving grace: Politicians rely on the research we collect during political campaigns. The research provides data to support issues, candidates or initiatives. When there was a threat to our existence, they made sure research was exempted while telemarketing was not. I believe this may be an avenue to gather support and funding for advertising campaigns to educate the public. Contact your senator and other representatives asking for support. Also contact organizations like IMRO, CMOR, MRA and CASRO asking them to get industry lobbyists to focus on this issue.

Higher standard

Many research firms are increasing their internal costs to address industry concerns. Due to current price sensitivity, these costs are usually not passed on to clients. Unfortunately, to stay in business they will eventually have to be. We likely will see some research firms shut their doors, but those that remain will help reinvent this industry that I love, and make it more sustainable.

I am concerned about the future for all of us. Personally, I am here to promote value-driven research and I hope that we can work together to keep the word “commodity” out of our vocabulary. After all, most commodities end up in storage, in another product or in someone’s belly. I can’t say I like any of these options.


Reference

1 “More Americans Go for Cell Phones, Drop Landlines,” San Francisco Chronicle, May 15, 2007. www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2007/05/15/MNGMOPR2HE1.DTL< /A>