My November 2006 Trade Talk focused on a conference I attended in Chicago last fall on the topic of respondent cooperation. The event was organized by Bob Lederer and the staff of RFL Communications Inc. - publisher of Research Business Report and its associated newsletters - and the Council for Marketing and Opinion Research and conference coordinator IIR.
Among many newsworthy moments at the gathering - which included a panel discussion with 33 industry figures, including client-siders, providers and research association heads - were expressions of concern about data quality from researchers at prominent consumer companies such as Procter & Gamble’s Kim Dedeker.
While much of the discussion stayed focused on respondent cooperation, the public comments of Dedeker and others drew attention to the topic of data quality and got the industry talking about this important issue.
To keep the momentum going, Lederer and RFL have organized the upcoming Client Summit on Research Data Quality, which will be held on September 19-20 at the Gleacher Center in Chicago . (For more information visit www.rflonline.com/clientsummit/index.asp .)
As the title suggests, it will be a client-only affair - clients in the audience, clients as presenters. Attendance will be limited to 70 participants. “There is real power in one client talking to another,” Lederer says. “They are not trying to sell anything or pitch a project.
“The idea for [the client-only summit] came from the respondent cooperation meeting last year. We got tremendous feedback from McDonald’s, GM and other people, who said they learned a lot from Kim Dedeker and the research-on-research that P&G had done. Many said they were unaware of this level of concern over data quality,” he says.
Lederer has speakers lined up from firms such as General Mills, Washington Mutual, ESPN, Eastman Kodak, P&G, Capital One and CBS. Scheduled topics include issues related to moving telephone research online, dealing with professional respondents and implementing quality checks into online data-gathering. “Our aim with every presentation will be to conclude with bullet-points for the audience: What are questions you need to ask internally? What are questions to ask suppliers in order to alleviate these problems?” Lederer says.
Expecting cries of foul from research firms, Lederer has spoken to many company execs to gauge their reaction to the client-only nature of the event. Only one person expressed reservations. “Everyone else said go and do it because [suppliers] are having a hard time getting through to clients about the issues surrounding data quality. Suppliers feel that this will elevate education and awareness and make people at the client companies realize that there are problems facing the industry.”
Another discussion
Lederer moderated another discussion on this same topic last month in San Francisco at the MRA’s annual conference. Panelists included the always-eloquent Bill MacElroy of Socratic Technologies, Jeff Miller of Burke, Carol Teter of P&G, Denise Offutt of Epson and Kathleen O’Reilly of International Insights.
(I didn’t attend as many presentations as I wanted to at this year’s MRA event - though as always I enjoyed chatting with readers as I worked our booth in the expo. Those that I did take in were uniformly excellent. Particularly good were talks by Michelle Adams of Frito-Lay, who detailed her efforts to transform the company’s consumer insights function, and Tom Asher of Levi Strauss & Co., who showed how much a group of interested, enthusiastic company employees can learn by spending a day immersing themselves in their customers’ lives.)
Similar to views expressed last fall in Chicago , the MRA panelists were insistent that the onus is on both the research providers and the client companies to strive for quality in data gathering.
Client-side researchers have to realize that, in most cases, you get what you pay for, that “cheap is expensive” in the long run if poor-quality data leads to poor-quality decisions. Further, “trust but verify” is a good phrase to live by. Vendors will do their best to deliver on their promises, but the purchasers of research services have to be willing to take the time and ask questions about panel respondents and how they were recruited, etc. Indeed, as MacElroy said, “The more we as researchers visit, observe, track and ask questions, the better the quality will be.”
Offutt has tried to ask questions but reported hearing replies such as, “You’re the first person who has ever asked about this” when she has raised data quality questions with prospective suppliers. In other cases, the phone has simply gone dead when she has asked about validation. A vendor could go a long way toward earning Offutt’s business by mentioning not only that it has a panel of 2.5 million respondents but by also then detailing what it has done to confirm and profile those respondents, she said.
O’Reilly said researchers have to maintain attention and awareness of what can go wrong. She said the third-party suppliers she works with have welcomed her scrutiny.
MacElroy said he has seen a huge change on the client side and that many are now much less focused on price and more focused on quality. “As more and more clients ask more questions about quality, the research firms who provide low quality will be weeded out,” Miller said.
Miller noted a change in research company advertising since the events in Chicago last fall. Ads used to tout a company’s speed of data gathering and now they trumpet data quality.
Alienate respondents
Panelists also spoke of taking steps in the questionnaire development phase to avoid fielding overlong or confusing surveys that alienate respondents and cause even the most sincere survey takers to lose interest. “It’s hard to label someone as a problem respondent who straight-lines during the last five minutes of a 20-minute survey. They are simply tired and we have done that to them,” said MacElroy, drawing a spontaneous burst of applause from the audience.
(Similar opinions expressed at other respondent cooperation/data-quality discussions have inevitably drawn the same type of confirmatory response from onlookers, which has always made me wonder: If everyone is so committed to making surveys shorter, who are the people still doing the half-hour surveys?)
While much of the current focus is on the quality of online data-gathering and the makeup of online panels, as O’Reilly noted, quality should always be top-of-mind, no matter what the methodology. “You have to go into your research knowing that there are limitations to each method and plan for them and work with them,” she said.
Courage and commitment
Lederer is to be commended for championing data quality and helping bring it to the fore. And kudos go as well to client-side people such as Dedeker for having the courage and commitment to speak out and engage their peers in discussing - and acting on - this issue. With an entire industry thinking about and working toward solutions, I’m confident real progress can be made. As P&G’s Carol Teter said in San Francisco , “This isn’t something that one person, organization or company can solve.”