Build a consensus, focus on quality

Editor’s note: Robert I. Tomei is chairperson of the ARF Online Research Quality Council and also the executive vice president and global director, 6thDimension Access Panels, at Taylor Nelson Sofres.

On August 3, 2007, the Advertising Research Foundation (ARF) launched its new Online Research Quality Initiative with the formation of an Online Research Quality Council. The core principle that will drive the activities of the council is that of knowledge-driven standards and metrics. Through various research-on-research studies, the council aims to address increasing concerns surrounding the quality of online research, especially regarding the repeatability of research projects and the qualification of respondents. The practical mission of the council is to develop a set of standards and metrics that research buyers and providers can support and follow. Council leadership is in the hands of a steering committee of six senior executives from market research companies: comScore, Harris Interactive, Ipsos Interactive Services, Synovate, the NPD Group and TNS.

The ARF received an overwhelming response to this announcement. Almost immediately after hearing about the quality initiative, more than 150 stakeholders, with strong representations from both research providers and client organizations, expressed their intention to take part. This level of excitement proves that the practice of online research and the value it provides are urgent concerns throughout the industry. Excellent work is being done in the industry today by such associations as ESOMAR, CASRO, ISO and others to develop standards for online research. However, three key aspects of the council distinguish it from other efforts:

  • The council will be guided by an extensive client advisory board that, to date, includes the committed, active participation of approximately 10 companies.
  • The work of the council will be driven by leveraging the industry’s existing knowledge and knowledge creation conducted by the council’s working committees.
  • Ultimately, the council will deliver a series of metrics - all placed in the proper context of quality standards and processes - for the industry to use in evaluating the quality of online research.

Online research that lacks quality and generates results based on questionable information can undermine the value of market research. Consequences could include a poor understanding of customers, flawed decisions and strategies that neither address actual market developments nor allow an organization to compete effectively. Given the immediacy of the concerns, the council set to work quickly, entering its discovery phase with a “call for knowledge” to collect, review and summarize the factors that determine the quality of online research.

Increasing concern

The compelling promise of online research is that it can deliver relevant business insight - fast and in a cost-efficient manner. In recent years, many research users and providers have acted on this opportunity. According to the July 2007 issue of Inside Research, approximately 38 percent of all survey-related research today is conducted online. However, there is an increasing concern within the industry that online research does not deliver a consistent level of quality that would allow decision makers to employ findings strategically and confidently. Prominent research users, providers and analysts have repeatedly voiced concerns about the quality and validity of online research studies as they are currently conducted.

Several factors threaten the credibility of online research:

  • Many research providers and their clients may no longer be aware of how much validation was performed when research initially moved into the online universe.
  • Clients can perform online research in many ways, but to deliver results quickly and inexpensively, stakeholders may discard traditional checks and balances.
  • Some companies that offer online research may not own the full spectrum of applicable expertise or may lack the resources to verify that people they target are a representative sample. Such companies may have extensive contact databases and proficiency in online marketing or list management, but lack essential research qualifications.
  • The online environment has changed the dynamics of research to give panelists an increased level of control. A relatively small number of highly active online respondents - people who participate in surveys exclusively for financial gain, often using software programs to automate survey completion - may completely skew survey results. Validation techniques exist to eliminate such isolated, unconstructive responses, but the implementation of these verification practices can add time, complexity and cost to research projects.
  • It has become more difficult and expensive to motivate, recruit and retain people to participate in research surveys, especially longer surveys. When projects are propelled by the need to deliver information fast and at a low cost, the temptation to dismiss the stringent procedures required to qualify participants and validate their responses may become overwhelming.

One should also question the current value relationship between the velocity of research projects and their low costs. Does it really make sense to expect that online research - delivering results in the shortest possible time - should be less costly than more traditional research practices? That attitude disregards the fact that the increased speed at which research delivers actionable, meaningful results represents a higher business value to the research user, who could consider adjusting the amount invested to the actual return received.

However, most of these considerations will remain academic as long as the research users are not involved in shaping the research industry that caters to them. Within the industry, we find initiatives and efforts to set quality benchmarks, but research users are largely absent from the equation. And, unless an organization has its own in-house research specialists and a solid understanding of what makes online research valid and worthwhile, it will be difficult and expensive for a client to conduct a reliable assessment of a provider or a project.

Refocus the discipline

The ARF’s Online Research Quality Council aims to refocus the discipline of online research with a commitment to essential, basic values, including all of the appropriate checks and balances to ensure optimal quality. The council believes that the only way to make a quality effort successful is to deliver meaningful metrics and standards, and that research buyers alone will have the power to enforce adoption of relevant standards across the industry.

In order to arrive at such effective metrics, the council’s client advisory board and the active participation of many research companies will drive the work of the council. Truly meaningful metrics would measure the quality of insights delivered through research, as experienced by the clients. Metrics need to be clearly based on what research buyers consider relevant and place an appropriate price-value relationship on that level of quality. While the council will leverage other standards efforts, the quality of metrics should not be solely influenced by existing standards that individual research companies may already implement. This is an opportunity for the entire industry to enhance those standards and implement them consistently going forward.

Most immediately, the Online Research Quality Council will focus on issues related to online research panels and progressively broaden its work to include all aspects of online research. As the council evaluates and defines metrics to assess research panels, its scope will include close consideration of the following dimensions of online panel research:

  • appropriate measurement of panelists’ response rates;
  • assessment of contact rates;
  • survey duration;
  • sampling criteria;
  • proper sample balancing methodologies;
  • best practices in panel management;
  • incentives for panel participants; and
  • data collection methods.

These standards and metrics will result from knowledge derived from the research-on-research studies that will be designed and implemented in the next few months. The council will publish its findings in a special section of the ARF Web site and plans to deliver a draft of global panel research standards at the ARF annual convention in March 2008 in New York City. As much as possible, the council will incorporate ongoing standards efforts that seek to establish guidance for recruitment, incentives, usage controls, business processes and other aspects of panel research.

Once these new metrics and standards have found acceptance by the larger community of research purchasers and providers, research users will benefit tangibly as they acquire and review online research vendors and projects. The ultimate goal of the council is to design an easy-to-use dashboard as a helpful evaluation tool.

The Online Research Quality Council wants to make it as easy, economical and meaningful as possible to conduct regular assessments of online research. With reliable assessments based on accepted industry standards for quality, research projects indeed can deliver valuable insights to help clients drive smart business decisions. For research providers, the ability to demonstrate adherence to such standards will help prove their commitment to client value and excellence in research.

Deliver more value

As the Online Research Quality Council succeeds in creating and propagating effective standards for the online research industry, providers and buyers will be able to focus on the insights provided rather than be concerned about the quality of information collection. This will yield an opportunity for all research companies to deliver more value to clients through business-sector expertise:

  • Given optimal standards for recruiting and managing panels, research will become accountable for delivering actionable insights and findings that accurately reflect the customer segments to which our clients are attempting to market and that allow organizations to focus on strategy and making business decisions.
  • As research users understand and apply the council’s guidelines, research providers will follow their clients’ lead and align behind the new quality orientation. Competing vendors will find many ways to distinguish themselves but they will share a commitment to quality along with the best practices to measure and substantiate quality over time. This alignment will have an enormously positive impact on the industry. Certain providers may not support the new quality standards, and potential clients of these companies will need to come to acceptable trade-offs between price, value and quality.
  • With a renewed focus on quality in the online research industry, perceptions and practices regarding the value of research will probably shift. When research quality becomes quantifiable as a result of the efficient application of practical metrics, quality considerations can inform purchasing and project decisions. Price and speed of research will always matter but the measure of quality will render them in a meaningful context. Quality can then drive financial and resource commitments.

Kick-off meeting

The ARF held the Online Research Quality Council’s kick-off meeting on September 10 in Manhattan. Participants included over 125 executives from a diverse set of companies, including a large number of organizations that regularly purchase research services. Council leadership shared its mission and working timetable, and introduced four working committees. Research industry executives from Burke Inc., Global Market Insite Inc. and Knowledge Networks presented research-on-research regarding the value of approaches and tools in use today.

The members of the council’s working committees will be participants in the steering committee and client advisory board as well as volunteers from organizations that purchase or provide research. Of the four committees, one is dedicated to outreach communications and securing funding. The other three will tackle questions of quality, metrics and standards:

  • Defining quality committee: The committee will investigate the key aspects of quality that require standards and industry alignment, including panel management processes, research design, questionnaire design, data processing, reporting and other issues. The committee also will consider issues related to the replication of research, validation practices and the representativeness of online panels. A large part of the effort will be dedicated to studying the role of respondents. The committee will define types of respondents and respondent behaviors, their impact on research results, the factors that influence respondent types and existing methods to identify and control respondent involvement.
  • Quality matters committee: This committee seeks to establish meaningful relationships of cost and value. For example, the committee will determine the cost of low quality, define the key components of quality in a market research setting, review possible thresholds for these components and consider how they balance with other values. In addition, the committee will find ways to improve the business processes and feedback loops involving suppliers and clients and to communicate quality and its value more effectively to people who are not research experts.
  • Metrics committee: This committee will gather and document currently-used metrics and standards throughout the industry, review best practices and metrics at play in online sampling approaches, identify ways to create consistent, industry-wide metrics and standards and develop strategies to drive effective adoption.

The council’s steering committee will meet every month. The working committees will meet as needed, updating the steering committee and the client advisory board every month. General council meetings will take place quarterly.

Rapid timeline

It’s noteworthy that the Online Research Quality Council is pursuing a very rapid timeline. Quality concerns in the industry are becoming more urgent and it would not be appropriate to plan on several years of process. Starting in January 2008, the council will field research-on-research studies. At the annual ARF conference in March 2008, the council will be ready to review status and preliminary results. In June 2008, the council will publish white papers documenting its accomplishments and findings to the industry and seek adoption of standards and metrics. From September 2008 onward, the council will communicate and reinforce its findings more broadly and promote adoption of the new metrics and standards across the industry.

Compelling benefits

Online research offers many possible applications with compelling benefits for the industry and research users. Once the Online Research Quality Council has made advancements in setting standards for online research panels, it will apply itself to other avenues of the online world. For example, online communities, blogs and social networking sites have excellent potential for research practitioners and users, especially if the council can develop and implement meaningful standards that are overwhelmingly supported by clients and providers.

If you are interested in the work of the ARF Online Research Quality Initiative, the council welcomes your ideas and insights. Please go to to review the latest findings and standards projects.