By the Numbers: Market research survey routers? Approach with caution

Editor's note: Efrain Ribeiro is COO of Lightspeed Research, Warren, N.J.

Routing techniques have been around since CATI was the state-of-the-art in market research. But, despite routers’ longtime presence on the survey scene, our experience with clients suggests that many buyers of online panel are unaware of their prevalence in today’s online research. There also is a lack of awareness of the many different ways routers are being utilized and the potential impact that they can have on the research results.

Routers are powerful tools and they certainly have a place in our online toolset, as they can improve a respondent’s survey experience and help match a willing survey taker with an appropriate survey. However, more study of how they are employed and their subsequent effects on the research results is warranted.

Due to their widespread use, I suggest that our industry needs to develop a set of evidenced-based best practices for routers, including a pledge by sample suppliers to transparency on both router use and methodology.

In traditional online panel research, pre-profiled respondents are drawn to produce a balanced, representative sample of the desired study target population. These prospective respondents are then invited by e-mail to participate in the survey and through a link in the e-mail are driven to the survey.

The best panel providers collect a wide range of information from their panelists to aid in the building of representative panels and in the management and administration of surveys across their panels. This process also allows the panel supplier to verify what recent surveys the respondent has participated in and apply the necessary category exclusions to help control for bias.

This was the approach most research-informed panel providers took initially. And based on my discussions with end-clients, this is the way most believe the samples for their studies are constructed.

But increasingly over the past decade, sample providers began taking advantage of growing Internet traffic by employing Web-intercept techniques to source surveys. They discovered that anyone who had access to high volumes of online traffic flow was able to create large, inexpensive sample by attracting respondents with online ads, profiling them dynamically and driving them to a set of open surveys.

Though this was a clever way of monetizing the rapidly growing online audience, little thought was put into the selection process and how to match respondents with open surveys. This initial approach produced unreliable research results and thus these early-stage online intercept surveys are now widely accepted as bad research.

While online routers are wonderful for making optimal use of participants when they are ready to take surveys, they can be dangerous if the selection process is not carefully designed to help reduce bias and control the survey opportunities. Rather than improving the survey experience, badly-designed and -managed routers can create a seemingly endless string of survey offers that annoy and repel potential participants.

Similar negative impact

I believe that some poorly-used survey routing is having similar negative impact on market research as the early intercepts did over 10 years or so ago. We need to do more to help clients who use online research better understand the ramifications of router use on research results. That is why I am excited to be part of the ARF’s Router Committee that includes both end clients and key online sample suppliers. This will give the industry – with client involvement – the initiative to investigate current router methodologies, their use for online sourcing of respondents and the eventual development of router best practices. Here’s why:

In today’s online market research, clients want to interview smaller and smaller targeted populations. The days of the census-representative “gen-pop” sample are past. More and more clients are interested in small and precise populations, with specific buying habits and attitudes, who reside in narrow geographic areas. For example, whereas clients of the past may have desired to target women aged 25 through 49, today the same clients want to focus on women aged 25 through 49 who drive a minivan, own a home in a suburban setting on the East Coast and regularly buy organic produce.

This example clearly presents a challenge for the online sample provider – it requires a large pool of pre-profiled respondents to meet the sample size. Today, average respondent qualification incidence of studies is running below 20 percent. This means that four of five willing respondents are turned away from completing a study they have been invited to. A survey router, carefully administered with the appropriate research sampling considerations, can alleviate this situation by helping match disqualified respondents with other study opportunities. So survey routing has evolved to help deliver those targeted samples, finding available online panel members and matching them to surveys they can qualify for and take. Responsible users of routers limit the number of surveys a respondent is exposed to and utilize the respondent’s survey history to guide him or her to the appropriate study.

But there is concern today with routing. At the moment it is not clear when suppliers are using routers to direct respondents to surveys or what the decision process is for matching respondents to surveys. At Lightspeed Research our priorities are both panel quality and ensuring that our panelists have a positive research experience so they will return to take more studies. Moreover, as part of Kantar, we are one of the biggest buyers of supplier samples and we have been concerned about the broad use of routers in today’s online research. As a result we have dedicated significant resources to investigating how the sample suppliers we utilize employ survey routing on the studies we commission.

In the course of our investigation we have identified suppliers who utilize routers more than 95 percent of the time and others who only use them occasionally and for very specific studies. Some will allow survey-takers to complete an unlimited number of surveys in one sitting. And to complicate matters, those suppliers who rely heavily on routers also depend less on pre-profiled panel sample and more on live Web traffic for sourcing. This broad range of activities is happening because there are no current rules or guidelines for the use of routers in our industry.

These learnings have prompted us to direct our suppliers 1) to use routers on our studies only when we explicitly ask for them; 2) to utilize panel-sourced sample on our studies; and 3) to be transparent on their routing methodology. These measures help us ensure the consistency and quality of the sample we get from our suppliers.

Critical part

Today at Lightspeed Research we use routers but have firm rules on the studies to include. As a result fewer than 5 percent of Lightspeed Research’s online interviews are currently router-sourced. Kantar firmly believes that survey routers are a critical part of the online research future. Their role in enhancing the respondent experience is essential if we are to maintain consumer interest and ongoing participation in the research process.

Further, we are confident that the ARF Router Committee and other industry efforts – with the participation of online sample suppliers – can help educate research buyers in the use and potential impact of survey routing on their research results. This initial step will set the foundation for future work to establish standards and practices that will help guide the responsible use of routers throughout the industry so that they can become a reliable and welcomed way to source online respondents.