Managing for maximum return
Editor’s note: Beth Rounds is senior vice president, strategic accounts at MarketTools, Inc., a Mill Valley, Calif., research firm. Heidi Dickert is the firm’s vice president, client services.
Recently there has been a hot debate over the quality of online samples. Companies who promote the “number of qualified members” as the most important criterion in evaluating a research panel may fail to mention critical factors that affect panel health. Researchers who choose quantity over quality may also compromise the integrity of marketing research information, and ultimately may jeopardize the business decisions made using the data.
Undeniably, panel size is important. A large pool of respondents is essential to achieving results that yield census representation or adequate numbers of lower-incidence audiences. However, it is equally important to optimize panel member usage, and balance new and experienced members to maximize overall panel health. Passive panel management can impact data integrity and survey results in a variety of ways. For example, our research has shown that panel tenure significantly impacts a respondent’s views. Therefore, a healthy panel must include an appropriate mix of new and experienced members.
When choosing an online panel, there are five key questions that need to be asked to ensure that the panel is representative and the quality of the data it provides meets the highest standards:
1. How frequently do panel members receive survey invitations?
2. What is the mix of new and experienced panel members?
3. What is the click-through rate of the panel?
4. How does the online provider handle integration of new members?
5. What is the panel turnover rate?
Optimizing online panel activity
Panelist activity level is extremely important in ensuring that respondents provide the most realistic input. We have found that using panel members too much will result in incomplete surveys, while underutilized panelists will more frequently provide biased data or engage in straight-lining or random clicking that negatively impacts survey results.
In conducting research on this topic, we’ve noted results that revealed “respondent conditioning” that may set in as respondents approach a total of 20 surveys, even if those surveys have been taken over a period of six months. To manage this issue, it is essential to distribute survey invites in a measured way to keep respondents productive and engaged. In fact, one survey invitation per week appears to be an optimal activity level to balance the risks of under- or over-use of panel members. This is well below the industry average of 2.5 survey invitations per week, and nowhere near the activity levels of some research panels.
Balancing new and experienced members
Balancing panel tenure is also essential to managing overall panel health. We have found that newer panelists tend to be more optimistic than their more experienced counterparts, giving higher ratings to product concepts. At the other end of the spectrum, very experienced panelists tend to be more critical of concepts. Therefore, a panel that is comprised of too many new or very experienced panelists fails to represent the broader population, and can compromise data integrity.
Research recently conducted by MarketTools, Inc. clearly supports these observations. Figure 1 shows “intent-to-buy” scores by experience level. Results between groups vary dramatically. The Top 2 box “intent-to-buy” for inexperienced panelists was 49 percent compared to 30 percent for the very experienced group. This variance was consistent throughout the results.
Minimizing turnover
A typical online consumer panel has 25 percent to 35 percent turnover each year. If you include inactive members, this could be as high as 50 percent. As demonstrated in the previous section, a panel that is tilted toward inexperienced or very experienced panelists will not represent the broader population. Therefore, keeping turnover low is important to prevent one group from dominating the panel population. It is equally important to implement programs that ensure the panel population remains appropriately balanced across experience level.
One such program guides the productive integration of new panelists. As seen above, new panelists tend to provide overly optimistic responses to survey questions. To combat this bias, we recommend creating an assimilation program to help new panelists move through the experience curve before they are exposed to live surveys. This important step enables the new panelists to become comfortable with the survey process and technology without having negative impacts on survey results.
Maximizing the “fun factor”
Lastly, it is important to make your surveys enjoyable and compelling in order to maintain engaged and active panelists. One way to do this is to develop and refine respondents’ tasks to make them more realistic and interesting. The ideal is to closely mimic the real-world activity you are asking respondents to consider. Exciting new technology, such as interactive shopping exercises and media environments, is expanding what is possible with online research.
Interactive retail environments are one such exciting innovation. Mock store settings are painful and expensive to create. Moreover, they typically fail to represent a broad geographic spectrum. Working with virtual retail environments, researchers are able to include a much more representative sample. But just as important, respondents are wowed by the experience. And intuition tells us that, because it is a familiar activity, their input is likely to be more realistic than if it was based on a less true-to-life online exercise.
Pay close attention
As you can see, the health of your online research panel plays a critical role in the quality of the data your clients receive. Panel tenure, composition, activity level and the survey experience itself can have a significant impact on results. If you pay close attention to these critical areas, and maintain effective programs to actively manage them, you’ll enjoy richer results from your panel. More importantly, you will deliver more accurate data to your clients so they can make better business decisions.