Compare and contrast

Editor’s note: Thania Farrar is multicultural research director at the Cincinnati office of research firm TNS. Neil Schwartz is chief research officer at the Torrance, Calif., office of TNS. Natalie Mayor is consumer and market knowledge manager at Procter & Gamble, Cincinnati. Dave Anthony is senior manager - global research analytics at Procter & Gamble.

Are you shaping good brand decisions with your selection of research methods? Today, high Internet penetration levels among the general population allow researchers to reach a vast majority of U.S. consumers via online panel methods. At the same time, it is becoming increasingly important to ensure proper representativity of fast-growing multicultural groups in research samples, primarily African-American and Hispanics.

However, with the Internet penetration among these groups currently at much lower levels than the general population, it is important to stop and ask ourselves if online respondent panels are suitable for drawing samples to research these groups with confidence. The onus is on marketing research companies to demonstrate the representativity of their online samples, or absent that, to offer the caveat that such samples have not been demonstrated as being representative of their populations. Client companies should demand this - most cannot afford to risk multimillion-dollar decisions or hard-earned brand equities on results that are misleading due to non-representative sampling. This article is intended to motivate marketers and researchers to take a hard look at their sampling methods, question the status quo and make the right method choices when it comes to learning about multicultural markets.

Substantial benefits

Internet-based consumer research provides substantial benefits in cost and timing. With nearly three quarters of the U.S. general population accessing the Internet, 30 percent of U.S. survey research spend is now conducted online and that percentage is growing.1 However, lower Internet penetration rates exist among African-Americans (49 percent overall; though lower among lower income/urban segments) and Hispanics (40 percent, lower among the critical mass of Mexican-origin and Spanish-dominant consumers).2 These groups are increasingly attractive to U.S. marketers given that they now represent over one quarter of the U.S. population, with Hispanics contributing to half of recent population gains. Traditional best-practice research for these multicultural groups has remained offline in large part due to their lower Internet penetration rates.

For African-Americans, traditional interviewing modes are mail, phone and central location testing (CLT); for Hispanics, the preferred modes are phone and CLT. These modes are often expensive and time-consuming, especially when compared to the efficiencies achievable online.

With these dynamics in mind, TNS, and its multicultural arm, TNS Multicultural, created the TNS Multicultural Consortium, consisting of founding members the Coca-Cola Company, Freddie Mac and Procter & Gamble and associate member General Mills. The charter of the Consortium is to form a group of non-competing companies interested in better understanding the multicultural universe as it relates to marketing, retail and research best practices.   The Consortium’s first priority was to conduct a research-on-research study to examine the ability of Internet fielding to provide representative samples of African-Americans and Hispanics. The study used a single questionnaire covering basic brand metrics across six categories, administered to parallel cells of mail and online African-Americans, and CLT and online Hispanics.

The research-on-research objectives were:

  • To determine how well African-American and U.S. Hispanics can be represented through traditional and online means.
  • To build an understanding of the conditions under which it is reasonable and not reasonable to use online samples to represent African-American and U.S. Hispanic consumers.

Specific actions to be taken were set from the beginning:

  • If results are similar between the methodologies, Consortium members could pursue their own methods-related research in their category (ies) for basic brand and category questions to ensure that the same business decisions can be reached online.
  • If results differ significantly, testing should not be moved online for these clients’ categories.
  • Additional methods work can be done in the future to examine situations in which results line up reasonably between the methodologies.

Parallel cells

Samples were designed to achieve well-matched aggregate demographics within ethnicity across modes and employed each mode’s best practices.   The African-American parallel cells were mail- and online panel-based, consisting of female heads of household aged 18-65. Additional specifications were identical between mail and online panel samples balanced to the African-American population on national geographic region, market size, age, income, household size and education. For Hispanics, parallel CLT and online cells were conducted in the respondent’s language of choice. CLT interviews were conducted via a Web-enabled computer-assisted interview. The samples consisted of female heads of household aged 18-65. Additional specifications for both modes were age, country of origin and language spoken at home most often. Furthermore, the online sample was balanced for national geographic region, market size, income, household size and education (per online best practices). The CLT sample covered the top seven Hispanic DMAs.

Each Consortium founding member contributed questions pertinent to their brands so that a variety of brands/categories and question types were covered. Figure 1 shows the categories and metrics which were examined for the online and offline modes.

For each ethnicity, individual key measures for each category were compared between modes, looking for the total number of significant differences at the 95 percent level of confidence. This tally helped determine the extent to which results lined up between modes.

The specific action criteria within each of the six categories (by two ethnicities) were set up before the research began as shown in Figure 2.

Did not meet the standard

Only one category - carbonated soft drinks within African-American - lined up between online and offline. The other categories within the African-American samples, as well as all categories for the Hispanic samples, did not meet the action standard.

Of the remaining five categories deemed “poor” among African-Americans, three of these results were due in large part to systematically higher brand awareness seen online. With knowledge of the reason behind the difference, these “poor” outcomes were footnoted to be cautiously considered by the client for further methods testing (Figure 3).

Differences among Hispanics

The differences observed among Hispanics online compared to CLT were many. At least three major reasons emerged. While overall quotas for both online and CLT samples were achieved, there were a number of differences within aggregate demographic quotas - finer breaks are not typically controlled for. For example, both Hispanic samples achieved about 65 percent Spanish-dominant respondents (that is, speak Spanish only or Spanish more than English, at home). However, Spanish-only speakers were 42 percent of the CLT sample and only 25 percent of the online sample.

It was also apparent that the online sample exhibited oddities that could be due to the type of respondents available online, especially for variables that are not under standard control. For example, nearly one third of online respondents were students, compared to only 5 percent via CLT. Younger online Hispanics were more Spanish-dominant when in fact the younger generations tend to be more bilingual.

Finally, there were also attitudinal differences between modes. For example, over half of CLT respondents agreed that “a woman can best contribute to her family by staying home to care for them,” compared to about one fifth online.   Hispanics who are online seem to be early adopters and are more acculturated than the representative population. They tend to have higher incomes and levels of education, much like the early experience with the general-market online audience.

The conclusion is that online Hispanic panel sources are not representative of the adult Hispanic market because the critical groups - Spanish-dominant and of Mexican origin - are not well-represented on the Internet. TNS’ 6th dimension access panel is subject to these characteristics even though its recruiting uses bilingual, content-appropriate invitations and sites. The universe TNS draws from - as is the case for all online panels and sites - is subject to the unrepresentative characteristics of those Hispanics currently available online.

Differences in metrics among African-Americans

The differences in parallel metrics among African-Americans occurred in brand awareness, usage, disposition and attitudes, despite the successful matching of online to mail aggregate demographics, requiring only minor sample weighting. The aggregate similarities mask fundamental differences between the online and offline respondents, again likely because critical segments such as lower-income and urban respondents are underrepresented in the online universe, and in TNS’ panel.

The reason carbonated soft drinks did better than other categories could be due to the fact that Coke is a universally recognized brand in a highly-penetrated category. On the other hand, financial services fared the worst (73 percent of parallel metrics differed), possibly due to the characteristics of this category or the nature of being online.

Question remains

So the key question remains: Are you shaping good brand decisions with your selection of research modes? It is especially critical to follow the market research tenet of using a mode that provides the degree of representivity needed to meet the study objectives. There is substantial business risk in using non-representative sources for making decisions intended for such broad “total” populations.

This research-on-research suggests the following business conclusions:

  • For Hispanics, online is not a representative method. Results from online data cannot be projected confidently to the Hispanic population. Rather, the offline modes of CLT and phone are recommended.
  • For African-Americans, online is generally not a representative method. Results from online data cannot be projected confidently to the African-American population. Again, offline modes, e.g., mail, CLT and phone, seem best. This research suggests there may be some exceptions to investigate online if a category/brand is very strongly penetrated (as was the case with carbonated soft drinks and Coke in this study).
  • Market researchers and clients must work together to understand the objectives of the research and resulting level of population representativity needed. This understanding is critical to selecting the appropriate methods and sampling plans to enable the research to accurately inform business decisions.
  • Even when samples appear demographically balanced, they may be concealing “nested” demographic differences, and may still reflect respondents whose attitudes and characteristics differ from their overall population in fundamental ways.
  • Currently, there are very limited uses for online research focusing on Hispanics and African-Americans. Such uses might include studies that do not require representation of the total populations, or for investigating online Hispanics, or online African-Americans.

It is the responsibility of market researchers to understand and communicate the limitations of their samples in order for their clients to measure the risk associated with the business decisions being made. TNS will continue to explore the issue of representativity of these populations via its online panel, and will update the industry on progress as the Internet penetration of these markets grows.

References

1 Pew Internet & American Life Project, Data Memo by Mary Madden, April 2006, and Inside Research, July 2006.

2 EMarketer article “Reaching Hispanic Internet Users.” April, 2006; and TNS Multicultural.