Data Use: Culture-based scale response bias among Hispanics

Editor's note: Jim Starks is a partner at Cultural Edge, a San Diego research firm. Starks is based in Fort Lauderdale, Fla.

It is hardly news to frequent practitioners of multicultural research that different cultural groups respond to scalar questions differently. In the U.S. particularly, scale response differences between Hispanics and non-Hispanics have long complicated market researchers’ attempts to draw meaningful conclusions about differences between them. It is broadly accepted that Hispanics tend to utilize extreme scalar points more frequently, frustrating direct comparisons with non-Hispanics.

There has been significant research on this topic of late, given the importance of Hispanics in the U.S. market. For example, a 2008 University of Florida Study1 explored the paradox that Hispanics report health care similar to or less positive than non-Hispanic whites, yet have more positive ratings of care. The researchers concluded that Hispanics exhibited greater usage of extreme responses (particularly high scores) than non-Hispanic whites. Additionally, Robert Culpepper (Stephen F. Austin State University) and Raymond Zimmerman (University of Texas, El Paso) concluded from a study published in the Journal of International Business Research in 2006 that “Hispanic respondents exhibited not only extreme response bias … but also used significantly fewer midpoints on the scale.”2

For those among us who specialize in multicultural research, any conclusion that generalizes the behavior of all Hispanics generally raises suspicion. We acknowledge that it may be true, but the premise lumps all Hispanics together as a homogeneous group and very often results in misunderstandings and poor interpretation of research results. Since we know that there are a wide variety of cultural differences among Hispanics (U.S.-born vs. foreign-born; recent immigrant vs. long-time resident; Cuban vs. Mexican, etc.), it is reasonable to ask if scale usage might not also vary. To be certain, it is not just pure curiosity. Many of us have noticed patterns in both qualitative and quantitative research.

Developed a test

In order to determine whether scale usage within sub-groups of the Hispanic population is more nuanced, we developed a test using a database of studies conducted over the past three years. We utilized six studies, some with multiple waves, that all contained a single question type. We felt it was very important to limit the potential for extraneous variance, so we included only studies that contained a scale running from 1 to 10, with only the endpoints anchored. The anchors in all cases were “10 means ‘applies completely’ and 1 means ‘does not apply at all’.” In short, we used only one specific question type to look for patterns of scale usage among Hispanics and thereby eliminated potential variation as might occur with different scales.

The studies, however, did cross categories and brands, ranging across beverages, auto and retail. The Hispanic samples consisted of males, females, more acculturated, less acculturated, married, single and a wide variety of other demographic characteristics. Some of the studies were telephone-based (CATI) and some were self-administered (intercept-to-Internet). Finally, the number of brands and number of attributes per brand varied across studies. In the end, we had 2,695 respondents and more than 150,000 brand attribute rating points from which to draw conclusions.

We focused on one specific type of response, the top three-box (an 8, 9 or 10), given its widespread usage in analysis of research results. We calculated the percentage across all respondent ratings in any given study that were top three-box scores. While the percentages ran from 0 to 100, the average Hispanic respondent used a top three-box score 56.5  percent of the time in the average study.

We chose to examine the following demographic variables, because they were common across all studies utilized, and we hypothesized that they could be predictors of top three-box usage: gender, age, language used at home, country/region of birth, time-in-country (U.S. residence), marital status, education, income and methodology. Across the categorical responses, it is possible to see variation of top three-box usage.

Only country/region of birth and time-in-country present very large differences in the raw data, but the numbers presented in Table 1 are means. It is possible to imagine that a female, aged 35+, who speaks only Spanish at home is quite likely to use top three-box ratings more often than a male, aged 18-34 who is English-dominant.

The analysis consisted of a simple backward stepwise regression, with the logit of the respondents’ top three-box score percentage as the dependent variable. The independent variables were effects-coded. Six variables were eliminated from the model as non-significant: methodology (CATI vs. CAPI), age, language used most frequently at home, time in country, marital status and income. This left three variables that are statistically significant predictors of top three-box scores: gender, country/region of birth and education. A goodness-of-fit test based on the chi-square statistic shows the resultant model to be better than the intercept-only model with a 99 percent probability.

We should point out that some of the variables removed do appear to have some effect on top three-box usage. In particular, time-in-country stands out. On closer examination, however, much of the difference in the time-in-country variable comes from the difference in U.S.-born vs. foreign-born, an effect already reflected in the variable country/region of birth. In short, time-in-country does not add much new information once we take country/region of birth into consideration.

It is a small matter to compute the relative importance of each. Given these three variables, the smallest proportion of top three-box scores given would be 42.3 percent and the largest would be 67.5 percent, a difference of 25.2 percent. The importance for each variable is simply the percentage of total variation contributed by each (holding all other variables constant).

The resulting intercept, coefficients and variable importance proportions are shown in Table 2.

Important clues

While we can show significant variation in top three-box usage across different subgroups of Hispanics, our analysis does not explain why. There are, however, some important clues in the data and some aspects of Hispanic culture that may lead us to compose some reasonable hypotheses.

The most significant variable is country of birth, and there are two relatively obvious implications. We assume that the moment of birth and its location are not the foundations of scale usage, but perhaps certain formative years in those cultures are. It could be early childhood, grade school or even secondary school where scale usage patterns are formed. While it makes perfect sense that U.S.-born Hispanics would exhibit scale usage patterns similar to non-Hispanic Americans, it remains an open question as to why Mexican-born and Central American-born Hispanics would be more likely to use top three-box scores than Caribbean-born and South American-born Hispanics.

Secondly, on the issue of country/region of birth, time-in-country does not appear to affect significant differences in scale usage for foreign-born Hispanics. While acculturation is a very complicated subject, it is reasonable to assume that Hispanics, even those living in the U.S. for a very long period of time, do not become exactly like their non-Hispanic counterparts in every way. Some behaviors appear to change relatively rapidly, some slowly and some not at all, since they do not alleviate any tension of living in a foreign host culture. This appears to be a reasonable hypothesis of why scale usage patterns do not appear to shift markedly with time-in-country.

The second most important variable is gender. In fact, in qualitative and quantitative research, this is perhaps the most notable. While it is easy to jump to conclusions regarding stereotypical Hispanic gender roles (machismo and marianismo), we find that these terms are too often misunderstood. In fact, we find that understanding of these terms relies more on caricatures than on the nuanced reality of culture-based gender roles among Hispanics. Still, in research, females are significantly more likely to express optimism, politeness and/or a lack of discrimination of brands and products at the high end than are males. This should certainly be a subject for further exploration.

Finally, education appears to have an impact on top three-box usage on attribute scales, particularly whether respondents have finished a secondary education. Again, why remains an open question. We can theorize that deference (or politeness) due to social status may play a role. Social status in Latin America is often driven not only by income, as we generally think of it in the U.S., but also by education, among other elements. The answer is far from clear and it may just be that less-educated Hispanics are more optimistic or do not tend to differentiate much among brands and products at the high end. Further research along these lines would be very interesting.

Some particular advice

The question remains as to what to do with these observations. In that regard, we do have some particular advice for practitioners of Hispanic research. There are three potential avenues to deal with the knowledge that some Hispanics are more likely to use higher scale points than others: sample management, normalization and use of other question types.

For any wave-based study or study where subgroup comparisons are important, we highly recommend careful sample management. While our preference is to recommend careful quota construction as part of any sampling plan, we acknowledge that costs and timing constraints are often prohibitive. In this case, weighting can be a useful tool, although not without its own pitfalls. At the very least we recommend controlling for country/region of birth and gender. If possible, educational attainment should be considered as well.

We are aware that some practitioners use various techniques for “normalizing” data among Hispanic respondents. The results of this study are a very poor basis for doing so. Using these results to normalize Hispanic response data on an individual level would result in less differentiation, not more - the exact opposite of the problem that we would intend to solve. Instead, we do believe that these results can be useful to normalize data on a group basis for relevant between-group comparisons or for comparing with norms.

We can encourage multicultural practitioners to utilize other question types that are not so sensitive to scale-usage bias. This study only deals with one type of scale (1-10, anchored) in one application (attribute ratings). It is possible that other scale lengths, such as seven-point or five-point scales, may be enough to shift behavior. Additionally, verbal scales may be more effective in eliciting differentiation among brands, products and concepts. As mentioned previously, other research does indicate that Hispanics in general do have a tendency to utilize extreme points on scales more frequently, but the effects may not be as pronounced.

Finally, practitioners of Hispanic research may choose to skip scales altogether in favor of choice-based respondent tasks. It is not uncommon in Hispanic research to have very little differentiation among brands, products and concepts when measured with scales and have very significant amounts of preference when measured with choices. Even choice-based questions with degree of preference follow-ups preserve a significant amount of differentiation.

Useful indicators

In conclusion, we have shown that top three-box scale usage among Hispanic subgroups can be affected by culture, resulting in non-random patterns of response. We have also shown that three variables in particular (country/region of birth, gender and education) are useful indicators of higher usage. In the end, we have offered some hypotheses for causes of this variation and some advice for avoiding the problems associated with this scale bias.

It is our hope that other practitioners of multicultural research will find these observations helpful in their own work, that customers of multicultural research will use these observation to request more effective study designs and that all readers are inspired to deepen their understanding of our diverse and unique American marketplace.


References

1 Weech-Maldonado R., Elliott M.N., Oluwole A., Schiller K.C., Hays R.D., “Survey response style and differential use of CAHPS rating scales by Hispanics.” Med Care. 2008 Sep;46(9):963-8.

2 Culpepper, R. and Zimmermann, R.A. (Fall 2006). “Culture-based extreme response bias in surveys employing variable response items: an investigation of response tendency among Hispanic-Americans.” Journal of International Business Research, 75-83.