Cross-culture calibration

Editor’s note: Susan Devlin is founding principal of The Artemis Group, a Green Brook, N.J.-based research firm.  Gary Nicholas is client services manager at NetReflector, Inc., a Seattle research firm.

Over the past few years, trends toward globalization have led international corporations to conduct more cross-cultural survey research to validate overseas business opportunities in today’s highly competitive global market. When time, budget and resources are no object, a company can afford the best that international research has to offer. However, today’s economic downturn has compelled many businesses to face the harsh reality of shriveling timelines, slashed budgets and limited resources, therefore limiting their international research choices.

Having said that, making concessions and trade-offs doesn’t necessarily mean that enterprises have to compromise the quality of their international research in the process. In the context of the legitimate constraints caused by today’s economy, this article is aimed at providing market researchers with best practices in identifying the pitfalls of country comparisons while highlighting the need for a delicate balance between design uniformity and cultural customization, and showing the outcome and implications of cultural calibration on research results.

The need to adjust for cultural effects

Companies with a worldwide presence must conduct research across all their global markets, present a combined corporate view, and contrast these markets to identify hot spots for special initiatives or new ventures. They need methods to minimize cultural effects and adjust results to differentiate response pattern differences due to language or cultural norms affecting surveys from real differences in performance or perceptions due to products, services or market conditions.

Although literature abounds about cultural differences1, little practical work exists about how to tackle cross-culture calibration of measurements. Some researchers propose avoiding a single global survey approach and suggest conducting independent research in each country without combining results and even with different content. Whatever approach you decide to take, the first issues you will face include harmonizing languages as part of your questionnaire design as well as implementing the challenges that data collection poses, as further explained below.

Questionnaire design hot buttons

Seeking a culturally neutral questionnaire is like seeking the Holy Grail – it can never be achieved fully. What is achievable is a questionnaire that works in all countries, minimizes differences, allows for country customization where necessary, and provides the opportunity for cross-culture calibration. Below, we discuss three questionnaire design issues, including language, question content, and scale, as well as address the importance and scope of survey validation.

  • Use locally acceptable language.

Although English is the de facto international business language, this doesn’t mean that using English in international business surveys is always acceptable. Except for select large business arenas, one cannot assume most of the target population is fluent, limiting the sample frame and introducing selection bias. Even for fluent English-speakers, nuances may be lost when English is a second language or a different dialect, subtly changing the meaning of questions. In addition, using English globally may be perceived as socially unacceptable or even an insult, reflecting negatively on the survey sponsor. This sensitivity varies culturally and by delivery method. In Japan  for example, it is not recommended to conduct telephone interviews in English. Web surveys on the other hand are acceptable, especially if translated into Japanese (see screen shot).

Thus, the native language is recommended as a first choice for interviewing, unless within the country or cultural group language is so splintered that a common language exists, e.g., Mandarin in China  and English in India for business surveys. In contrast, to capture the opinions of the majority of ethnic groups in South Africa, research might need to be conducted in English, Afrikaans, Zulu, Xhosa, Sotho, etc. Even a common language may require different scripts for different countries, e.g., Spanish in Latin America vs. Spanish in Spain.

No matter the extent of the resources you have available for your international research project, questionnaire and material localization is an essential requirement. After localization, a critical quality-control check is to have it translated back by a different translator, to see if the meaning has changed by comparing the back translation to the original text. In addition, we strongly recommend a check by a native speaker in the sponsoring country who will best know subject-specific jargon or technical terms. It is important to stress that localization doesn’t just mean translation, especially when it applies to a survey. Proper localization, which sometimes implies content change, requires a certain amount of expertise. Here are some initial considerations to take into account:

  • Seek globally acceptable questions.

If you want to measure and compare culture, strong evidence already exists in social science literature (e.g., Hofstede, Triandis2 ) of universal social dimensions that can be integrated effectively into research instruments to measure the effect of culture and recommend strategies accordingly.

However, business research measures attitudes regarding products and services and the understanding of the impact of culture on these measures, rather than the direct measurement of culture. While many questions can be used successfully across countries, titles, use of names and order of questions may be affected by cultural norms. After finalizing the general questionnaire content, but before translation, surveys should be reviewed for each target country to identify problem areas.

Inquiries about price are a good example of a culturally sensitive question. Asking about the acceptability of price usually gets a fairly high response rate in the U.S.  , but such inquiries are impolite in many European countries and get a lower response rate. However, asking whether prices are “competitive” is more acceptable. While response rate is still lower in Europe , the difference would be reduced significantly.

  • Choose easily transferable scales.

Even within one culture, seeking an effective scale can be challenging. While there is no best rating scale, a good one should minimize positive response bias and maximize reliability3. We haven’t found any scale that, after translation, consistently has equivalent properties across cultures (scale calibration is discussed further later in this article). There are some practices, based on experience, which minimize differences.

   While both fully verbalized scales and numerical scales (with just verbal anchored end points) are used effectively in the U.S., we prefer anchored numeric scales for cross-cultural research. What might appear to be subtle differences due to translation of scale points often introduce significant changes in the use of scales. Numerical scales in cross-cultural research have the advantage of limiting this risk.

We have found that 1-10, 0-10 or 0-100 rating scales have a greater diversity in how they are used cross-culturally than other scales. Their association with grading systems, which vary cross-culturally in meaning, may cause discrepancies in use. 1-5 or 1-7 scales are more neutral.

End-point wording and translation should be pre-tested to understand where a scale splits between negative and positive. Subtle changes can drastically change positive response bias, especially when taking into account cultural tendencies. For example, top-box use is much stronger in Latin countries (e.g., Italy  or Brazil ). While using a very strong upper anchor helps in these cases, it never fully combats this tendency. In contrast, countries like Germany tend to avoid top-box, suggesting the wording should be a slightly softer choice. U.S. respondents are fairly neutral between these extremes.

Informational scales (income, ethnicity, profession, etc.) have other problems - some cultures view them as too private to reveal in a survey. Where units or labels change, direct translation may be desirable from an analysis point of view, but may be awkward for respondents.

  • Validate questionnaire country by country.

Without survey validation, you may turn “garbage into gospel.”   Validation has many phases, from pre-testing the survey to statistical analysis of the reliability, validity and usability of items, based on the resulting data. In cross-cultural research, it is recommended to validate the survey in multiple markets. For the pre-test, there is a tendency to test in a local market first to iron out large problems in wording and flow, in an effort to avoid the cost and time of double translation. This is very risky, as discussed above. If your resources are limited, initial pre-testing should be done at minimum in one key representative market for each diverse cultural area. For example, in one study conducted of 12 North American, European and Asian markets, the U.S., Italy and India could be chosen as representative markets for each respective geographical area. Ultimately, pre-testing is needed in every market to ensure the survey works. Post-survey validation is also needed in all countries, or major cultural groups, to help interpret results.

Implementation challenges

Besides questionnaire development, survey design includes selecting the right delivery vehicle and a sample plan, which offer additional challenges in the context of international research.

  • Match delivery channel to cultural requirements.

Using one delivery method is recommended to provide greater consistency. However, this may not always be possible, depending on the culture and logistics of the countries involved. In Japan  for example, telephone interviews are generally not accepted. Face-to-face interviews are more the norm, but online surveys are now also common practice. In India, face-to-face interviews are often cheaper and more practical as telephone service is too unreliable for most interviews.

Web surveying provides excellent, cost-effective alternatives to phone or face-to-face interviews, enabling the collection of international data in local languages. The survey is programmed once and centrally managed, without requiring local software installation. Whatever online survey technology you choose to use, make sure it has been designed to accommodate multiple languages, including double-byte Asian languages, as well as the automated translation of all error messages, navigation directions and buttons.

If you are on a tight timeline, the speed benefit that online surveys offer is undeniable, not just from a data collection standpoint, but also from a reporting perspective. To illustrate this, let us quote the example of a major U.S.  high-tech company that wanted to measure the level of satisfaction of its customers with their worldwide partners. The firm translated an online survey in 26 foreign languages, and within four weeks, its customer satisfaction measurement program was up and running and data could be reported upon in real time across all 26 languages.

Overall, the delivery method chosen should be culturally appropriate locally, provide quality control and be cost-effective. What satisfies these needs will depend on the nature of each study and the set of countries you conduct research in.

  • Plan for sample inconsistencies.

Purchased sample frames often differ dramatically across countries. In some markets, researchers can choose population sample frames that are proportionate geographically, by income or by education. However, in many other countries, only data about urban populations is available, or gathering data among rural populations is very costly or even impossible. Therefore, each market needs to be evaluated individually, without ignoring the global objective of the study. Even if internal customer records are used to draw samples, it is important to make sure that the practices or sources for these records are consistent across markets.

Differences may be addressed through weighting. However, this requires advanced, detailed understanding of differences.

  • Match social and regulatory market differences to interviewing practices.

Understanding social and economic market differences is important to response rate and response validity, e.g., who is the right respondent in a business or household, what is acceptable to control interviewer gender, or is there a constraint on interviewing hours. In-depth local knowledge on the above-mentioned areas increases cooperation rates and, ultimately, the study’s success and accuracy.

Finally, researchers must know local regulations. Laws differ widely regarding acceptable practices in the use of data or the identification of respondents.

The end goal: parsing out cultural factors through analysis

Next, we must differentiate universal response patterns or performance differences from response patterns that are culturally caused.

  • Integration of culture-free questions.

The most common recommendation found in research is to measure the true impact of culture through a series of culture-free questions integrated into the survey.

Differences found on those questions, due to individual differences rather than culture, can help adjust other responses. However, the practical application of this to adjust for question and scale differences on performance measures has not been fully addressed.

  • Cross-country scale calibration.

Let’s consider this example: a manufacturer distributing products and supporting services globally conducts a satisfaction or loyalty survey about their customers’ experiences. The survey uses a common rating scale for nearly all items. The goal is to provide global metrics for a corporate scorecard to represent the customers and to identify which markets need to be targeted for improvement initiatives.

The trellis plot in Figure 1 shows the difference between “% Satisfied” and a simple cross-country worldview average. Bars to the left correspond to country values which are lower than the average. Large dots depict significant differences. Bars and dots to the right are doing better than the average.

Country 1 is neutral, close to the world view. Country 8 is low on all measures, and the differences are significant in most cases. Countries 4 and 5 score higher than the norm.

Are these cultural differences or problems that should be addressed? The scatter plots suggest that to some extent, cultural differences are clouding the issue. Much of the difference between “% Satisfied” for Countries 4 and 8 compared to Country 1 follow a pattern, independent of the item or brand being evaluated.

Since it is very unlikely for performance differences to all be in the same direction, this suggests a hypothesis for calibration in this case: Systematic differences between responses that are independent of the question or brand being evaluated can be attributed to culture or to systemic country differences in questionnaire or delivery. By modeling the pattern for each scale point, a calibration formula for each country is developed, using one country as the base. Country 1 was chosen, as its response pattern is most neutral.

The trellis plot in Figure 2, showing the calibrated results, suggests very different conclusions about where the challenges lie. Rather than being a problem area, Country 8 tends to follow a global norm along with Country 1. Country 4 is still the greatest success story. Problem areas are identified in several other areas masked before.

Advance planning

No matter the global research trade-offs or concessions a company needs to make because of the constraints imposed by today’s economy, a successful implementation will always require thoughtful advanced planning of design and data collection, expert knowledge about research norms in different countries, and analytical capabilities to separate out cultural effects to get the best possible results.

By using the techniques for cross-cultural factors demonstrated in this article, a company can:

  • understand and appropriately respond to the needs of its international customers,
  • differentiate cultural issues from performance issues, and
  • expand the breadth of its global reach to appeal to more audiences.

References

1 See extensive review of literature and comprehensive reference list in Hofstede, G. Cultural Consequences: Comparing Values, Behaviors, Institutions, and Organizations Across Nations. Second Edition. London : Sage,  2001.

2 Hofstede, G. Cultural Consequences: International Differences in Work-Related Values. London : Sage, 1980. Triandis, H.C. et al. Handbook of Cross-Cultural Psychology (Vol.1). Perspectives. Boston : Allyn & Bacon, 1980.

3 Devlin, Dong, and Brown. “Selecting a Scale for Measuring Quality”, Marketing Research   (1993), Vol. 5, No. 3, pp. 12-17.