Editor’s note: Julia Lin is project manager, research data and analytics, research operations at the Singapore office of Gartner Inc. She can be reached at julia.lin@gartner.com.

In international business research, especially IT research, the content can be highly technical, with a lot of unfamiliar terms or jargon in the questionnaire. From October to December 2007, Gartner conducted a global consumer study to understand consumers’ usage of and behavior regarding telecommunication technologies. A lot of time and effort went into the questionnaire, making sure that the technology terms were clearly defined and explanations were provided. The translated questionnaires were checked by industry experts in the respective regions who are also native speakers. Despite these efforts, we still encountered some mishaps in certain countries. Fortunately pilot studies were conducted and such mistakes were uncovered at an early stage.

Shown below are a number of cases that went awry (or ended up yielding very strange results!) in the field.

Case 1: Mobile instant messaging (IM) usage in Russia

Our data said : 60 percent usage of mobile instant messaging at least once per month in Russia.

Market reality : 1 percent usage of mobile instant messaging.

What went wrong?  Respondents might have mixed up IM with short message service (SMS) text messaging; 60 percent could be both IM and SMS (some early quasi-IM services were delivered through SMS, which could also cause confusion).

Measure taken : Include explanation of the difference between mobile IM and SMS.

 

Case 2: Mobile music downloaded in India

Our data said : 43 percent of respondents downloaded music (complete songs) using a mobile phone at least once per month in India.

Market reality : Actual number unclear, but expected to be lower.

What went wrong?  Respondents might have mixed up mobile music downloading with ring-tone downloading.

Measure taken : Include additional description and explanation of what mobile music downloading is.

 

Case 3: Access to mobile video streaming or TV in Nigeria

Our data said : 25 percent usage of mobile video clip downloads, video streaming or mobile TV entertainment (typically using the 3G mobile phone technology) at least once a month in Nigeria.

Market reality : Actual number unclear, but expected to be lower.

What went wrong?  There is a possibility that respondents over-claimed the usage. Among top three mobile services operators (Glo, MTN, Celtel) only Glo offered 3G services in 2007. 

Measure taken : Filter data based on mobile operator usage. Remove the relevant usage data if it is from a MTN or Celtel user.

 

Case 4: Use mobile phone to access live video in China and France

Our data said : 17 percent of respondents in China currently use a mobile phone to access live or recorded video stored on other devices, such as a PC or video camera; 10 percent in France.

Market reality : Such service is nonexistent or with very little visibility in China; the actual number in France is unclear but was expected to be lower.

What went wrong?  Respondents might have confused streaming live or recorded video with downloading videos from PC to mobile or uploading videos from mobile to PC, i.e., loading the videos (stored in PC) via USB I/O or Bluetooth to the mobile or loading the videos (stored in mobile) via USB I/O or Bluetooth onto PC.

Measure taken : Change “access” to “stream” and explain what stream means: play back a file while downloading it, instead of waiting until the download is complete.

 

Case 5: Penetration of cable and satellite TV in India

Our data said : Almost 100 percent penetration of cable TV or satellite TV in India (60 percent penetration of cable TV and 54 percent penetration of satellite TV).

Market reality : 60 percent of the nation’s television households subscribe to the cable or satellite services in India.

What went wrong?  The study was conducted in metros and tier-one cities of India, where penetration rate of TV is quite high, with almost every household owning a TV.

Measure taken : Highlight the geographic coverage of the study when quoting the data.

Scare off respondents

Terms and jargon are open to misinterpretation, especially after translation; excessive use of jargon could scare off the respondents and result in lower cooperation rates. Therefore jargon should be avoided as much as possible, especially in business-to-consumer research, when the target audience may not know industry terms. If jargon must be used, all the terms should be well-defined and thoroughly explained to ensure data quality. For personal surveys, interviewers need to be properly briefed, so that they can use the jargon correctly and confidently. A fieldwork manual including all the explanations should be prepared for easy reference during interviews.

Use of abbreviations is not encouraged, but if they must be used, they need to be explained. Never assume that the person being interviewed understands the terms correctly. Translation needs to be checked by professionals with industry knowledge. Though costly, back-translation is sometimes necessary in order to improve the validity of the study.

A misrepresented or biased sample can be dangerous. The incidence of usage can get inflated if, for example, the sample of respondents interviewed is skewed toward high incomes, even if everything else was properly administered. Quotas of age, gender and household income or socioeconomic class are often necessary to ensure that data can be projected to the general population. Sometimes weight should be applied to adjust the amount of influence each respondent has on the overall data set.

A pilot study should be conducted to understand what could go wrong and what preventative measures should be taken, especially in countries with historically problematic data. Sometimes a second pilot is necessary to test whether the revised definition works or not. Interim data is also important in revealing bad data.

Solid local team

In a global study, working with an international agency with a solid local team is crucial to success. Researchers who understand the market well are able to provide valuable insights and defend the data when necessary.

When examining data from an international study, bear in mind the diversity of behaviors in different countries. A particular technology can be mature in some countries, making related terms popular and less likely to be misunderstood, while in other countries the technology can be embryonic, and more care should be given in explaining its terminology.

Even within one country there can be a discrepancy of economic and technological development in different regions, especially in big countries like China and India. It’s a common practice in commercial research that studies are only conducted in major cities in China and India; such data can not be used to extrapolate to the population and represent the country as a whole. Researchers should always put the data in the right context by highlighting the geographic coverage to the stakeholders of the research.

Last but not least, as users of the data, we need to be judicious and sometimes take the data with a grain of salt. If in doubt, always consult an industry expert.