Editor's note: Based in Singapore, Ati Sinaga is knowledge specialist on the knowledge management team at SSI. Based in London, Pete Cape is SSI’s director, global knowledge management.

Western researchers have voiced concerns about the quality of the responses coming from Chinese respondents, especially when it comes to noticeable over-claim of specific brand usage or other behaviors. In all other respects the data is “good” and consistent but there remains a nagging doubt about what is being claimed. We need to remember, of course, that China is a rapidly growing economy and, at the same time, a country that most researchers will not be personally familiar with. Stereotypes abound therefore whilst Western consumer goods brands are literally falling over themselves to exploit a market that is, in gross terms, the second largest economy in the world.

Given the Chinese psyche, it was our view that some of the China-related data-quality concerns could be caused by the ways questions are being presented and answer choices being given. We therefore devised experiments that varied the question and answer style. We also thought it possible that the over-claim may be more related to what might be described as “aspirational” categories. Almost as if the respondent is answering what they would like (or expect to get) next, not what they have right now. Our experiments therefore covered a number of different categories.

As all researchers should be aware, many of the problems in international research stem from translation issues. China, being a society only relatively recently open to the rest of the world, has an additional problem over and above the simple translation of words. This is the problem of translating a concept that might be totally alien to the Chinese whilst being known, even if not well, outside of China. We found in the leisure category, for example, that poor translations were responsible for both over- and under-claim of participation in some outdoor pursuits – proof that more time and care than ever, particularly in back-translation, needs to be applied in the China market.

The notion of aspirational answering is intriguing. We chose the automotive category to test, since it is rapidly maturing and therefore is on the cusp of being mainstream and non-aspirational. To get a relatively clean measure of ownership, one sample answered the question using an open question. This produced answers not dissimilar to expectations, given available sales figures. Volkswagen, for example, which had a 20 percent share of sales in 2011, was owned by 24 percent of the sample. GM (with 16 percent of sales in 2011) got 16 percent mentions. These brands are both mainstream and the prompted version of the question, asked to a matched sample, produced not dissimilar results. Volkswagen was claimed by 28 percent and a GM marque, Buick, was mentioned by 9 percent (having gotten 8 percent in the open question).

There is, however, a marked difference when looking at more aspirational brands. For BMW, eight times as many people chose this in the prompted version than wrote it in in the open question. BMW was the most aspirational brand on our list: 33 percent of Chinese car owners said they would like to own one. Mercedes-Benz, the second-most desired brand, had three times as many people claiming ownership in the prompted than in the open. There was a clear relationship between over-claim and aspiration.

Intriguing finding

An intriguing finding arose from this first experiment. In the prompted version of the questionnaire only 8 percent used the code “none of these” and thus it appeared that we had captured 92 percent of the brands using our list. However, coding out the open question using only these same brands implied that “none of these” ought to have been something more like 30 percent. Could it be that Chinese people, as well as being aspirational, were somehow averse to using “none of these”?

In order to check this hypothesis we repeated the experiment using two versions of the automotive brand list. The first was extensive, covering almost all the brands currently available in China. The second was a subset of these with either “none of these” or “other – please specify” as the final code. In neither case using the shortened list did the correct predicted number of people use either “none of these” or the “other” category. Using the full list, however, produced data comparable to our original open question.

Chinese respondents appear to avoid using non-substantive answers. While we cannot explain this behavior, we can control for it in our questionnaire design. From this research it is apparent that the Western practice of using a truncated brands list with a catch-all “other” should be avoided. Either a full brand list or an open question, possibly with a self-coded follow-up, should be used in preference.

Just to prove beyond doubt that respondents were not deliberately misleading us, a third experiment took those who had been exposed to the truncated brand list and interviewed them yet again but this time using the full brand list. At this point the data fell back into line with expectations.

Problems will still occur when the category cannot be adequately described in an open question without the presence of the brands to make sense of it. In another experiment within the premium bath and skincare sector all the brands are aspirational and all suffered huge over-claim. A full brand list of all bath and skincare products was unwieldy (and impossible to procure) and the open question became one of self-classification (since one person’s premium brand is another’s “day-to-day”). In these instances the standard Western approach of “Which have you bought in the past X period?” might better be replaced with “Which brands do you currently have in your bathroom?” Other category problems occur when the brand of interest straddles two or more categories: For example: is golf a sport or a pastime?

Further problems occur when the category is highly aspirational and particularly when the brands are few and well-known. We were unable, through any questioning style, to get an accurate read on foreign travel. We did no further research into this category but would have looked to the respondent to validate themselves somehow as a traveler to that country.

Solving these seemingly intractable problems does require a different, possibly radically different, approach to question design, but it also requires the researcher to have faith that the respondent wants to tell truth and to work with them to that end.