Editor’s note: Timothy Taylor is vice president, financial services at MORPACE International Inc., a Farmington Hills, Mich., research firm.

In market research, we often run into situations where we are trying to understand what may be driving significant differences in satisfaction levels between two separate groups of respondents. This article aims to help researchers discover (or rediscover) a useful analytical tool to better understand the true differences in satisfaction between respondent groups.

Let’s take the example where there are two independent samples - one consists of users of Product A and the other of users of Product B. Let’s say that Product A receives significantly higher satisfaction ratings than Product B. Are these differences due to the fact that Product A is actually better than Product B? Or could there be variation in the demographic composition of the two user groups that could be accounting for the observed difference in overall satisfaction?

For instance, certain demographic groups are known to give consistently higher satisfaction ratings. So if more of these individuals make up the population evaluating Product A than Product B, does that account for the observed difference in overall satisfaction between the two products?

It’s a critical question, and one that might be addressed through traditional weighting - by adjusting the populations so that the demographic profiles of the two respondent groups are nearly the same. However, weighting may not always be the best solution given that the model used for weighting the data may be somewhat arbitrary. For example, should the Product A group be weighted to match the demographic profile of Product B, or vice versa? Or, should both groups be weighted to some separate standard? Also, if the weighting factors become too large, undesirable error could be introduced into the analysis.

To address this problem, a technique called analysis of covariance or ANCOVA can be successfully applied. This technique has been around for many years and is well known among statisticians. However, non-statisticians may not be as familiar with the technique and therefore be more likely to turn to basic weighting to answer the type of question posed here.

The ANCOVA technique looks at the correlation between a dependent variable (overall satisfaction in this case) and the covariate independent variables (the demographic variables) and removes the variability from the dependent variable that can be accounted for by the covariates. Differences in the residual dependent variable as a function of the original independent variables are then tested for significance. The focus of this analysis is whether the observed differences in satisfaction are still true after the differential demographic composition of the groups has been taken into account.

Case study example

To illustrate how this technique might be applied, and what the output might look like, the following is an example from an actual research effort (the client and product names have been kept confidential).

In this example, a major bank was looking at the satisfaction levels of one of its cobranded credit cards and comparing it with satisfaction levels with its basic, non-rewards card. What it found was troubling: The rewards card showed a 15 percent lower top two box satisfaction rating than the basic card. However, a quick demographic analysis revealed the rewards customers to be younger, higher-income Caucasians - known for often having lower satisfaction ratings than their counterparts.

So the question became, is the large gap in satisfaction due to these demographic differences, or was there something about the rewards card itself that was contributing to the lower satisfaction levels?

To answer this question, we conducted an ANCOVA analysis which looked at whether there were significant differences between the rewards card and the basic card on five key measures - both before the ANCOVA analysis and after.

As can be seen in Table 1, prior to conducting the ANCOVA analysis, significant differences existed between the rewards card and the basic card on all five key measures. After the effects of age, ethnicity and income had been controlled by running the ANCOVA model, significant differences in the key satisfaction measures between the rewards card and the basic card still existed.

It was concluded from this analysis that the variation in the demographic composition of the two respondent groups was not driving the differences in the satisfaction ratings between the two cards. Rather, something else about the card itself, or the customer experience, must account for the differences.

In this particular case, once the influence of demographics was ruled out, and a thorough analysis of verbatim comments was completed, it was concluded that customer concerns over the financial stability of the bank’s particular cobrand partner were adversely impacting satisfaction ratings with the rewards credit card.

Significant advantages

While ANCOVA has been in the researcher toolbox for many years, it may not be the solution that immediately comes to mind when trying to sort out the issues presented in this article. In fact, many researchers might first think of using weighting to try to level the playing field between the two samples. However ANCOVA offers significant advantages over the use of weighting, including: the absence of arbitrary decisions on which sample is weighted, the ability to handle multiple differences in the samples at the same time (eliminating the need for complex, tiered weighting schemes), and, as a result, the model can be expanded to multiple degrees of complexity.

In short, the ANCOVA technique provides a telling way of describing true differences between respondent groups - controlling for compositional variation in the samples in a reliable manner.

Importantly, the ANCOVA technique can also be quite helpful in monadic research designs where the sample is split into various groups - with each group evaluating only one particular item (product concept, company positioning, etc.). This is done to reduce respondent fatigue. However, differences in the sample composition among the groups can obviously adversely impact the ability to make reliable conclusions. In this case, ANCOVA can be used to successfully sort out whether the differences in evaluations seen across the cells in a monadic design are real or a based on difference in the composition of the samples.

Â