William McLauchlan, Ph.D., is principal, McLauchlan & Associates, Cincinnati.
I would like to express my thanks to Quirk's for providing the opportunity to respond to Doug Grisaffe's remarks. His critique of my article needs to be addressed lest the reader be left with several inaccurate and potentially misleading conclusions.
While I will address the specifics of each of his key criticisms below, I would like to begin by stating that I believe Grisaffe has missed a major point in my article. That is, he seems to have read the piece as though I was in some way "blaming" multiple regression for problems associated with messy data sets. In fact, the focus of my article, as reflected by its title ("Regression-based satisfaction analyses: proceed with caution") did not "blame" regression. Instead, it pointed out a number of serious consequences that can occur when using multiple regression analyses with messy data and that caution should be exercised.
Further, I would suggest that messy data exists with much greater frequency than Grisaffe would lead one to believe. If all of Grisaffe's customer satisfaction data sets behave as well as the reader is lead to believe, then I truly believe that Grisaffe and his organization are to be commended. I suspect that such perfectly behaved data is the exception.
Have made this general remark, I would like to speak to each of Grisaffe's key points.
1. Hypothetical data
Of course my data set was contrived. It was chosen precisely for the purpose of illustrating suppressor effects and the inherent dangers of misinterpreting regression coefficients with the wrong sign. I do not suggest for a second that these kinds of outcomes are "problems" with regression. Clearly these are data problems.
While Grisaffe is quick to point out what he calls the "non-typical" conditions associated with my hypothetical data, he has neglected to point out the even more atypical relationships in his own example data (Table 2). Here is the correlation matrix Grisaffe failed to provide:
Attribute
1 2 3
Overall .53 .30 .71
Att. 1 .00 .00
Att. 2 .00
Are data sets with between attribute correlations of 0.0 more typical than data sets with the potential for suppressor effects? I suspect this is not the case.
2. Nonsignificant attributes with high stated importance
As before, I did not suggest that the situation where an attribute garners a high stated importance rating but is not a determinant attribute in a regression-based analysis was, as Grisaffe has attributed to me, a "problem with multiple regression." Again, it is a data issue.
Of course, my data points were chosen carefully. They were selected to illustrate the danger of using a "model" that is clearly based on a world-as-it-is perspective to make predictions about future overall customer satisfaction.
I certainly agree with Grisaffe when he states that most consumers would give high importance ratings to safety in airline choice. I also concur that, in general, stated importance ratings tend to overweight the importance of low probability events. However, while airline safety may have relatively little to do with why people choose an airline on any given day, I suspect it has a tremendous amount to do with airline choice following a disaster. (I also suspect that airline safety has some non-zero importance weight at all times.)
The larger point here is that regression-based analyses are status quo assessments of the reasons for satisfaction. And, while Grisaffe's airline example is clearly demonstrative of this world-as-it-is perspective, how does he ever know that airline safety would get high stated importance ratings unless he asks?
More fundamentally, how do we know what the "minimum requirements" are in any product category? Grisaffe seems to imply that it may be the attributes that are not significant predictors of overall satisfaction that are the minimum requirements in a product category. I am not comfortable with that assumption.
Finkbeiner (1992) stated that the "use of the (regression) model for predictive purposes must be kept in proper perspective...Limitation of the data to the world-as-it-is creates just as much restriction (if not more) for preference regression as it does for benefit/loss charts. Because of this limitation to the world-as-it-is, extrapolation of attribute performance beyond whatever is represented in the marketed products is not safe. For example, if no differences exist between companies on price, then price cannot account for any variance in overall satisfaction. If however, one company changes its price significantly from all the others, then price will certainly become an important variable...As a consequence of this dependence on the world-as-it-is, preference regression has a tendency to rediscover sameness in company performance and to recommend the known, safe solutions to performance improvement (p. 143)."
3. Stated importance versus derived importance
Grisaffe refers to my Sawtooth Software Users Conference paper (McLauchlan, 1992b). In this paper, I hypothesized that stated importance ratings might do a better job with feature-based categories and that derived analyses might do better with image driven categories. What I found empirically were stated and derived measures that, in fact, correlated quite well with each other in three product categories that span the gamut from predominately feature advertised (commodity chemical) to largely image advertised (beer).
4. Regression fits CSM theories and management goals
Grisaffe states, "CSM researchers are interested in finding ways to increase scores on the overall measures (p. 13, Feb. '93, QMRR)." In Joel Huber's discussion of my Sawtooth paper (Huber, 1992), he made the point that "correlational measures can be quite misleading in that they assume that the range of attributes will remain constant. Stated importance measures, by contrast, implicitly ask respondents to indicate how important an attribute would be if it does change. This line of reasoning suggests that correlational measures are appropriate when managerial action will preserve attribute expectations in the market, while stated importance ratings are more appropriate when they will alter them...Correlational measures may be good for generating ideas, but can produce very misleading managerial recommendations (p. 314)."
Huber goes on to recommend conjoint tasks as a way of dealing with the shortcomings of both stated importance and regression-based satisfaction analyses. I heartily concur.
Of course it is not surprising that no other linear model than multiple regression will best-fit Grisaffe's (or anyone else's) customer satisfaction data set. Regression, as applied here, is a least squares procedure in which the objective in deriving the coefficients is to minimize the residual sums of the squares. Having said this, the fact that no other linear "model" will better fit the data is not, in itself, grounds for applying the model. (The whole issue of linear versus non-linear relationships in customer satisfaction data will not be discussed here. I would, however, suggest that the reader not make the presumption of linearity lightly.)
Consider again Grisaffe's own hypothetical data set. Huber, like Finkbeiner, cautions that "derived importances, reflecting the correlation between preference and the attribute ratings, depend critically on the variability of the attribute in the market (p. 314)." Certainly derived importances also depend critically on the variability in overall satisfaction. If respondent 2 in Grisaffe's data phoned in sick during data collection, we would have no variability in the criterion and a model which cannot be specified at all. Demonstrating a best-fit relationship in a data set is not a sound basis for presuming a meaningful model. Recall my own hypothetical data set which, in spite of the suppressor effects, and in spite of Grisaffe's discounting of these effects, still yields a "best-fit" equation.
5. A note on regression and causality
Grisaffe speaks extensively about plausibility as an important basis for inferring causality in regression-based satisfaction analyses. I happily acknowledge that the implication of causation can be reasonable given an empirically demonstrated basis for that plausibility. And while his smoking example fits the category of having a plausible basis for inferring causation, customer satisfaction is not cigarette smoking and lung cancer.
If it is plausible to assume that a high correlation between overall satisfaction and satisfaction on an attribute such as "on-time delivery" resulted because a high degree of satisfaction with delivery performance "caused" a high level of overall satisfaction, is it not just as plausible that a high level of overall satisfaction "caused" "halo" effects in some attribute ratings, such as "on-time delivery"? I think it is.
As Kerlinger and Pedhazur (1973) stated, any student of elementary statistics knows that there can be a high correlation between two variables when in fact there is no direct or causal relationship between the variables. Further, a high correlation in one sample may be zero in another.
Huber states that "causality is equivocal in derived importances" and that "the point is not to resolve these two interpretations but to understand the problem with trying to tease them apart. Many hours have been spent with path models and causal modeling to try to parse causal paths, but the most common outcome from such careful analysis is an acknowledgment of how difficult it is to derive causal relationships from cross-sectional data (p. 314)."
Similarly, Kerlinger and Pedhazur write that "the reality is that... independent variables are usually correlated. Consequently, interpretation of regression analysis data is often complex, difficult, even misleading (p. 77)." Darlington (1968) indicated "it would be better to simply concede that the notion of 'independent contribution to variance' has no meaning when predictor variables are intercorrelated (p. 169)."
Grisaffe gets no disagreement from me when he advances the premise that improving the quality of "various components and processes of our business" should lead to higher overall satisfaction measures. However, this is not the point. The issue is the identification of those components likely to impact overall satisfaction to the greatest extent. Grisaffe argues for multiple regression as a tool for identifying those components. Based on the reasons advanced in my original article and supported by the arguments made here, I must continue to disagree.
6. Good regressions from good designs/should complexity prevent use?
I do not "assign risks" to regression as a statistical technique. I do not expect regression to compensate for anything including "messy" data. Neither do I advocate "statistical fishing" using stepwise models. Nor does the complexity of regression lead me to recommend a simpler approach. My article simply points out several important and real-world issues that can impact a regression-based analysis and its interpretation.
Of course, sound designs are critical in any research effort. Grisaffe's expectation of sound research design in customer satisfaction appears to be synonymous with regression models which use the "right" items that are "already known to matter," that exhibit no multicollinearity and that result in models with high R-squared values. I am not convinced that we always know enough about the issues and variables to assure these kinds of outcomes.
I maintain that regression-based satisfaction analyses are not as straightforward as might appear at first glance. It is my hope that the issues raised in my original article and the points made by Grisaffe in his rejoinder will continue to foster meaningful debate. All of our customers will benefit.
References
Darlington, R.B. (1968), "Multiple regression in psychological research and practice," Psychological Bulletin, 27, 229-245.
Finkbeiner, Carl. (1992), "Alternative applications of preference models to customer satisfaction research," in Sawtooth Software Conference Proceedings, Ketchum, Idaho: Sawtooth Software, 127-159.
Huber, Joel. (1992), "Comment on McLauchlan," in Sawtooth Software Conference Proceedings, Ketchum, Idaho: Sawtooth Software, 313-315.
Kerlinger, Fred N. and Pedhazur, Elazar J. (1973), "Multiple regression in behavioral research," New York: Holt, Rinehart and Winston, Inc.
McLauchlan, William G. (1992a), "Regression-based satisfaction analyses: proceed with caution," Quirk's Marketing Research Review, October, 1992, 10-13.
McLauchlan, William G. (1992b), "The predictive validity of derived versus stated importance," in Sawtooth Software Conference Proceedings, Ketchum, Idaho: Sawtooth Software, 285-311.