Customer satisfaction and company dissatisfaction

Editor’s note: Elizabeth Bergman is an adjunct professor at California State Polytechnic University, Pomona.

Collecting customer satisfaction data has become ubiquitous across the corporate world. The data collection effort is so pervasive that conflicts can arise when departments in companies have differing satisfaction results. I was on the client side for many years and recently saw the negative effects of such conflicts: “interpretation” turf wars, endangered research budgets, and ultimately discrediting of customer data collection efforts.

The purpose of this article is to provide guidance to corporate managers trying to understand how to use and reconcile multiple and different surveys conducted across the organization. In other words, can survey results from different instruments be viewed in tandem, or must they be understood separately, and if so, how, why and to what effect?

The company in this case study is a leading software manufacturer where I worked for some time. This company has a number of different surveys touching customers at different points in the product life cycle. Customers are surveyed after a product is shipped, when a problem is fixed, after a consulting engagement, and annually regarding their overall experience with the company. In and of themselves, each of these surveys provides a useful bit of information. At almost any point in the company’s product life cycle management can tell how the company performed for the customer. However, managers frequently engaged in “data posturing” to insulate themselves from criticism about their own scores. Common questions were: “Why are your survey’s scores of me lower than my survey’s scores?” and “How do we correlate the two surveys?”

The answer is that each of these surveys had been crafted to present specific information, and should likewise be used for only those purposes, otherwise methodological problems such as validity and reliability can allow management to draw inaccurate conclusions.

Validity, reliability and time

First, we tackle the issue of validity. In the case study company, two surveys were at issue, one originated by the customer support organization (CSO), a high-volume call center; the other by the customer satisfaction and corporate planning department (CSCP). Both measure customer satisfaction, albeit for different purposes. The CSO survey is a trailer survey and as such is designed to explore the customer’s experience with a transaction or at the end of an engagement. The CSCP survey is a relationship survey designed to cover a broad range of questions, painting a picture about the customer’s whole experience with the company. Relationship surveys have a battery of questions created to understand what drives customer loyalty and customer retention.

One can imagine how problematic it would be to use a transaction survey to predict loyalty, and likewise how difficult it would be to ascertain transaction effectiveness from a survey about a customer’s overall relationship with a company. Information derived for one purpose would be used for another purpose; this is a problem of validity. For example, a measure of a subject’s psychological reaction to a package (e.g., pupil dilation) does not constitute a valid measure of purchase intention. “By validity we mean the extent to which our measurements reflect what we intend them to, or what we claim they do.” (Kachigan 1986;219, italics in the original).

Next, the issue of reliability. Reliability applies to a measure when similar results are obtained over time and across situations. Two dimensions underlie the concept of reliability: repeatability and internal consistency. Repeatability refers to the test-retest method of administering the same scale to the same respondents at two separate times to test for stability. This is critical and was clearly not the case with the CSO and CSCP survey instruments and methodology; the scales were different and the respondents were different. Internal consistency refers to the homogeneity of a measure, which is the idea that measuring an attitude may require asking several similar questions. While this was also not the case across the two survey instruments, it is the lesser of the methodological problems and not the focus here.

For our purposes, the question about how to correlate the two surveys raises the issue of repeatability. “The correlation between successive measurements on a set of objects with respect to a variable is referred to as test-retest reliability.” (Kachigan 1986)

According to statisticians, an unreliable set of scores - a set of scores that do not correlate with themselves upon re-measurement - is essentially equivalent to assigning the scores to the objects in a purely random manner. “How could a randomly assigned set of scores possibly correlate with another set of randomly assigned scores? It would be like trying to correlate two columns of random numbers.” (Kachigan 1986)

While the case study company did not, separately, have unreliable scores in the CSO and CSCP surveys, it did create the situation of unreliability when trying to merge unlike scores. Attempting to correlate unreliable variables will produce exactly the same result, a zero correlation - “A chain is no stronger than its weakest link. So just as a measurement system can have no validity if it has no reliability, the degree of validity is limited by the degree of reliability.” (Kachigan 1986)

Finally, there is the issue of temporal variation, that is, surveys that are done at different times of the year and at different points in the customer’s interaction with the company. “In many cases, such as equipment and other durable products, customers use products over a relatively long life cycle (usually in years) made up of a sequence of distinct phases. These products may require the investment of significant resources by the customers and the longer the life of the product the greater the potential for customer satisfaction to change (usually decrease) over time.” (Ramos 1996) Furthermore, time lapse influences ability to properly remember and communicate specific factors, and if the time between measures is long, there may be attitude change or other maturation of the subjects. Thus, a transaction survey done immediately after the relevant event will likely have higher scores than relationship surveys done later in the product life cycle.

However, for present purposes to illustrate a point, we will suspend belief and ignore the earlier question of whether a randomly assigned set of scores can correlate with another set of randomly assigned scores. We in fact force the correlation of two columns of arguably random numbers. Data from the 2002 CSO survey and corporate survey were merged and analyzed. The results indicate that there is no significant relationship between the corporate measure of CSO overall satisfaction and CSO’s own measure of overall performance. The correlation coefficient of .103 is not significant at a p value of .111.

Next, a bivariate regression was run and plots were created to further test the relationship between CSO customer satisfaction and the corporate measure of CSO satisfaction. In other words, does one drive the other? Is one the predictor of the other? CSO customer satisfaction is modeled as being dependent on the corporate CSO measure. Results indicate that about 1 percent of the variation (r-squared of .011) in CSO’s scores can be explained by the corporate measure. This is a very weak result. Furthermore, the F-statistic is very low at 2.55 and not significant, indicating that this is not a very good model for explaining CSO satisfaction. Finally, regression coefficients and P-plots indicate low correlation between the predictor and criterion variable; indeed the assumption of a normal distribution and linearity is questionable. Therefore, it would not be correct to say that an increase in corporate satisfaction will result in an increase in CSO satisfaction or vice versa.

(Note: similar results were obtained when using the corporate customer satisfaction “overall performance” variable instead of the corporate CSO satisfaction variable. Additionally, the CSO variable was used as the predictor of both corporate overall and CSO satisfaction to no different effect. All permutations of the equation were insignificant.)

Therefore, for reasons of validity, reliability and temporal variation, comparing customer surveys across functional areas within the company should be done with a cautious eye and only to illuminate directional trends in customer data. Causality and correlation across survey instruments and output with different scales, respondents and time periods is suspect.

Scale issues

But what about the other question: “Why are your survey’s scores of me lower than my survey’s scores?” This too may be a function of misunderstood methodology - or trying to compare apples to oranges. However, before tackling the negative aspects of fruit comparison, let’s examine the positive possibilities of data comparison.

As mentioned in the last section, comparing customer surveys across functional areas within the company can be done to illuminate directional trends in customer data. This view is shown in Figure 1. The CSO survey and the CSCP survey (referred to as “corporate”) question counterparts are plotted for one quarter in 2002. While the scores are slightly different for each question, the trend between the two surveys clearly follows a similar pattern. Scores that are high for the CSO survey are also similarly high for the corporate survey; likewise, dropping CSO scores are mirrored by dropping corporate scores. This is true for all questions except 31H on e-support effectiveness, which shows a virtually identical score in the two surveys.

Figure 1

While the scores may be slightly different on similar questions in the CSO and corporate surveys, the similar data pattern between the two surveys tells us that what customers rate as important appears to be mapping across customer groups exposed to the company. Customers want to reach the company quickly and have a quality resolution to their problem. The time it takes to get to that resolution is somewhat less important than providing them with quality service.

Now let’s return to the issue of score comparison and the differences between the CSO and corporate surveys. First, it should be noted that the scales for the two surveys are different. CSO is using a seven-point scale and corporate is using a five-point scale. While two points does not seem like much, in actuality the scale differential accounts for 10 to 15 percent of the “percent satisfied” score discrepancy and almost 17 points because of the normalization process used by the company in creating score indices. This is possible because of the different point values associated with each increment value in the inflated normalized 0-100-point scale. Inflating the seven-point scale causes each increment to equal 16.66 points; in the five-point scale each increment equals 25 points.

In the corporate survey more than 60 percent of customers rate the company a “4” or “very good.” The normalized score index (NSI) conversion for a respondent selecting 4 on the corporate survey is 75. Similarly, for respondents desiring to rate CSO “next to highest” or a 6, the NSI conversion is 83.3. Because of the index inflation factor the two-point scale differential turned into an eight-point difference between the two respondents giving the company the same rating. When this effect is aggregated it can be exacerbated.

Let’s look at what happens when we take 10 respondents who use the 3 and 4 or “good” and “very good” on the corporate survey, and 5 and 6 on the CSO survey. The CSO survey scale uses “very important” for the 7 rating only. One can generally assume that anything above a 4 in a seven-point scale is a “good” rating. Thus, we can assume that the 10 respondents in this example desired to rate their experience as “good” or better. As we can see, 10 individuals wishing to designate an organization or department as good or better will rate CSO a 74.97 and corporate a 62.50 in the aggregate — a 12.47-point difference.

CSO    

Corporate   

Select
5 & 6 rating

Select
3 & 4 rating

66.64

50

83.3

75

66.64

50

83.3

75

66.64

50

83.3

75

66.64

50

83.3

75

66.64

50

83.3

75

74.97

62.5

As Myers (1999;192) notes, “constructing an index that is accurate, fair, and sensitive is more difficult than it might seem. Management has many options in designing an index, but some options can lead to consequences that were not intended.” That is the crux of the issue at the case study company and it suggests that research managers would do well to remember and reinforce the intent and purpose of company survey(s). In CSO the objective was to monitor help-desk performance and pay bonuses; in CSCP the objective was to ascertain customer loyalty and create programs to increase customer retention. The latter sought to reveal a few key drivers from a myriad of attributes; the former sought score movement. Thus, company management was in the crosshairs about irreconcilable objectives as much as score differences.

It is worth noting that “moving the needle” is a common problem in customer satisfaction measurement programs. It occurs because most programs are based on repeated surveys of samples of customers, usually at frequent intervals, and any changes are found to be minimal. The difficulty, in part, is due to rating scale insensitivity. This causes both management and employee frustration and can serve to undermine compensation plans based on insignificant or immovable customer satisfaction targets. While there is no simple solution to this problem, there are ways to minimize it. One way is to increase scale sensitivity by expanding or lengthening the range; from a five-point to a seven-point or from a seven-point to a 10-point. Short scales are typically not sensitive enough for use in repetitive customer satisfaction surveys due to the “satisfied customer” phenomenon — present customers, by virtue of that status, tend to be reasonably satisfied or they would not be customers.

How should it be viewed?

How then should the CSO customer support survey data and the corporate customer satisfaction survey data used in this example be viewed by management - in tandem or separately?

The short answer to this question is that all surveys done by the company must be viewed as output from the instruments for which they were designed. Survey results do not mix; they provide unique, application-specific snapshots, if you will. The CSO and corporate surveys provide two views of the company. The CSO survey is a trailer survey and is a transactional, functional, process measurement offering indicators and guidance in efficient unit operation. The corporate survey is a relationship survey providing a corporate-wide view of all business units and most functional areas within the company. Research managers must understand and reinforce survey objectives - an effort requiring ongoing information dissemination and education.

This perspective on, and utilization of, the customer satisfaction data will afford companies a much more productive management tool than trying to engage in head-to-head comparisons of output across functional areas. Each survey that is done offers information on another piece of the customer’s experience. The pieces are not mutually exclusive but rather paint a picture of the whole product life cycle experienced by the customer. Different scores from different instruments do not produce irreconcilable results; one is not right and the other wrong. Each tells a part of the customer’s story.

References

Kachigan, Sam. 1986. Statistical Analysis. New York: Radius Press.

Meyers, James. 1999. Measuring Customer Satisfaction: Hot Buttons and Other Measurement Issues. Chicago: AMA Press.

Ramos, Maria. 1996. “Key measurement programs for a customer satisfaction system in a business-to-business market.” Quirk’s Marketing Research Review, April 1996. (access online at www.quirks.com by entering QuickLink #0007).