Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Data Use: An analysis of the impact of survey scales



Article ID:
20131105
Published:
November 2013, page 22
Author:
Adam S. Cook

Article Abstract

Adam Cook examines many options for survey scales and offers some research-on-research that explores the effects of various scale point ranges.

Editor's note: Adam Cook is director of research and development at Pilot Media Inc., Norfolk, Va.

Survey scales are important because they help differentiate the degree to which people feel toward certain questions. Yes-or-no responses are not always an option in consumer perceptions and feelings. But what’s the best numeric scale to use in analyses and what scale is easiest for the respondent to interpret?

If it’s a paper or phone survey, sophisticated and/or easily-misinterpreted scale questions can be a real challenge. A scale can’t have too many radial points (e.g., you don’t want to ask on a scale of 0 to 100, where 0 = x and 100 = y) or too many descriptors defining each point (e.g., completely, somewhat, rarely, never, neutral, etc.). If we don’t include a number of options, our ability to analyze differentiation in responses becomes more limiting (e.g., a 1-to-3 scale doesn’t give us a whole lot of information to differentiate between responses). See Figure 1 for varying examples of scale questions.

Thankfully, interactive online surveys exist and they have some real untapped potential for finding the sweet spot between maximizing participation and analytic reliability or differentiation. Given the challenges presented in traditional collection methods, I’m going to focus on the ideal for interactive online scales.

Even vs. odd number of scale collection points. The options are many: 1-to-10 (10 options) or 0-to-10 (11 options); 0-to-7 (eight options) or 1-to-7 (seven options); 0-to-5 (six options) or 1-to-5 (five options). I’ve usually heard the value of even-numbered scales is that they require respondents to choose or lean toward one of the extremes on the scale presented. I understand the desire to acquire definitive feelings but the truth is that some people have no feeling, one way or another, toward certain things (an inconvenient reality to some decision makers). I believe eliminating the neutral option interjects bias into the results and ultimately the analysis. I feel it’s “extremely important” to use odd-numbered scales, ones that have the option to select a true mid-point of neutrality. But sometimes you have to deal with the scale that may be given to you to analyze. We’re not always in a position to choose.

Number of scale points: three, five, seven, nine, 11, 13 . . . ? Well, we know one scale point isn’t an option and I’m ruling out even-numbered scales. Here’s what I do know (from the book Marketing Research: Methodological Foundations): “Research indicates a positive relationship between the number of scale points and reliability.”

Having a large number of scale points is important for analyses. If you’re using a radial point scale collection method, having a scale exceeding 0 to 10 or 1 to 11 (11 points) can look overwhelming. With a radial scale point display, I would recommend not exceeding 11. It’s important to note: If you are going to display the numbers on the scale, your maximum scale should probably be 0 to 10. Many books have been written on the importance of this scale and its ease of translation to potential respondents. Zero is typically defined as bad and 10 is usually associated with the highest of marks. A scale of 1 to 11 wouldn’t work because 11 is not commonly associated with perfection or rankings. Sometimes 1 is considered the best as well without a clear definition (very similar to the defining an ace as high or low in card games). This lack of a clear association may create confusion.

Here’s where an interactive scale can help us overcome participation and visual fatigue. The use of sliding-scale displays enables us to remove the need to display numbers (see options five and six in Figure 1 for visual examples of sliding scales). With the numbers coded into the background, you need not worry about confusion or overwhelming radial point displays. In fact, the scale coded into the background is ultimately up to the analyst developing the interactive survey. The sliding scale can even get us a scale greater than 11 points to help maximize reliability. Technically we would be limitless, but 0 to 1 billion sounds like a bit much. If you don’t like the idea of 101 points spanning 0 to 100, simply create 101 points spanning 0 to 10. It’s simply moving the decimal point to the tenths (e.g., 0, 0.1, 0.2, 0.3 … all the way up to 9.8, 9.9, 10). I have yet to see this offered as a scale option but I’d love to have this capability. This brings us to our last quandary of where to start our scale.

Starting a scale with a 1 or a 0. If you’re displaying numbers, it’s actually pretty arbitrary in terms of what you use as long as the numbers are clearly defined. It’s ultimately the number of scale points that dictates the strength of your analysis. A 1-to-10 scale is essentially the same as a 0-to-9 scale or, as crazy as it may sound, a 2-to-11 scale. I would hope it’s self evident that if you’re using a low-end descriptor of “not at all, none, never” or anything that’s a definitive null, then 0 is the best number to start with on the scale. I really don’t have a case for using 1 and I’m not completely sure why scales do start with a 1 for display or analysis purposes. Until I hear a solid case or rationale for using 1 to start the scale, I’m going to stick to 0 when given the option. Data collection with 0-to-10 also has the easiest conversion to percentage analyses.

Obviously the choice and preference in scales used is ultimately yours but it is worth considering the implications of your choice. Figure 2 represents my preferences.

Scale analysis options and pitfalls

Why median scores are a bad idea. Medians are good for analyses that incorporate extreme outliers in data. Household income is probably the best example of when to use a median in analysis. One billionaire can make an average income analysis skyrocket. Since we’re analyzing a scale, there’s a distinct and established range. The percent differences can be significant from one data set using averages in analysis versus another using medians. See Figure 3 for a random example of 100 respondents analyzed using medians versus averages.

The average analysis was 19 percent higher than the median analysis in 2011 and 11 percent lower in 2012. Conversely, when you analyze the change from 2011 to 2012, we see a 12 percent increase in average scoring, while the median analysis shows a 50 percent increase. If this isn’t enough to put the nail in coffin of median analyses on scale questions, I don’t know what is.

Why rounding averages is another bad idea. In Figure 4’s example of two different average scores, where Group A = 4 and Group B = 5, Group B’s average score is 25 percent higher than Group A’s. What if Group A’s average was actually 4.49 and Group B’s was 4.51? The difference would be minimal. What if Group A’s average was actually 3.50 and Group B’s was 5.49? Group B’s average score would be 57 percent higher. The range in difference is anywhere from 0 percent to 57 percent. This would indicate to me that’s it’s a bad idea to round results in analyses. See Figure 4 for examples of the impact in rounding averages.

Why percentage groupings can actually misinterpret results. I’ve seen reports and analyses that group numbers together from a scale question. See Figure 5 for an example of grouping 7s or higher on a scale of 0 to 10. All of the examples used in Figure 5 have 25 percent scoring a 7 or higher.

It didn’t occur to me until recently how inaccurate these groupings can be in analysis. When you start creating a number of different scenarios, some random and some extreme, the variations are a wake-up call. In Figure 5, Example 3 Minimum and Example 4 Maximum share the same 25 percent scoring a 7 or higher, but Example 4’s average score is 19 percent higher than Example 3’s.

At its most extreme in scoring, shown as “Extreme (-)” or “Extreme (+),” the maximum can be 300 percent higher than the minimum. See Figure 6 for additional analysis comparisons in averages (non-rounded) versus the grouping method for 50 percent and 75 percent scoring 7 or higher on a scale of 0 to 10.

Converting the scale analysis into percentage representation. Scale analysis can be converted into a percentage analysis (as already seen in Figures 5 and 6). If you’re using a 0-to-10 scale, simply moving the decimal point converts your average scores into percentage representation. At its lowest, the entire sample giving 0s equates to 0 percent and at its highest, the entire sample giving 10s equates to a 100 percent. If you’re using a scale other than 0-to-10, you’ll need to use a less-obvious conversion formula. The conversion formula for varying scales other than 0-to-10 can be found in Figure 7. For an example of scale impact and conversion on a scale of 1-to-10, see Figure 8.

Not all analyses are equal. In fact, some can be downright deceptive. My advice: When you can, use 0-to-10 scales, conduct average (non-rounded) analyses and convert to percentage analyses when needed.

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 501: FACILITATION - PRACTICAL TOOLS, TIPS AND TECHNIQUES
August 11-13, 2014
RIVA Training Institute will hold a course, themed 'Facilitation - Practical Tools, Tips, and Techniques,' on August 11-13 in Rockville, Md.
BURKE DESIGNING EFFECTIVE QUESTIONNAIRES: A STEP BY STEP WORKSHOP
August 12-14, 2014
Burke Institute will hold a workshop focused on designing effective questionnaires on August 12-14 in Las Vegas.

View more Related Events...

Related Articles

There are 2054 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

A report on trends in communicating more effectively with 50-plus consumers
People in marketing and advertising need to drastically alter their perceptions of 50-plus consumers, as Baby Boomers turn 50. There will be a need to look more closely at the social and economic changes that characterize today’s mature market, and advertisers will need to change their strategies accordingly. The mature market is growing in number and affluence. Also mentioned are specific actions the travel industry and certain companies are taking in regard to the mature market’s shift.
Trade Talk: Hallmark sends greetings from Social Media Land
This article contains a discussion of Tom Brailsford's presentation on social media data at The Market Research Event in November 2011.
Data Use: Segmentation models and the overfitting problem
Predictive segmentation modeling techniques vary in regard to the size of the sample needed to achieve a given level of reliability. This article discusses model building that captures innate systematic processes rather than discrete, temporal events, or those that are sample-specific or peculiar to a given sample.
Data Use: Getting the most from your paired preference testing
Blinded product—paired—preference testing is a common marketing research technique. Using an example, this article addresses how to best conduct paired product preference testing to avoid ambiguous results, which includes repeating the pair testing with respondents and conducting a binomial test statistically.
How improvement-gap analysis can produce better satisfaction surveys
Beyond measuring a customer’s satisfaction, it’s perhaps even more critical to measure the importance of factors that contribute to their satisfaction. The author offers advice and examples of how to do so.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer
TURF excel-based simulator
03/25/2014 by Werner Mueller
Question writing in which person--I or You?
03/17/2014 by Shalan Gilmeister

View More