The Center for Strategy Research, Inc. Vol 5 Issue 3   April 2009


Hello!

Although CSR is best known for its “qualitative-into-quantitative” methodology, we are frequently retained to develop and manage traditional, large-scale, quantitative surveys. With that in mind, this month’s edition of Research with a Twist takes a look at the use of measurement scales in quantitative studies.


Julie Brown
President

Mark Palmerino
Executive Vice President



Doing Justice to Scales
I’m a big fan of crime show dramas. It doesn’t matter what the medium is either — movies, television, DVD, you name it. If there is a “who dunnit” to solve, I can’t stay away.

Having watched hundreds (thousands?) of episodes at this point, it occurred to me the other day that no matter how complicated the case at hand, at the end of the hour, it all comes down to “guilty” or “not guilty.” There may be all kinds of twists and circumstances and nuances involved along the way, but in the end, the verdict is one or the other.

I also got to thinking about how guilty / not guilty is a “scale.” It’s a simple one (see “Nominal Scales” below), but nevertheless, it serves as a means of categorizing outcomes.

In research (and life), things are rarely tied up so neatly. In practice, we rely on a variety of different scales as tools for capturing and classifying the broad and complex range of research participant preferences, decisions, ideas, attitudes and opinions.

And so with that in mind, this month’s newsletter takes a look at four of the most commonly used — and all too frequently, misused — scales.

(Quick aside: These four levels of generally accepted measurement scales were originally defined by the psychologist Stanley Smith Stevens. And while they have become an accepted part of today’s research methodology, there is some debate over whether these are the only, the best, or the most appropriate classifications.)

The four measurement scales:


1. Nominal scales

Sometimes called a categorical scale, nominal scales classify entities into categories where no order is implied. These types of scales simply count the frequency of cases assigned to various categories.

Here’s an example of a nominal scale:

Which of the following cocktails do you drink the most? (Please select one)
  Martini   Cosmopolitan   Zombie
  Gimlet   Manhattan   Margarita


With a nominal scale, the only measure of central tendency that can be applied is the mode — the value that occurs most often.

Nominal scales are best when trying to assess a universe. For example, if you were conducting research on the items people shop for, the brands they recognize, or, as in the example above, the cocktails they drink.


2. Ordinal scales

Ordinal scales involve the ordering of items along a continuum of the characteristic being scaled. A good example of an ordinal scale is what results when we ask people to rank things in order of preference. So, for example, we might ask a bar manager to rank five gins in terms of how well they taste in a Martini:

Order of preference Brand
1. Hendricks
2. Tanqueray Rangpur
3. Tanqueray 10
4. Plymouth
5. Beefeater


Notice that with an ordinal scale such as this, while we learn the bartender’s order of preference, we don’t have any information regarding how much more one brand is preferred to another. It’s a simple ranking, with no indication of interval distance between the choices.

One of the mistakes frequently made with ordinal scales is an attempt to deduce an interval from the data. If, for example, twice as many survey participants chose Hendricks as Beefeater, it would be incorrect to say that Hendricks is “liked twice as much.” All we know from this data is that twice as many people selected it.

Ordinal scales are best suited for comparisons or ranking of many products or options.


3. Interval scales

Interval scales (also known as cardinal scales), have equal units of measurement. They are very popular in quantitative research because using them allows us to interpret not only the order of scores but also the distance between them. The equal units of measurement of an interval scale allow us to apply statistical analysis, calculating things such as the average, standard deviation, etc.

An example of an interval scale:

Please indicate your experience with Hendrick’s Gin by scoring how well it performs in the following characteristics (with 5 meaning very well and 1 meaning not well at all)

Hendrick’s Gin is (circle the appropriate score on each line):
Smooth 1 2 3 4 5
Citrus-flavored 1 2 3 4 5
A good value 1 2 3 4 5
Attractively packaged 1 2 3 4 5
Juniper-flavored 1 2 3 4 5


Keep in mind that with interval scales, the zero point (if there even is one) is arbitrary, a fact that has implications for data manipulation. In measuring temperature, for example, while the degrees are evenly spaced (interval scale), it is inaccurate to say that 50 degrees Fahrenheit is “twice as warm” as 25 degrees Fahrenheit. It’s 25 degrees more, but not twice as warm (despite what your local weatherman may tell you!).

(Quick aside #2: Despite common usage in research, the example above is not quite a “true” interval scale, since the distance between the numbers — 1, 2, 3, 4 and 5 — is not necessarily uniform. But in practice, and for the purposes of data manipulation, many researchers treat it as such.)


4. Ratio scales

The highest level of measurement is a ratio scale, a measurement that offers the benefits of an interval scale together with a fixed origin or zero point. Times, lengths and dollar expenditures are all good examples of variables that are ratio scaled.

Ratio scales permit the researcher to compare both differences in scores and the relative magnitude of scores. For instance, the difference between 10 and 15 dollars is the same as that between 20 and 25 dollars. And 20 dollars is twice as much as 10 dollars.

Other than market sizing and purchase pattern research, in the research studies that most of us conduct, we do not employ ratio scales. This is because things like attitudes, beliefs, preferences and the like, don’t have a real “zero point.” So in these cases, we tend to use interval scales for the highest analytic insight available.

Here’s the Twist: Crime shows nearly always employ the simplest of nominal scales: “guilty” or “not guilty.” That’s fine for TV, and anything more complicated would be both unsatisfying and too long for a 60-minute episode.

As researchers, however, we can’t let our desire to simplify complex decisions lead us to be “guilty” of constructing scales from which we draw and share inaccurate or inappropriate conclusions.

— Mark

Click here to share this newsletter with a colleague.

One question that our clients often ask is “What’s the best number of points to use on a scale — three, five, seven…more?”

As with many research-related questions, it depends. Some insights from others in the field:

  • Three-item scale: “Using only three answer options takes up a lot less visual space and is far less daunting than five options. Thus, we always try to limit the response options to three, as that number offers the robustness to capture what we need while still remaining visually inviting.” — Ziggy Zubric, “Less Really is More When it Comes to Response Scales
  •  
  • Five-point scale & seven-point scale: “To explore the relation between scale length and reliability, we conducted a meta-analysis of the results of many past studies… In general, we found that five- or seven-point scales produced the most reliable results. Bipolar scales performed best with seven points, whereas unipolar scales performed best with five.” — Jon Krosnick, professor of communication at Stanford, “The Optimal Length of Rating Scales to Maximize Reliability and Validity
  •  
  • Ten-point scale: “A five-point scale is totally inappropriate for customer satisfaction studies. Why? It lacks enough granularity and robs companies of a burning desire to take corrective action. It commonly leads executives to believe that ‘80% rate us four or five; that’s great, let’s move on,’ without realizing that it simply means that 80% are at least somewhat satisfied. Further, many people will never rate anything a ‘five,’ resulting in ‘four’ including those who are really very satisfied and those who are only somewhat satisfied. To avoid this topping effect, use at least a 10-point scale and count nine and 10 ratings as fully satisfied. This will also allow easier analysis of what bottom-line effects satisfaction has, since such tools as regressions work better with a more granular score.” — Brad Bortner, “Best Practices: Why Customer Satisfaction Studies Fail
  •  
  • Eleven-point scale: “The 0-to-10 scale has many significant advantages: Customers find that the scale makes intuitive sense…, most of the world already uses the metric system…, customers may refuse to give anybody a perfect score…, customers will transpose the top and bottom on a 1-to-10 scale…, scales with fewer points seem more susceptible to grade inflation…, the 0-to-10 standard is being adopted by many of the world’s leading companies.” — Fred Reichheld, Fellow with Bain & Company, The Ultimate Question, p. 98–99 (regarding the Net Promotor Score).
All good points (kind of makes you wish for “guilty” and “not guilty,” doesn’t it?). We often point out that the best answer is study-dependent, and will vary based on such issues as the overall topic and objectives, the number of scales being presented, the complexity of issues being researched, and the complexity of analysis required at the end.

 

Doing Justice to Scales

Mixology (Putting research into practice)

Twist and Shout

About Us


We’re delighted to announce that Mark will be speaking at The Market Research Event this year.

This event, which is scheduled for October 19th through 21st in Las Vegas, brings together hundreds of Market Research leaders and strategists and is one of our favorites.

We’ll be sharing more about Mark’s topic and exact timing as we get closer to the event itself, but wanted all Research With A Twist readers to know – so you can make plans to join us!



When they call the roll in the Senate, the Senators do not know whether to answer “Present” or “Not Guilty.”


— Theodore Roosevelt



Problems? Click here to send us an email with your request.
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result, our clients gain more actionable and valuable insights from their research efforts.

 
Understanding What People Really Think


WordPress Lightbox Plugin