The Center for Strategy Research, Inc. Vol 2 Issue 7   July 2006


Hello!

Some of you may recall that in our January newsletter, we asked for topics that you’d like us to address in this newsletter. We were bemused by the number of you who asked us to review “basic stats.” (As one of you said, and you shall remain nameless: “But please don’t make it too complicated!”)

With the dog days of summer now upon us, we thought this might be a good time for, “Dr. Palmerino’s (not too complicated) Summer Course in Sadistics.” So, pull up a chair, pour yourself a tall one and dig in!”

As always, your thoughts and comments are appreciated.


Julie Brown
President

Mark Palmerino
Executive Vice President



Summer Sadistics Part One: Sample Size…
How Much is Enough?

We got a call last week from a client; a VP of a multinational firm. Back in 2004, we had fielded a large research project for her company, in an attempt to understand how certain internal changes had affected the company’s 10,000 employees. In completing this work, we spoke with a sample of 300 people.

Knowing this, the VP made the following request: “This year, I’d like to conduct the same research again, but only in my division. I’ve got just 110 people, so I’m assuming we can get by with interviewing about 15 or so?”

Unfortunately, our answer was, “Hmmm… do you have a few minutes to chat?”

Surveying 15 people out of a population of 110 — even though representing a much larger percentage of the total group than does surveying 300 out of 10,000 — is not nearly enough to achieve the same level of accuracy and reliability as in the first study. (Believe it or not, it would take 80 out of the 110.)

And that, in a nutshell, is one reason why statistics (the scientific discipline upon which market research is built) can be so frustrating to the average business person. Much of it is counterintuitive.

This particular example — the relationship between sample size and population — is a frequent cause of confusion. With that in mind, today’s newsletter attempts to shed some light on this tricky topic.

First, a bit of background…

Fundamentally, when we are conducting market research, we are “estimating.” We may be searching for a “truth” that already exists (e.g. What percent of 18–24 year old males prefer Coke over Pepsi?), or forecasting an event that has not yet happened (e.g. Who will win the next presidential
election?). Whatever the particular question of the day, in most cases, we analyze a subset of the whole in an attempt to extrapolate this analysis to the greater population.

To better understand the question of sample size, let’s consider a simple example… baseball.

Suppose a particular baseball player were up to bat one time and got one hit. That player’s average would be 1.000 (“batting a thousand”). Now, suppose that same player gets up to bat again, but this time strikes out. Now, his average drops in half and he’s only batting .500 (1/2).

But what if that same player instead got a hit in his first 29 times up to bat? At that point, he’d still be batting 1.000 (29/29). But now, if he strikes out on his next at bat, his average only drops to .967 (29/30)!

The critical difference of course, is times at bat. In the early going, each at bat has a tremendous influence on a player’s average. With each successive at bat however, and even if the results are extremely different from what occurred before, the impact on the total is less.

And that, while admittedly an oversimplification, is exactly what’s going on when conducting market research:

  • The first few surveys may exhibit tremendous variation, and trying to make generalizations for the group as a whole based on a handful of results is risky (just as watching one baseball player bat twice doesn’t necessarily tell you how good he is… unless he’s Red Sox star David Ortiz, and then even once is enough!). When conducting any type of market research therefore, you need to speak with a minimum number of people, simply to reduce the amount of variation which exists with very small numbers. (More on what constitutes “minimum” below.)
  • On the other end of the continuum, and again, keeping our baseball example in mind, once you’ve asked the same question of 30 (or so) randomly selected people from a given population, you’re pretty close to the “real” answer.

    Sure, that 30 number can go way up depending upon a number of factors (which, incidentally, will be the subject of our next newsletter), but for a simple, yes/no type of question, once you’ve gone past 30, there just isn’t that much variation left. And while surveying additional people at that point will improve the certainty of your results, it’s unlikely (again, provided you’ve selected participants randomly) that the next 30 people are going to respond in a decidedly different way than the first 30.

But what about our opening question? What is the relationship between required sample size and population? As you may have guessed by now, there isn’t one (or at least not a strong one). As counterintuitive as it may seem, the sample size needed for a particular level of reliability and certainty on a given survey is largely independent of population size. In fact, once you get over a population of a couple of hundred people, the formulas that we statisticians use in determining sample size don’t even contain a variable for population!

In practice, the magic minimum number is about 30 (I could give you a detailed mathematical explanation of why it’s 30 rather than 20 or 40, but you probably just want to take my word for it). And while there are many factors which can make this number go up significantly (e.g. number of subgroups involved, level of accuracy needed, etc.)*, for a given group, on a single question, as long as you (correctly and randomly) survey 30 or more people, you’ll get quite close to accurately estimating the underlying “truth” that’s out there.

(*Join us next month when we examine these factors in more detail.)

— Mark

Click here to share this newsletter with a colleague.

Mixology (Putting research into practice)

It’s counterintuitive, but it’s true: Once a population reaches a certain size, it ceases to have a significant impact on the precision and reliability of a market research study.

If you want to see for yourself, take a look at the chart below. The rows represent a range of Sample Sizes; the columns represent a range of Population Sizes. Each of the blue cells in the middle shows the error rate for a given combination of sample and population size.

Notice that for a given Sample Size, if we move from left to right — from a Population of 100 Million to 1 Million to 10,000 — there is virtually no impact on the error rate. In other words, while increasing the Sample Size reduces the error rate somewhat, for any given Sample Size, the size of the population — whether ten thousand or 100 million — is inconsequential.

Size of Population
100 Million 1 Million 10,000
Sample Size Error Error Error
1200 2.83 2.81 2.65
600 4.00 3.99 3.88
300 5.66 5.65 5.57
200 6.93 6.92 6.86
100 9.80 9.80 9.75
60 12.65 12.65 12.61
















Source: Table constructed using readily available Internet sample size calculators.

 

Summer Sadistics Part One: Sample Size…
How Much is Enough?

Mixology (Putting research into practice)

Twist and Shout

About Us


We are proud to announce that CSR has been admitted as a full service member of CASRO (Council of American Survey Research Organizations).

Founded in 1975, CASRO is the trade organization representing research companies and organizations in the United States and abroad. Helping to advance and promote research standards, business ethics and practices or research processes, CASRO encourages members to offer their clients the highest quality of service and to grow their employees’ skills and proficiencies.



“Doubt is not a pleasant condition, but certainty is absurd.”

— Voltaire



Enter your email here to subscribe to “Research with a Twist


Problems? Click here to send us an email with your request.
About Us
The Center for Strategy Research, Inc. (CSR) is a research firm. We combine open-ended questioning with our proprietary technology to create quantifiable data.

 

(617) 451-9500
Understanding What People Really Think


WordPress Lightbox Plugin