Some of you are receiving this newsletter for the first time because you visited CSR’s booth at last month’s LIMRA Marketing & Research Conference and you expressed interest in receiving this. First, thank you for stopping by and introducing yourself… we enjoyed meeting so many of you!… and… we hope you enjoy reading our newsletter as much as we enjoy creating it!
The ultimate measure of a man is not where he stands in moments of comfort and convenience, but where he stands at times of challenge and controversy.
– Dr. Martin Luther King, Jr.
Questions? Click here to send us an email with your request.
Summer is here, and the livin’ is easy – or is it convenient? We hope that you enjoy our newest edition of Research with a Twist, titled, “What’s Wrong with Convenience, Anyway?”
What’s Wrong with Convenience, Anyway?
Last month, in the wake of the New England Patriots’ fourth SuperBowl championship (and some other nonsense), we talked about the rise (or inflation) in the use of convenience samples among our market research brethren, and the decline (or deflation) in samples randomly selected. A convenience sample is one in which participants are selected because of their accessibility to the researcher, rather than a desire to reflect the population as a whole, which is traditionally achieved by random sampling.
Since that newsletter, the 2014 Patriots have officially received their super-diamond-y SuperBowl rings, and now we watch baseball for a while. Baseball is not as hard-hitting as football, or as fun, and it’s even mildly boring, but it’s conveniently accessible in the summertime, leading us to wonder, what’s wrong with convenience, anyway?
Well, convenience is fine in many situations, but in sampling, not so much. Some of the challenges associated with using convenience samples, which we discussed last month, include “cherry picking,” lack of a comparison group, and self-selection bias. Most importantly, though, the use of convenience samples invalidates the principles of statistics that we often apply when analyzing results. The following are some of the implications of that “Inconvenient Truth”:
Representativeness loses to convenience
When designing a research study, many of us struggle to achieve specific numbers of completes overall and within targeted populations in order to achieve statistical reliability and/or validity. In other words, we construct sample and target plans in order to convincingly assert that the research results will be actionable and projectable to the population at large.
What many of us have forgotten, however, is the extent to which the use of convenience samples renders typical statistics virtually worthless!
With non-randomly-selected samples, we are not assured that the sample is representative of the population from which it was selected. For example, what if CSR surveyed 1,200 New Englanders about “Deflategate,” and the results show that 99% support Tom Brady? Would the results be reliable nationally (meaning you can count on selling more cereal in the U.S. with his picture on the box)? Or valid and projectable to the U.S. population at large? The answer is yes – as long as the “population at large” comprises only New Englanders too! The bottom line is that it’s not just the quantity of the research participants that is important, but the quality, in this case, defined in terms of representativeness.
You don’t need as many as you may think
So, how many research participants do you need in order to have actionable, valid results? Our late, great co-founder and Research Director, Mark Palmerino (insert appropriate toast here), put it this way in a baseball-season newsletter several years ago:
“To better understand the question of sample size, let’s consider a simple example… baseball. Suppose a particular baseball player were up to bat one time and got one hit. That player’s average would be 1,000 (“batting a thousand”). Now, suppose that same player gets up to bat again, but this time strikes out. Now, his average drops in half and he’s only batting .500 (1/2).
But what if that same player instead got a hit in his first 29 times up to bat? At that point, he’d still be batting 1.000 (29/29). But now, if he strikes out on his next at bat, his average only drops to .967 (29/30)! The critical difference of course, is times at bat. In the early going, each at bat has a tremendous influence on a player’s average. With each successive at bat however, and even if the results are extremely different from what occurred before, the impact on the total is less.”
Ipso facto, in a market research study, for a simple yes or no question, once some relatively low number (say, 30 to 50) randomly-selected research participants have been surveyed, results are not likely to vary much. And while surveying additional people will improve the certainty of the results, it’s unlikely (again, provided participants are selected randomly) that the next 50 people are going to respond in a significantly different way than the first 50. See, baseball might be mildly boring, but we learn stuff from it!
So, in today’s convenient world, what alternatives do we have?
The objection we hear most often when we propose qualitative approaches to our clients for larger, more strategic studies is that cost is prohibitive. Clients tend to recognize that having in-depth conversations with clients and prospects gains more insight than surveying them online, but often believe that the methodology is not affordable.
However, as we’ve shown, one key factor is that sampling randomly reduces the number of research participants that need to be included in order to accurately reflect the population or sub-group that you want to understand. Interviewing fewer, but more randomly selected participants can make it cost- effective to have a meaningful interaction with your customers and prospects, particularly when using CSR’s “twist” on qualitative research. You can then take the emphasis off “how many responded” in order to focus on “what did they say”, “how”, and “why”!
Now, let’s get back to waiting for Tom Brady’s return, er, uh, I mean, watching baseball!
Here’s the Twist:
The inconvenient truth is that random sampling is at the core of market research. Study participants that are easy to procure can undermine the reliability and validity of your results. The good news is that if sample is randomly selected, the number of research participants does not have to be huge, which makes qualitative research (done the right way), a more viable option in many cases. Viva la Random!
Mixology (Putting Research into Practice)
In studies where total completes are key, consider channels and venues you wouldn’t normally use to obtain feedback from populations that might be under-represented in your primary recruiting methods. Focus as much attention on the difficult-to-reach targets as the convenient using the following practices:
Use phone or face-to-face interviews to supplement the target research participants that are reluctant to participate in on-line surveys.
Send invitations to participate in online surveys at different times of day to attract the attention of a broader audience.
Offer incentives that are appropriately (and likely, differently) oriented to the needs of different populations.
Change the emphasis from the total number of completes to assuring that you have conducted your study in ways that include the less-conveniently-reached members of the population you wish to survey.
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result our clients gain more actionable and valuable insights from their research efforts.
Secondary Sidebar Widget Area
This is the Secondary Sidebar Widget Area. You can add content to this area by visiting your Widgets Panel and adding new widgets to this area.
The Center for Strategy Research, Inc. 101 Federal Street · Suite 1900 Boston, MA 02110