Research with a Twist
blue lineVol. 11, Issue 4, May 2015
 CSR - Center for Strategy Research
 In This Issue…

twistAndShoutTwist and Shout


CSR will be presenting “Lessons From History: Learning About Strategy From the Battle of Gettysburg” at the LIMRA Marketing & Research Conference in Baltimore on May 27th to 29th. If you are planning to be there, contact us here, we’d be delighted to meet.

 Quote of the Month
The eye sees only what the mind is prepared to comprehend.

– Robertson Davies

 click here to subscribe to research with a twist
Questions? Click here to send us an email with your request.
(617) 451-9500

Ever wonder about the connection between NFL football and proper sampling procedures? We have! Please enjoy our newest edition of Research with a Twist, titled, “Inflated in New England.”

articleOneInflated in New England
As New Englanders and rabid Patriots fans, we’d love to be focusing solely on our fourth and most recent Super Bowl win right now. It was an instant classic: The thrill of victory, snatched from the jaws of defeat in the last seconds of the game.

Instead, we’re distracted by tangential, irrelevant matters – something about air pressure, and footballs, I think. But looking at the Boston media and talking with people in this area, the issue could just as well be the extent to which other sports towns are incredibly envious of the awesomeness of Tom Brady. Friends outside of this area seem to be talking about something called “Deflategate.” Sounds terrible, and I imagine it has to do with being let down (or “deflated”) after their own team’s less-than-championship seasons.

Of course, this whole situation leads us to think about market research. If we conducted a survey about The Pre-eminence of the Patriots, or The Awesomeness of Tom Brady outside of New England, we think that the results might be very different from the results of a survey conducted within New England. Not sure why (other than envy), exactly – we’d need to conduct qualitative research to understand that!

These regional differences are an example of one of the reasons that, in learning the market research business, we were trained to design studies around random sample selection. However, we notice that today, much market research leverages “convenience sampling,” which is when, “subjects are selected because of their convenient accessibility and proximity to the researcher.”

Convenience samples are certainly easier to identify and acquire, and therefore less expensive and faster to contact than random samples. This is why so many of us use them. In our experience, however, selecting research participants based on convenience rather than on a random basis creates a variety of problems for our businesses. Many of us have forgotten that the use of convenience samples invalidates the principles of statistics that we often apply when analyzing results.

An approach that seems to save money and time might actually be wasting it. Here are a few of the most prominent problems associated with convenience sampling:
  1. “Cherry Picking”

    One of our clients asked us to re-design a “wins and losses” study (apropos of the topic of football) among its customers and prospects in order to understand the impact, both positive and negative, of the organization’s sales process. The study our client had been conducting was presenting a much rosier picture than client research team members believed to be the case based on other feedback that had been gathered.

    Upon investigation, it turned out that the sales staff at our client company was providing all of the contact information for the study – for clients from whom they had won business, and from prospects with whom they’d not (yet) lost any opportunities.

    They provided this contact information with no input or guidance from the research team. It was so difficult to get even a few names from the sales people, that the researchers had been grateful for what they were able to get. Hence, the rosy picture – the sales team provided only the names of the most “friendly” customers and prospects.

    This is a classic example of “cherry picking” – selecting sample so that you only hear good feedback. That’s fine for sports fans – apparently, outside of New England, people are claiming Tom Brady had something to do with football maintenance? Who knew? We are tempted to say, “Who cares?” But in business, ignorance is not bliss, it’s potentially fatal. Obtaining feedback only from the customers and prospects your sales staff wants you to talk to can lead to serious lapses in sales strategy.
  1. No Comparison Group

    Another sampling pitfall we see on a regular basis is lack of a comparison group – people whose feedback acts as a baseline providing contrast, and therefore context, to what is being tested – because that comparison group might be less convenient to include in the study than the group being tested.

    In the NFL, for example, winning four Super Bowls might seem like no big deal unless you know that only three teams have won it more times. In the research world, for example, in the type of study discussed above, we don’t just interview “wins” and draw conclusions about best practices. We include “losses” in the study so we can understand both what works and what doesn’t from people who were both positively and negatively impressed. It’s through the contrast of the two groups that we gain the most insight.

    Including a control group can be costly and time consuming. But think of how much information is lost by having no point of comparison. For example, using the “Wins and Losses” study diiscussed above, we often hear about the quality of salespeople from both “wins” and “losses” in equal measure. This tells us that quality salespeople, while terrific to have, don’t win you more business, because your “losses” think your salespeople are great too. But had you only interviewed “wins”, you’d be tempted to think that your salespeople were a differentiating strength.
  1. Self-Selection Bias

    Without doubt, the most common way that research has been compromised by convenience sampling is through allowing research participants to self-select. Particularly online, we can’t “make” clients and prospects participate in research, therefore many of us don’t even try – we only talk with (online surveys) or listen to (monitoring social media)- those who volunteer.

    What are we missing by not talking to people who don’t immediately volunteer to provide feedback? Who knows? Just like we’ll never know what Tom Brady would be able to accomplish in the first four games of the 2015-16 season. There’s an old saw about marketing being 50% effective, but we’re not sure which 50%. Might the answer lie in the opinions of those we don’t survey?
  1. Invalidating Statistical Principles

    This issue is particularly critical and complex. So much so that we plan to devote our next newsletter to the topic!
Here’s the Twist:

While convenience samples seem, well, convenient – as well as less expensive and less time-consuming than randomly selected research participants, this approach carries inherent challenges. The impact of these convenience sampling on our research and our business decisions could be, well, deflationary.

– Julie

mixologyMixology (Putting Research into Practice)martini
The actionability of market research is thoroughly dependent on the representativeness of the research participants included in the study. Here are three best practices that CSR applies when it comes to improving sample quality:
  • Avoid cherry-picking: When relying on internal stakeholders, such as sales staff, to provide sample, executive ownership of the process is key. Assurance from senior leaders that the truth is more important than “CYA” can mean the difference between a bowl of cherries and plain old cherry-picking. Or, rely on non-human sources, such as sales and marketing support systems, to generate sample.
  • Create a point of comparison: It is not always possible to include a comparison group in research studies. When cost or logistics are prohibitive, we recommend a simple, but effective substitute, where the survey-takers become their own control group. If conducting a satisfaction study, for example, ask research participants to rate how satisfied they are not just with your company, but with other companies they work with.
  • Build research into the sales and marketing process: To minimize self-selection, encourage everyone to participate, and make it as easy as possible for them to do so. Let all of your customers and prospects know that part of the process in working with you is providing feedback because you really want to learn from them. Follow up with those who don’t quickly complete that satisfaction or onboard survey. This might improve both your sampling and perceptions of your brand at the same time.


aboutUsAbout Us
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result our clients gain more actionable and valuable insights from their research efforts.
understanding what people really think
WordPress Lightbox Plugin