While there, we became intrigued by the standing of qualitative research in the market research pantheon – can qualitative be rigorous? Systematic? Reliable? Scientific??Read this month’s newsletter to find out why we were “Blinded with Science!”
Julie Brown President
Mark Palmerino Executive Vice President
Blinded with Science…
At the LIMRA conference, both Mark and I attended an especially engaging panel discussion on “Leading with Emotion to Reach Today’s Customer,” with a focus on the marketing and sales of insurance products (guess we’re in the right business if this is what we find scintillating!). Buying insurance can seem like a purely rational decision (“I must protect my assets,” etc…), but there are emotions that drive it as well (fear, wanting the best for loved ones, wanting to leave a legacy, etc..).
In thinking about this, I was struck by how much better qualitative approaches are than quantitative at understanding and disambiguating the emotions and thoughts that underlie many purchase decisions. How in the world, in a web survey or a quantitative phone interview, would we be able to learn all of the emotional components of the decision, and how they all fit together to drive the purchase?
However, in talking with researchers throughout the event, as often happens, I heard more than once that “others in my organization do not perceive qualitative as rigorous.” This, as many of you who know me might guess, makes me both feel and think, “Where’s my soapbox?”
There are so many ways in which qualitative research can be rigorous. See, for example, a great piece in The Market Research Event blog, written by Annie Pettit, Research Director of Peanut Labs, detailing myriad ways in which most qualitative research is systematic, and therefore similar to scientific research methods. (Reading this of course gave me a “She Blinded Me With Science” earworm for about a week and a half. It was worth it!)
At CSR, our mission is to design and execute qualitative research that is as systematic, rigorous and reliable as possible. In fact, our firm was founded on the premise that market research would serve companies better if it combined the strengths of both qualitative and quantitative methods. In the spirit of Thomas Dolby, the following are just some of the ways that we are “blinded with science” to help our clients understand their customers and prospects better:
Use a large number of qualitative interviews
Part of the reason that qualitative research can be less reliable than quantitative approaches is that it’s often conducted among very small numbers of people. A focus group is typically 8 – 10 people, and many times two groups are all that are done.
For example, we recently received an RFP from a company that is planning to conduct a strategic study to understand the market for a new product it is launching. The company wants to confirm that there is a demand for the product among its B2B audience, and elicit feedback on how the product should be designed. Suppliers were required to propose completion of the study by conducting 2 focus groups – 20 respondents, maximum.
Kudos to them for recognizing that qualitative is the way to go in this situation – identifying needs and exploring how products would best meet those needs is certainly not best accomplished through closed-ended questions and ratings scales. And kudos to them for doing research at all. However, 20 is a pretty small sample size – the results will be directional at best – not reliable in a statistical sense. Many say that seeking any kind of reliability from qualitative feedback is unwise.
Well, we can!
At CSR, we always advocate for our clients to include at least 30 research participants – for each group that is important. So, in the example above, if there are two different types of decision-maker for the new product – one a financial executive, and one a human resources manager, then we’d recommend 30 interviews, minimum, for each group, or 60 interviews total. For a full accounting of why this is our magic number (and why it is for statisticians in general), see our July 2006 newsletter, titled “Summer Sadistics,” written by CSR’s Research Director, and statistics guru, Mark Palmerino.
“30” (or multiples thereof) is not only more statistically reliable, it’s also an easier target for non-research colleagues who don’t know much about research to get their minds around – have any of those at your company? ☺
Another reason that qualitative is frequently seen as less than scientific, and perhaps simply anecdotal, is the nature of the typical qualitative interaction. For example, in focus groups, not everyone answers every question. And even when they do, they could very likely be influenced by the preceding person’s response. Or, in IDIs, probes are often left to the interviewer’s discretion, so that research participants are not asked certain questions unless they’ve raised that particular topic themselves. And the order of questions asked can often by driven by the research participant.
We wonder about this often. Even when questions are very open-ended, why can’t they be asked in the same order of all research participants? For that matter, why would we want all of the research participants in the same room when we’re asking the questions (will those group members go to the store together to purchase our products)? And why can’t everyone be asked the same probes throughout each interaction?
Well, they can!
In comparison to other qualitative approaches, CSR’s is highly structured. We ask everyone within reason (we wouldn’t ask, for example, someone who doesn’t drive what they’d like to see in an auto insurance policy) the same questions in the same order, in a one-on-one interview setting. Probes are built into the guide, although our interviewers are trained to ask additional, clarifying questions as needed to understand the research participants’ points of view.
Advanced Coding Techniques
So, if we have a large number of interviews, say 60, or even 120 or 240, and if we’ve asked all research participants the same questions in the same order, why can’t we count each response to understand the frequency of responses? And why can’t we create data tables based on our calculations of the frequency of responses? And why can’t we make statistically reliable comparisons between groups?
Well, we can!
Similar to the way in which open-ended questions can be coded within an online survey, qualitative interviews can be coded in their entirety. Using CSR’s proprietary content analysis software, we are able to translate qualitative feedback into quantitative data – the “missing link” in most market research approaches. Our method couples the original verbatim answers with a variety of statistical analyses.
This allows our clients to create plans and take action based on what people really think and feel, not on their choice of a limited range of pre-selected options. And rather than anecdotal, unreliable qualitative results, your internal stakeholders will be happily … blinded with science! Science!
Here’s the Twist: Most researchers recognize that qualitative research is an excellent tool for understanding the feelings, attitudes, and beliefs of our target audiences. But there’s always a “but” involved – but qualitative is not rigorous, or systematic or reliable. By conducting highly structured, open-ended, one-on-one interviews among a larger group of research participants than we might otherwise consider for focus groups, then content-coding the responses to achieve statistically reliable results, we can have our proverbial cake and eat it too… qualitative research that’s … scientific! (Truly, poetry in motion!)
Mixology (Putting Research into Practice)
Some of you may be saying, “What’s the big deal? Why do I need structured and reliable qualitative data?” Well, let us share with you some of the practical benefits of our approach for our clients:
Precise counts of all responses: Rather than just saying “a majority of our audience said X,” or “a handful said Y,” findings can be more fine-tuned – “While 30% say Z, two times as many (60%) say ZZ.”
Comparisons between groups: When using the magic number of 30 for each key audience type, findings comparing those groups can be statistically reliable: “Finance executives are significantly more likely to find the product concept appealing than HR managers (80% v. 50%)”.
Ability to conduct advanced analytics: If the total sample size is big enough (i.e., 160 qualitative interviews or more), techniques like regression and factor analysis can be used on the data. We’ve partnered with many clients, for example, to create a segmentation of their customers or prospects. These results are similar to any segmentation, but based on the actual words of the research participants!
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result our clients gain more actionable and valuable insights from their research efforts.
Secondary Sidebar Widget Area
This is the Secondary Sidebar Widget Area. You can add content to this area by visiting your Widgets Panel and adding new widgets to this area.
The Center for Strategy Research, Inc. 101 Federal Street · Suite 1900 Boston, MA 02110