The Center for Strategy Research, Inc. Vol 7 Issue 7   November 2011


It’s easy to overlook important environmental changes which may affect your market research, particularly if the research is similar (or identical) to projects you’ve completed before.

Today’s newsletter highlights the importance of bringing fresh eyes to every project.

Read on for more…

Julie Brown

Mark Palmerino
Executive Vice President

Baby You Can Drive My Car (Or Truck)
Here in Boston, each Labor Day weekend, it happens at least once.

Torrential rain? Mass exodus of tourists leaving Cape Cod? Red Sox crumbling? All likely possibilities, certainly, but I’m referring to something else.

The one thing that happens over Labor Day weekend, year after year, is that someone — usually a college student in town for the first time — attempts to drive a rental truck on Storrow Drive, one of the main east-west thoroughfares through the city. The problem, as the soon-to-be inconvenienced student discovers, is that the overpasses on Storrow Drive are exceedingly low; sooner or later the truck gets stuck!

It invariably makes the news and, assuming nobody gets hurt, it’s always cause for a chuckle. I admit to chuckling right along with everyone else, year after year.

Until now. Because this year, I was the geographic neophyte, making my way back home from Brooklyn in a rented U-Haul. My daughter Emily has started graduate school at Hunter College in Manhattan, and I had just moved Emily and her possessions into her new apartment.

We said our goodbyes the next morning and I hopped on the Hutchinson Parkway, blindly following the cheerful instructions of the woman whose voice came out of the GPS. There were many signs declaring “No Trucks Permitted,” but neither I nor the cheerful woman seemed to notice.

So I kept driving. More warning signs… more driving.

Suddenly, it registered. I still wasn’t sure if the truck I was driving was considered “a truck” in the eyes of the Hutchinson Parkway, but with the overpasses getting progressively shorter, and despite the protests of Ms. GPS, I decided to take the safer path and make my way over to the more vertically hospitable route 95.

I arrived home a few hours later without incident, but my driving adventure did get me thinking about parallel challenges when it comes to market research:
  • Critical assumptions are made without anybody realizing the danger.

    Both my GPS and I made an unconscious assumption: My vehicle is a car, not a truck. I saw the warning signs at the entrance and along the parkway, but I was so used to disregarding them (“I’m not a truck”) that I didn’t become aware of this important distinction until it was almost too late.

    Similarly, closed-ended surveys make critical assumptions regarding both the questions and the answers. Because both are written prior to the arrival of the respondents, the survey-builder (the market researcher) decides what’s appropriate and logical beforehand, in each situation.

    Often it works, but many times it doesn’t. Warning signs include a large number of respondents selecting “other;” patterns of responses that don’t make sense (e.g., the choice you expected to rarely be picked is selected frequently); and comments such as “This is a silly question” or, “I really don’t know how to answer that.”

    The challenge here, of course, is that like driving in the wrong vehicle, by the time you realize the problem — if you realize the problem — it’s hard to recover. It might require more research (expensive) or, even worse, bad decisions regarding the best way to move forward in your business.

    Uncovering hidden assumptions is one of the reasons we recommend a qualitative phase at the beginning of any research project (see Mixology below for more on this).
  • Doing the same thing repeatedly can cause you to stop thinking.

    Just as I ignored environmental factors which indicated my world had shifted, researchers who have done the same thing, in the same way, for years, often overlook important changes.

    For example, maybe I’ve managed a particular satisfaction study for the past 15 years. The questions and the audience are intentionally uniform, allowing for year over year comparisons and quite a bit of efficiency in the process itself.

    But the Internet, social media, changes in the market, or any number of other things, could make that tried and true survey ineffective. Asking people about their retirement plan, for instance, when the value of that investment has been pummeled twice in the last five years, might suggest a very different question than it once did.

    Here as well, researchers need to make sure we don’t miss big changes (i.e., “Are we still driving a car?”).
  • Poor definitions lead to problems.

    When I got home from my trip, the first thing I did was Google the question of trucks and the Hutchinson Parkway. I’m still not sure what the answer is, though, since the official restriction is for “commercial trucks,” not trucks in general. It’s not even clear if this is a safety issue, a policy decision, or something else entirely.

    Poorly defined screening questions introduce confusion as well, greatly increasing the risk of including the “wrong people” in a study. In general, the more objective the questions — age, sex, income, occupation, etc. — the better. Questions that require more subjective thought or definition, such as “Are you a self-directed investor?” can lead to inaccuracies or different categorizations than you intended.
Here’s the Twist: Each of these points is an example of how market research — particularly when conducted by those with a great deal of experience in a particular area — can be blown off course.

For the best results, it behooves us to approach each project with eyes open, heads up and a healthy suspicion of voices — cheerful or not — that insist on pointing us in the same old direction and in the same old ways.

— Mark

Click here to share this newsletter with a colleague.

One way of mitigating the three problems described above is to build an exploratory, qualitative phase into your survey process, typically performed before a one-time quantitative study and/or between survey cycles of an ongoing study, and as a means of keeping a pulse on how things may be changing.

In practice, here’s what we do…

  • Evaluate the question and answer options. By including in-depth, one-on-one interviews we are able to figure out the best answer sets. By asking people to describe their banking experiences and preferences, for example, we can make sure that the choices we offer in closed-ended surveys are relevant and meaningful.
  • Evaluate the environment. How do you make sure the research path you’ve used in the past is still working? Open-ended research around big topics — economic, political and social — will help you understand important trends. Asking respondents to describe how the economy has affected their overall purchasing behavior, for example, will provide insights that keep your research on track.
  • Evaluate the screening process. When you can’t rely solely on objective questions of age, sex, income, etc., make sure your screening questions are as unambiguous as possible. Take the time to define any terms that may be open to interpretation (e.g. “self-directed investor”) and ensure that those involved — both screener and screened — are working with the same definitions of those terms.


Baby You Can Drive My Car (Or Truck)

Mixology (Putting research into practice)

Twist and Shout

About Us

As we have for many years, we will be attending The Market Research Event, scheduled this year in Orlando, Florida, November 7–9, at the Peabody Hotel.

Please respond to this email if you are also planning to attend. We look forward to seeing you there!

“Not to mince words, Mr Epstein, we don’t like your boys’ sound. Groups of guitarists are on the way out.”

— Decca records executive to Brian Epstein, upon hearing the Beatles for the first time

Problems? Click here to send us an email with your request.
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result, our clients gain more actionable and valuable insights from their research efforts.

Understanding What People Really Think

WordPress Lightbox Plugin