The Center for Strategy Research, Inc. Vol 7 Issue 6   September 2011


As market researchers, we often assume that more – more questions, more options, more participants – will lead to better results and insights.

As a recent NY Times article points out, however, in many cases, less is indeed more.

Read on for more less…

Julie Brown

Mark Palmerino
Executive Vice President

Three “Enhancements” That Will Hurt Your Research
Maybe you saw this column in the NY Times last July, the one in which Nasahn Sheppard, director of industrial design at Smart Design, explained what went into the development of the Flip camcorder.

It’s a fascinating piece, and one which makes an excellent case for the benefits of simple, focused, single-purpose design. As Sheppard explained:

“Throughout the development of the Flip, there was one question the team repeatedly asked ourselves: ‘What can we take out, not what can we put in?'”

This approach is certainly not the norm.

Whether developing products or services, most companies work from the position of “What can we add to make this better?” The problem, of course, is that sooner or later, “better” becomes too complicated, too confusing and too expensive. (Microsoft Word, anyone?)

It’s not the norm in market research either. Over the years, we’ve been involved in any number of engagements in which a seemingly simple project, with the objective of addressing a relatively straightforward set of issues, begins ballooning into something much more complex and complicated.

To help avoid this “bloated ballooning” in your own work, beware of these three common traps when planning your research:
  1. Too many questions.

    Adding more questions doesn’t necessarily improve the research.

    First, because even if expanding the number of questions didn’t increase cost (it often does), at some point, the additional information makes your decision-making harder. Too much collected information can camouflage the answers you’re truly in need of and distract from what really matters in the context of your research.

    Second, because it tends to dampen participation. Even if you end up with a beautifully constructed survey that satisfies all your internal requests, you still need to find enough people who are willing to participate.

    As survey length increases, you’ll find it both harder to recruit participants and more likely that those who do agree either abandon the survey part way through or simply rush to the end to earn their reward.
  1. Too many options. Even a relatively short survey, if not structured well, can be mind-numbing for participants.

    Grid formats, for example, are notorious for this, thanks to their multi-dimensional layout which requires respondents to “learn the instrument” in order to participate. (See Mixology, below, for more on using grids.)

    Similarly, surveys that offer many, many choices for each question or that fail to use skip patterns efficiently can also cause participants to fatigue and/or abandon surveys that might otherwise have captured useful information.
  1. Too many people.

    In terms of participants, good research is not strictly a numbers game; we’ve written here many times before of the value in “going deep” when developing survey instruments. While it may feel more comprehensive to involve hundreds (or thousands) of people, not all respondents are created equal.

    For example, rather than 70 closed-ended questions of 2,400 people in an online survey, there are times where asking 30, good, in-depth question of 120 participants will yield more useful information. Not only does a free-flowing discussion format engage participants more, it gives you much more insight into what they really think and why.

    The open-ended approach is also particularly useful in the early stages of research when the range of relevant questions and associated answers is not yet well understood.
So how much — questions, options, people — is too much? As with most worthwhile topics, it depends.

Here’s the Twist: The important thing to remember is that just as the efficiency and user-friendliness of The Flip was as much about what was left out as it was about what was put in, great research is not simply about more.

Simple, focused, bloat-free design is a pretty good mantra for going after the insights you need.

— Mark

Click here to share this newsletter with a colleague.

As mentioned earlier, there’s a great deal of “it depends” in optimizing instrument development. That said, we rely on the following rules of thumb, based on our experience as both research professionals and participants:

  • Keep closed-ended surveys to 10 minutes or less. You’ve probably noticed the number of retailers who have adopted a one- or two-question e-mail follow-up format in querying customers. They recognize that most people no longer have the time or inclination to get involved in long format, self-guided surveys.
  • No more than five to seven grid-formatted questions. Yes, the information these provide is valuable. But if respondents have to work hard at completing the survey it will feel much longer to them, leading to abandonment and less thoughtful responses.
  • Aim for 30 minutes with an open-ended format. It’s not hard to keep someone engaged for half an hour when they are actively involved in a two-way discussion. You can comfortably cover 5–7 topics well in this period of time.
  • We considered adding one or two additional recommendations, but in the spirit of less is more, we’ll stop here!


Three “Enhancements” That Will Hurt Your Research

Mixology (Putting research into practice)

Twist and Shout

About Us

We will be attending LIMRA’s Group and Worksite Benefits Conference, scheduled for Sept. 7 to 9 in Chicago, and hope to see you at our booth if you are also planning to be there!

Any intelligent fool can make things bigger and more complex… It takes a touch of genius — and a lot of courage — to move in the opposite direction.

— Albert Einstein

Problems? Click here to send us an email with your request.
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result, our clients gain more actionable and valuable insights from their research efforts.

Understanding What People Really Think

WordPress Lightbox Plugin