The Center for Strategy Research, Inc. Vol 3 Issue 1   January 2007


Hello and Welcome!

We hope your holidays were safe and enjoyable.

We start the year off by taking a look at the limitations inherent in choosing the answers before asking the questions. If you’ve ever heard that chilling restaurant phrase, “no substitutions,” you’ll know what we mean.

As always, please click here to send us your thoughts and comments.


Julie Brown
President

Mark Palmerino
Executive Vice President



Special Orders Do Upset Us

A few weeks ago, in celebration of my wife’s birthday, we went out to dinner with our friends John and Linda. The restaurant we chose is one of my favorites, and each time I go, my biggest problem is deciding what to eat from among the options — I’d like to have them all!

My friend John, unfortunately, is not so lucky. John’s got a wide range of somewhat severe food allergies, and when he looks at a menu, he sees a long list of things that he needs to stay away from: shellfish, dill, nuts, to name just a few. For John, reading a menu typically leaves him wondering if he’s going to find anything he can eat at all. Fortunately, this restaurant was more than accommodating, but that’s not always the case.

It struck me later that night, that a restaurant’s menu is very similar to a closed-ended survey: Someone has laid out a number of “likely choices,” well before the guests arrive. When they do, they’re left to choose from the available options.

The limitations with this preconceived approach are many, and one of the reasons why closed-ended surveys — like restaurants that don’t accept “special orders” — can be so frustrating. Consider that…

  1. Unforeseen information can’t bubble up.

    It’s not irrational to think that a person who’s well- versed in the subject matter and very familiar with the audience (be they survey respondents or diners) could come up with a reasonable range of options ahead of time.

    The problem, however, is that by doing so, you are bounding the results at the outset. And while your closed-ended survey may indeed reveal which choice is most preferred among those offered, you’ve shut down the possibility that unanticipated information, answers, thoughts and insights may come to light.

  2. Significant changes may be missed.

    One nice thing about a closed-ended survey is that it makes year-to-year comparisons easy. You can see that question #23 went from 37% to 46%, and this type of tracking data is often quite useful.

    But what if question #23 is no longer relevant? What if you’ve upgraded your back office with a new computer system in the last year, and as a result, it no longer makes sense to measure processing speed in the same way?

    By the same token, what if the introduction of the new system has introduced new issues, many of which were not addressed by the old questions?

    If you simply conduct each year’s research using last year’s questions — and without the benefit of uncovering new issues and challenges that an open-ended approach will reveal — the relevance of your results may quickly decrease.

  3. You’ll cause unnecessary pain and suffering to respondents, and reduce the quality of information uncovered as a result.

    Okay, “pain and suffering” may be an exaggeration, but it’s true that picking choices from among survey questions (particularly if these choices are read aloud over the phone) is both boring and tiring. Instead of engaging respondents in a discussion that allows them to voice their opinions in the way they prefer, closed-ended surveys are tedious. And as a researcher, tedium works against you.

    Respondents who become tired or bored will often “game the system;” they’ll choose answers based on what they believe is a shorter path to survey completion. For example, if you’re conducting a survey regarding the use of computer equipment in my office, and each item I mention leads to its own set of follow-up questions, I may deliberately stop mentioning additional items!

    Compare that to open-ended questions, which energize respondents by making them think, and which give them the freedom and space to be heard.

In summary, of course, things are rarely so black and white, and closed-ended questions do have their place. Indeed, if your budget and timeframe permit, it’s often best to combine open- and closed-ended questions together in a given project. Using closed-ended questions as your only tool, however, may force respondents to choose from a menu, which may or may not satisfy your hunger for useful information.

Click here to share this newsletter with a colleague.

Mixology (Putting research into practice)

And speaking of the limitations of closed-ended surveys, take a minute to review the excerpt below from a recent edition of BenefitNews.com (emphases in article are ours):

Worker dissatisfaction becomes a mystery

As the New Year begins, many people are resolving to leave their current job and seek greener pastures. But why are employees dissatisfied with their current job?

About 60% of workers have resolved to find a new job in 2007, a recent survey of 5,173 workers by Job.com reports. What’s intriguing is that 35% of respondents selected the answer of “other” as their motive for seeking new employment.

Twenty-eight percent of respondents desire a higher salary, while 17% would like to improve their working conditions and 13% indicated that they need better health care benefits. Just 4% wish for a promotion, dislike their commute or dislike their boss.

There is plenty of room to speculate the “other” reasons that most job seekers are unhappy with their current jobs, says Brian Alden, CEO of Job.com. They may be overworked or, conversely, have jobs that aren’t challenging enough, he notes. “Maybe they feel that their company’s financial stability is in jeopardy. It may also be that individuals want to take advantage of the improving job market as we enter the 2007 year,” he adds. “It’s extremely interesting to learn that we can no longer generalize the reasons for why individuals make finding a new job their New Year’s resolution.”

This is a good example of what happens when a set of options is lacking. Respondents do the only thing possible and choose “other” — not necessarily because they’re unsure of what they want as the article implies, but perhaps because they’re constrained by the survey itself. After all, how likely is it, really, that an employee doesn’t know why he or she wants a new job?!

Without an open-ended approach, we’re left to “speculate” about what “other” might mean.

Have a great month, and we’ll see you (choose one)
__ next time __ next year __ never again __ other.

 

Special Orders Do Upset Us

Mixology (Putting research into practice)

Twist and Shout

About Us


We’re delighted to announce that CSR continues to grow, and has been enhanced by the recent addition of Lew Hollerbach as Research Manager.

Lew comes to us from Aberdeen Group, where he was a Research Director covering service providers. At Aberdeen, Lew provided strategy advice, market positioning analysis, and messaging assistance for new products, based on IT industry market research. He wrote numerous analyses, research reports, and appraisals of markets, companies, and products for clients such as Qwest, Sun Microsystems, and numerous start-ups. Lew was a frequent speaker and panelist at industry events, and was quoted in the Washington Post, Wall Street Journal, and many trade publications.

Lew is a 30-year veteran in the information technology and training development fields. As computers became faster, smaller, and more versatile, Lew (by his own admission) became slower, bigger, and more versatile. He has accumulated extensive experience in project and business management, client relationship development, and education and training –
in both domestic and international settings.

At CSR, Lew is responsible for managing our unique coding processes and programs, including quality control, and is also managing client relationships. As he is an avid pilot, like Mark, we all look forward to watching dog fights in the skies over suburban Boston when they are not working together to further enhance our unique coding processes and programs, and generally enhancing CSR’s value to its clients.



“You never know what is enough unless you know what is more than enough.”

— William Blake



Enter your email here to subscribe to “Research with a Twist


Problems? Click here to send us an email with your request.
About Us
The Center for Strategy Research, Inc. (CSR) is a research firm. We combine open-ended questioning with our proprietary technology to create quantifiable data.

 

(617) 451-9500
Understanding What People Really Think


WordPress Lightbox Plugin