A few weeks ago, in celebration of my wife's birthday, we went out to dinner with our friends John and Linda. The restaurant we chose is one of my favorites, and each time I go, my biggest problem is deciding what to eat from among the options — I'd like to have them all!

My friend John, unfortunately, is not so lucky. John's got a wide range of somewhat severe food allergies, and when he looks at a menu, he sees a long list of things that he needs to stay away from: shellfish, dill, nuts, to name just a few. For John, reading a menu typically leaves him wondering if he's going to find anything he can eat at all. Fortunately, this restaurant was more than accommodating, but that's not always the case.

It struck me later that night, that a restaurant's menu is very similar to a closed-ended survey: Someone has laid out a number of "likely choices," well before the guests arrive. When they do, they're left to choose from the available options.

The limitations with this preconceived approach are many, and one of the reasons why closed-ended surveys — like restaurants that don't accept "special orders" — can be so frustrating. Consider that…

1. Unforeseen information can't bubble up.

   It's not irrational to think that a person who's well- versed in the subject matter and very familiar with the audience (be they survey respondents or diners) could come up with a reasonable range of options ahead of time.

   The problem, however, is that by doing so, you are bounding the results at the outset. And while your closed-ended survey may indeed reveal which choice is most preferred among those offered, you've shut down the possibility that unanticipated information, answers, thoughts and insights may come to light.

2. Significant changes may be missed.

   One nice thing about a closed-ended survey is that it makes year-to-year comparisons easy. You can see that question #23 went from 37% to 46%, and this type of tracking data is often quite useful.

   But what if question #23 is no longer relevant? What if you've upgraded your back office with a new computer system in the last year, and as a result, it no longer makes sense to measure processing speed in the same way?

   By the same token, what if the introduction of the new system has introduced new issues, many of which were not addressed by the old questions?

   If you simply conduct each year's research using last year's questions — and without the benefit of uncovering new issues and challenges that an open-ended approach will reveal — the relevance of your results may quickly decrease.

3. You'll cause unnecessary pain and suffering to respondents, and reduce the quality of information uncovered as a result.

   Okay, "pain and suffering" may be an exaggeration, but it's true that picking choices from among survey questions (particularly if these choices are read aloud over the phone) is both boring and tiring. Instead of engaging respondents in a discussion that allows them to voice their opinions in the way they prefer, closed-ended surveys are tedious. And as a researcher, tedium works against you.

   Respondents who become tired or bored will often "game the system;" they'll choose answers based on what they believe is a shorter path to survey completion. For example, if you're conducting a survey regarding the use of computer equipment in my office, and each item I mention leads to its own set of follow-up questions, I may deliberately stop mentioning additional items!

   Compare that to open-ended questions, which energize respondents by making them think, and which give them the freedom and space to be heard.

In summary, of course, things are rarely so black and white, and closed-ended questions do have their place. Indeed, if your budget and timeframe permit, it's often best to combine open- and closed-ended questions together in a given project. Using closed-ended questions as your only tool, however, may force respondents to choose from a menu, which may or may not satisfy your hunger for useful information.

And speaking of the limitations of closed-ended surveys, take a minute to review the excerpt below from a recent edition of BenefitNews.com (emphases in article are ours):

Worker dissatisfaction becomes a mystery

As the New Year begins, many people are resolving to leave their current job and seek greener pastures. But why are employees dissatisfied with their current job?

About 60% of workers have resolved to find a new job in 2007, a recent survey of 5,173 workers by Job.com reports. What's intriguing is that 35% of respondents selected the answer of "other" as their motive for seeking new employment.

Twenty-eight percent of respondents desire a higher salary, while 17% would like to improve their working conditions and 13% indicated that they need better health care benefits. Just 4% wish for a promotion, dislike their commute or dislike their boss.

There is plenty of room to speculate the "other" reasons that most job seekers are unhappy with their current jobs, says Brian Alden, CEO of Job.com. They may be overworked or, conversely, have jobs that aren't challenging enough, he notes. "Maybe they feel that their company's financial stability is in jeopardy. It may also be that individuals want to take advantage of the improving job market as we enter the 2007 year," he adds. "It's extremely interesting to learn that we can no longer generalize the reasons for why individuals make finding a new job their New Year's resolution."

This is a good example of what happens when a set of options is lacking. Respondents do the only thing possible and choose "other" — not necessarily because they're unsure of what they want as the article implies, but perhaps because they're constrained by the survey itself. After all, how likely is it, really, that an employee doesn't know why he or she wants a new job?!

Without an open-ended approach, we're left to "speculate" about what "other" might mean.

Have a great month, and we'll see you (choose one)
__ next time __ next year __ never again __ other.