The Center for Strategy Research, Inc. Vol 3 Issue 6   June 2007


As Albert Einstein once remarked, “When you are courting a nice girl, an hour seems like a second. When you sit on a red-hot cinder, a second seems like an hour.”

Truer words were never said, and one need not be a renowned physicist to understand that when it comes to survey length as well, indeed, everything is relative. Today’s edition of Research with a Twist addresses that age-old survey question: “How long is too long?”

As always, please click here to send us your thoughts and comments.


Julie Brown
President

Mark Palmerino
Executive Vice President



How Long is Too Long?

I’m a big Harry Potter fan. I’ve read all of them, and as the publication date of the seventh (and final) book approaches on July 21st, I have to confess that I’m practically counting down the days until its release.

Some of the books in the series are immense (this new one will be 784 pages), and yet I know from experience that when I sit reading a Harry Potter book, time seems to stand still. So much so, that I’ve learned the importance of committing to a stop time whenever I begin reading, so that I don’t stay up all night turning page after page!

This isn’t the case with many books, of course, nor is it the case with most surveys; respondents are only willing to spend so much time discussing a given topic. And although there’s an obvious and direct correlation between how long you engage a person and how much information you are able to gather, one of the biggest challenges in crafting a research survey is determining optimal length.

The objective is to counterbalance the survey-giver’s thirst for information with the survey-taker’s attention, interest and patience, and to make sure that the former doesn’t become a burden on the latter.

In practice, finding this balance can be more art than science. However, we do keep four factors top of mind when setting survey length:

  1. Survey type. Among the many reasons we love open-ended surveys is the simple fact that respondents are consistently more willing to stay engaged for long periods of time when they are free to speak their minds. For example, we’ve frequently done surveys averaging 75 minutes per respondent, a duration that, to quote the good Dr. Einstein, would be like “sitting on a red-hot cinder” if the survey were of the “check the box that best applies” type.

    Open-ended surveys give participants more room to talk about what’s important to them, a key ingredient in keeping a person engaged. In addition, since the length of an open-ended survey is ultimately controlled by how much the respondent wants to talk in the first place, they don’t feel trapped, as is often the case in closed-ended surveys.

    There is a need to develop rapport at the beginning of an open-ended survey (see Mixology below for more on this), and it does require a certain amount of charm on the part of the interviewer. Under these conditions, however, interviewees are typically happy to talk at length.

  2. Survey topic. The more interesting the topic is to the respondent, the more time he or she will be willing to give. Interviewing Fortune 500 CEOs on the subject of innovation, for example, will give you much more leeway in survey length than would interviewing that same population on the topic of laundry detergent (unless, of course, you’re talking to P&G’s CEO). Innovation is likely to be a subject CEOs care about, think about and wrestle with, all of which adds up to a greater willingness to talk.
  3. Survey population. Open-ended, business-to-business interviews will typically last at least 30 minutes, and there seems to be a certain expectation of greater survey length when contacting people at their place of business, on a business topic. Asking a consumer, on the other hand, to participate in a 30-minute survey often leads to a refusal. The different populations have different expectations, and survey length needs to be geared accordingly.

    A member of a given group is also likely to be more engaged if he knows more about who is behind the survey. Happy (or angry) customers of a given company, for example, will typically be willing to spend more time. This is an important consideration to keep in mind when sponsor anonymity is not a requirement.

  4. Survey payoff. Offering money or similar compensation (gift certificates, sweepstakes) is a common reward for participation in a consumer study. For certain populations and situations, however, it ceases to be a major incentive (e.g. you can’t pay a CEO enough for their time), and other approaches are much more effective.

    For senior, business-to-business participants, for example, a promise of the rolled up results of the study in question can be a valuable reward for participation. Offering to redirect a stipend to one of several prearranged charities may also be more effective than a simple cash payout.

    In all cases, the promise of “payment” (however it’s constructed) goes into the respondent’s calculation of how much time he or she is willing to commit. And although it varies by individual, the greater the perceived value, the more willingness on the part of the respondent to endure additional survey length.

In conclusion, determining survey length is an essential consideration in the crafting of any research project. Too short and you won’t get the information you need; too long and you risk a negative impact on completion rates, information quality and even company reputation.

Thanks for reading, and as always, feel free to call me as late as you like with comments or questions… I’ll be awake anyway, reading Harry Potter and The Deathly Hallows.

— Julie

Click here to share this newsletter with a colleague.

Mixology (Putting research into practice)

Establishing rapport is an essential piece of any open-ended survey. The candidness and depth of information that this format provides assumes a high degree of comfort on the part of the respondent, and it’s up to the interviewer to create an appropriate atmosphere.

We stress three things in this regard, when training our interviewers:

  1. Start general before you get specific. Highly detailed or sensitive questions — even if they contain necessary information — should not be asked at the beginning. Instead, begin with more general inquiries, such as “So, tell me what it’s like to be a branch manager,” and save the others until after you’ve made a personal connection.
  2. Engage in active listening. The more a respondent feels he’s being heard, the more detail and depth he’ll provide in his answers. Clarifying probes, allowing the respondent to complete a thought before moving on to the next question, and even murmuring “uh-huh,” when appropriate, all encourage the respondent to continue talking.
  3. Adjust your speaking speed. Matching the respondent’s speed of speech is another factor in creating rapport. This is important, since residents of different parts of the country (or world) will vary in their typical speed. For example:
    • Easterners speak at 220 words per minute
    • Westerners and Midwesterners speak at 180 words per minute
    • Southerners speak at 140 words per minute

    In addition to these regional differences, interviewers keep tuned to changes in the respondent’s speed of speech, which may indicate that he/she is becoming frustrated (speeding up) or confused (slowing down) and modify their pace and speech patterns accordingly.

 

How Long is Too Long?

Mixology (Putting research into practice)

Twist and Shout

About Us


CSR has been retained to apply its in-depth interviewing and associated “qualitative-into- quantitative” coding methodology by several clients in the past, and in fact, an article detailing the research for one such client was published last year by the American Society of Training and Development (Follow this link to request a free copy of the article).

In addition, and although we don’t normally use this newsletter to announce client appointments, we are pleased to share that we’ve been retained by one of Australia’s leading banks to interview employees about their training and other development needs, expectations, and preferences. We’re proud to add an additional continent to our ever-growing list of client locations!



“Long enough to
reach the ground.”

— Abraham Lincoln, in response to the question, “How long do you think a man’s legs ought to be?”



Enter your email here to subscribe to “Research with a Twist


Problems? Click here to send us an email with your request.
About Us
The Center for Strategy Research, Inc. (CSR) is a research firm. We combine open-ended questioning with our proprietary technology to create quantifiable data.

 

(617) 451-9500
Understanding What People Really Think


WordPress Lightbox Plugin