If you've ever lived with a dog, I don't think the
following anecdote will come as a surprise to
you…
Our 4-year-old Cavalier King Charles spaniel,
affectionately known as Dickens, isn't allowed to eat
off of our china. He's permitted to use his own doggie
dish, but we try to draw the line at slurping dog
tongues on food and dishes meant for human use.
How well does it work? Well, you decide:
When we
are around, he is the perfect little gentleman. If
we have occasion to leave a dish unguarded, even
for a minute, it's a different story. How do we
know? First, because we'll sometimes notice that
things are missing from the dish after we return, and
there will be Dickens, wearing one of those guilty
dogfaces that we pet owners know all too well.
Second, and much more incriminatingly, we
sometimes catch him up on a chair, happily wolfing
down our food!
Clearly, dogs behave differently when they think
nobody is watching. Guess what? So do people.
CSR recently completed research, in which we
conducted 63 in-depth interviews with market
research executives, to better understand the
decision-making process around various market
research initiatives (see Twist and Shout
sidebar to
download your own complimentary copy).
Among the many findings was this: Researchers
are highly satisfied with Internet-based research.
(Just over half indicated that they are very satisfied
with this methodology and no one expressed
dissatisfaction with it.) Specifically, respondents cited
the Internet's cost effectiveness (28%), ease of
reaching respondents (22%) and, most importantly,
its fast implementation or access to results (41%), as
reasons for their high satisfaction.
There is, however, one significant drawback with
Internet-based research, which many of these
research professionals cited: Anonymity. Like a
dog alone in the room with
enticing food for the taking, survey-takers who
answer questions on their own, without the
involvement of an interviewer, will often behave
differently (i.e. badly).
These differences may include misrepresenting
oneself (e.g. claiming decision-making authority or job
responsibility beyond what it actually is), racing
through a survey, choosing survey answers
indiscriminately, and others. All of these things
add up to reduced survey validity.
Internet-based surveys, therefore (and by the way, the
same could have been said of paper-based surveys
twenty years ago), require careful development, as
well as vigilance in analysis, to ensure that the
results truly reflect the views of the population in
question.
Consequently, we employ a number of tactics
when using Internet-based surveys, to identify and
(possibly) discard surveys that are less than ideal:
- Time Checking. "Speeders," is an
industry term for survey-takers who move too quickly
through a survey to have read (much less
considered) all of the questions. By assigning a
minimum threshold of survey completion time (e.g.
five minutes) and noting those which fall below, we
may remove these from the mix.
- Mid-Stream Verification. Including a
"question" such as, "When you read this question,
check answer number seven," in the middle of a
survey — and flagging those people who
don't check answer number seven as
instructed — we uncover individuals who are
not being attentive.
- Repetition. Often, we'll include a question
which asks for the same information in a different
way, later on in the survey. Here as well, internally
inconsistent answers are helpful in identifying those
survey-takers who are not engaged or not answering
truthfully.
- Pattern Observation. Some survey-takers
are known as "Straight-liners." These people pick a
particular answer (4 on a 5 point scale, for example)
and simply check that answer over and over again.
To uncover this behavior, we agree on a reasonable
number of straight line choices in a row; anything
beyond that is flagged.
- Unreasonable Options. People who
misrepresent themselves and/or their qualifications
can be hard to uncover and the use of unreasonable
options is one way of getting at this. The idea here is
to offer choices that no truly knowledgeable person
within the population being surveyed would select. If
one of these answers is chosen, it's a good sign that
the survey-taker is unqualified or not engaged.
- Logical Inconsistencies. Certain types of
answers logically go hand in hand, and when they
don't appear that way, we've got another red flag. For
example, if you state in one part of a survey that your
company has 5,000 employees, but then check a box
later on that says your company is too small to qualify
for insurance, there may be an issue.
Of course, none of these is absolute, and part of
the challenge is deciding how strictly to apply these
rules. The most conservative approach,
naturally, would be to exclude anyone who
had any error. In doing so, however, we might also
exclude valid respondents, who may have simply
made an honest mistake or misread a question.
Keep in mind as well, that a "one strike and you're
out" policy towards errors could result in the
elimination of a large percentage of completed
surveys (e.g., 30%, 40%, 50% or more), which would
significantly raise the cost and time involved in
completing the research.
In the end, and whatever you decide to do, the
important thing to remember is that your survey
results — and by extension, the critical
business decisions you make based on these results
— are only as good as the individuals that you
allow to participate and the answers they provide.
As the old joke goes, "On the Internet, nobody
knows you're a dog." Woof, woof!
