 |
 |
Luxury in the Eyes of the B(MW)-Holder |
 |
 |
|
| |
Have you ever been frustrated by the wording or
sequencing of a closed-ended survey? If so, then,
as Alice Roosevelt Longworth would say, "Come sit
next to me."
It's not that I enjoy complaining, however, a recent
experience with an online, quantitative survey left me
feeling both confined (at my inability to express myself
within the constraints of the survey) and concerned
(that the end client may not be getting the most
accurate information).
The survey was purportedly about "luxury"
automobiles and indeed, in one of the first
questions, I was asked if I "currently drive a luxury
automobile."
The answer choices available to me were "Yes" and
"No." I was immediately flummoxed.
What constitutes a luxury automobile? I
don't consider my nine-year-old, base level model,
100,000-plus miles BMW a luxury automobile
(although I certainly recognize that some might). So,
after some consideration, I checked "No."
Next question: "Would you consider buying an
automobile in 2009?" Again, I was given the
exceedingly unsatisfying options of "Yes" or "No." (I
was starting to sense a pattern.)
Now, to be honest, I really wouldn't mind breaking in a
new car this year (hey, who wouldn't?), and my
husband and I talk about a replacement for my aging
roadster on at least a monthly basis (usually about the
time a new repair is required). But I'm just not sure
that
the family budget will support this as a priority before
the end of the year. So I checked "Yes," focusing on
the word "consider" in the question.
Next question: "Would you consider buying a
luxury automobile in 2009?" Again with the luxury
automobile vagueness. And the answer choices? You
guessed it… "Yes" and "No."
OK, so now, I'm thinking, even if I would
consider a car in 2009, this wouldn't be the year
for that SLK I've been eyeing. So, my answer: "No."
And then, the kicker: "Please tick all the brands
you would consider purchasing in 2009."
This was accompanied by a comprehensive list of
auto manufacturers, one of which was BMW. Now, I've
been pretty happy with my wheels, so I checked BMW,
along with at least four other names.
Wrong! The survey immediately shut down, and a
warning note flashed that I had given inconsistent
answers. The survey was terminated. Goodbye.
Let me just stop right there and tell you that I really do
have better things to do with my weekend mornings
than worry about a survey cutting me short. But I was
pretty annoyed. So I emailed the administrator, made
clear that I actually had been paying attention, and
explained my predicament.
Today, now several days later, I'm still waiting for an
answer. And while I'm beginning to suspect that I'll
never hear back from these folks, the experience
did give me a renewed appreciation for why research
is so often criticized for not delivering what it
should.
This particular survey had (at least) three significant
problems in the way it was constructed:
- A lack of definition regarding what
constitutes a "luxury brand." Had the survey either
started with a clarifying question such as, "Which of
the following brands do you consider luxury brands?"
or, "What do you consider a luxury automobile?" or,
worst case, if the questionnaire provided a list of
which
brands "should be considered luxury for purposes of
the survey," the survey-taker and the survey
administrator would have been on the same page.
Instead, the survey writers assumed
that
their definition of "luxury" was the same as that
of the participants, an assumption that got us off
on the wrong foot almost immediately. Furthermore,
the limited, "Yes or No" option prevented me from
expressing my own uncertainty.
(Note: This highlights one of the
benefits of
conducting qualitative research as a first step,
prior to fielding a large-scale, quantitative study. This
type of definition ambiguity would almost certainly
have been brought to light through open-ended
questioning.)
- An overzealousness in weeding
out "bad
survey takers." We certainly understand the
market researcher's desire to weed out "low quality"
survey participants from the sample base. These
include the speeders (those who move too quickly
through surveys), the cheaters (those who check
boxes at random) and the repeaters (those who take
surveys many times over in exchange for the stipend).
But in our zeal to uncover the bad guys,
we do
ourselves a disservice by throwing out well-meaning
folks who are victims of the survey itself or, of their
own, understandable, human inconsistencies.
If, instead, the administrators had allowed
me to
continue, and had programmed in an opportunity for
me to explain the apparent inconsistency (an
open-ended comment box at the end, for example),
they
might not have had to toss out my responses
entirely.
- An abrupt termination at the first
perceived
inconsistency. The other problem with the way the
survey ended is that it was just bad form. The
message I got — loud and clear — was
that there was something wrong with my answers.
While we all understand the reluctance to reward
survey respondents who don't take their participation
seriously, we also need to appreciate that not all
inconsistencies in responses are the participant's
"fault."
Even putting aside what are just the
basics of
common courtesy, if one hopes to enlist participants
in
future studies, it's never a good idea to "slam the
door" (or, in this case, the termination screen) so
abruptly in the face of a survey taker.
Here's the Twist: It's easy to mistake
poor quality surveys for poor quality participants. And
while it certainly makes sense to weed out those who,
for whatever reason, may bias the data, we need
to be equally vigilant about not improperly and/or
rudely dismissing the well-meaning majority of
participants.
Just as we would never simply hang up the phone
on
a telephone survey participant whose answers were
believed to be inconsistent, online surveys takers
should be treated with care and respect. When we
consider how important the cooperation of survey
participants is to our profession, not to mention the
increasing difficulty of getting people to participate in
the first place, it's in our best interests to treat people
as the 'luxury" participants that they are.
— Julie
|
| |
|
| Click here to share this newsletter with a colleague. |
 |
| |
While terminating a research participant based on
one possible inconsistency is not a great tactic,
we do realize the importance of identifying and
removing those who are legitimately questionable.
To this end, we recommend a "three strikes and
you're out" approach.
Specifically, we build into our surveys several
"trap" questions — questions whose
purpose is to uncover those participants who are not
giving us quality information. Typically, we'll include
five to seven of these, with each mistake constituting a
"strike." (Completing the survey too fast — i.e.
"the speeders" — is also counted as a strike.)
Those who reach three strikes are removed from
the dataset. For those with one or two strikes, we
review their pattern of responses on a case-by-case
basis to determine if there are any other oddities to
suggest removal.
For (much) more detail on this approach, check out
our earlier issue, "Who Let the Dogs In", from May
of 2007.
|
| |
|
|
|
|
We're delighted to announce that Mark will be
speaking at LIMRA's Group Benefits Leadership
Conference at Boston's Marriott Long Wharf
in
September.
Together with one of our clients, Mark will
discuss how our innovative approach to a compelling
research issue helped this client better understand
the
challenges that members of key sales and distribution
teams face.
|
|
If you haven't got anything nice to say about anybody, come sit next to me.
— Alice Roosevelt Longworth
|
Problems?
Click here
to send us an email with your request.
|
