Our 100th edition of Research with a Twist will be published within the next six months! As we consider various ways to celebrate this milestone, we’d love to hear whether you have a favorite edition, which one, and why. For those of you who respond to this request (via firstname.lastname@example.org), we will send you a CSR-branded “dog days of summer”-themed token of our appreciation.
Dogs are better than human beings, because they know but do not tell.
– Emily Dickinson
Questions? Click here to send us an email with your request.
If “dog days” are the final days of the summer season, does that mean we’re now in the puppy days of summer? Read this month’s edition of CSR’s newsletter, Research with a Twist, entitled “Lie Down with Dogs…” to hear our take on what dogs and market research have in common.
Lie Down with Dogs
Dogs. We love them.
Over the weekend, my family (me, my husband Todd, Roman, age 9 and Becca, age 12) just adopted a 10-week old Golden Labradoodle whom we’ve named “Mojo”. The puppy must have some kind of “mojo” for sure, because on the ride home from the breeder, he was sitting between my kids in the backseat, looking like he had never rested anywhere so comfortably in his life, totally at home.
I notice that my family and I tend to love the puppy most fanatically for his human-like qualities. The more he acts like a human, the more charmed we are – like when I came home from the grocery store the day after we got him, and found him and my son taking a nap together on the couch. Similarly, we found it unbearably “AAAWWW”-inspiring when one of CSR’s clients recently revealed to us that when he and his wife travel, they place their pup in Doggie Day Camp (complete with petcam, so the proud parents can check in and see junior during the day). If they sent a child to camp, it wouldn’t be nearly as intrinsically cute.
These kinds of behaviors are charming and cute because dogs AREN’T human. I’d never include Mojo in a market research study. For one thing, even though he consumes a lot (shoes, human food, MY IPAD!), he’s not the purchase decision- maker. And, let’s get real – his paws are already way too big to accurately use a keyboard to answer an online survey.
So, why are there so many “dogs” on online research panels these days? I ask myself this because I had a recent experience with online panels that would make your hair curl like a Doodle. As many of you have likely noticed, such stories seem to be common. Here’s some data to add to the industry-wide discussion that’s developed on this topic over the past several years, as well as an elegant solution to the problem:
Quality, quality, quality
With dogs, it all begins with quality. The reason we trekked to Maine, three hours away from home, for a type of dog that we could have gotten at a local pet store was that a reputable breeder lives there. When we arrived at the breeder’s place, we saw, “in person,” Mojo’s mom and dad (and seven siblings!). We have papers verifying his ancestry.
The same principle applies to research panels. Essentially, we don’t know the pedigree of our research participants when we use panels, and for that matter, when we do any online research. As our late great Research Director, Mark Palmerino, an ardent advocate of the benefits of in-depth interviewing, observed in our 2009 newsletter “Who let the dogs in?,” “Like a dog alone in a room with enticing food for the taking, survey-takers who answer questions on their own, without the involvement of an interviewer, will often behave differently (i.e., badly).”
Doggone it if I didn’t see that up close and personal this Spring! We were working on a study for an insurance industry client, and contracted with a panel to complete 400 surveys among benefits decision-makers across a range of company sizes (100 to 5,000 employees). We implemented our standard quality control processes for surveys (such as, participants are asked “duplicate” questions, a minimum time limit for completion is set, we check for straightlining, etc.). Participants exceeding a set QC “strike” limit were removed from the survey.
In the end, after a two-week deadline extension, demanded by the panel provider, and the addition of a second panel provider because the first couldn’t deliver on what they proposed to us, we got only 360 completes who passed our “strike” system! This was out of the over 600 completed surveys that the panel provider had sent us, assuring us that they had each completed the survey in a satisfactory manner. According to my paws, that means that about 40% of the people (if they were people) who completed the survey clearly were not paying attention to it. “Who let the dogs in?” is right!
Quality is at our fingertips
We are researchers (like our doggy research daddy Pavlov), so of course in this story there is a control group! A second sample source for the study was our client’s customer list.
With this group, our goal was to complete 200 surveys among the same audience, among the same range of company sizes (100 to 5,000 employees) – these participants were administered the same survey as the panelists, receiving the same incentive. For both groups, the sponsor of the survey was not known.
While the proportion of acceptable surveys from the providers was 60% (although according to the panel providers, it was 100%), the percentage of our client’s customers who passed our “strike” process was 98%! And keep in mind, also, that the panel providers claimed that they were cleaning the data before it got to us – we don’t know how many completed surveys they decided to remove from the data before sending it to us. Woof!
Based on this experience, and many more like it, a solution that we at CSR recommend to our clients is DIY – create your own panel, whether it’s customers, prospects, or a combination of the two. We’ve developed groups of willing research participants to branded and unbranded panels; in-person, phone, and online channels; and across diverse populations, including brokers, benefit administrators and decision-makers, analytics executives, and a wide variety of other executives, including C-Suite members.
The studies conducted using these custom panels are based on unquestionably high-quality research participants. And after all, what the heck are we doing in market research if we aren’t talking with the people who can help us improve our products, services and processes?
Until the panel quality issue has been resolved across the industry at large, we think that maintaining a custom panel is the best way to interact with target audiences on a regular basis. In this case, DIY means “Dogs Included Yesterday” – no more dogs in today’s research!
Here’s the Twist: Our experience with a recent study, where we found that 40% of the completed surveys had to be removed from the panel-provided portion of the study, while just 2% had to be removed from the portion of the data based on customer contact info (even though the study was NOT branded), led us to recall the old saw, “Lie down with dogs, and you get fleas.” Thus, we recommend making your own beds!
Mixology (Putting Research into Practice)
The following are three best practices CSR has developed in creating custom panels:
One great way to recruit is from an (accurately completed) online survey: Adding a question asking about interest in participating in an ongoing research initiative after the firmographics are collected can be useful in building a research panel. Gathering contact information and checking it during the recruiting process, either by phone or by using a separate online tool for verification, such as LinkedIn, guarantees high-quality participants on your custom panel.
Offer research interactions in which you would like to participate: One of the reasons that panel quality has deteriorated over the past several years is that surveys can be awfully dull and boring. Shortening or eliminating grid-style questions, limiting repetition, and/or adding a phone, video, or in-person element to the research mix can make the interactions much more engaging to participants.
Provide compensation, monetary or otherwise, throughout the engagement: We build several rewards into the process over a typical one- or multi-year initiative. While the largest form of compensation is given to the participant at the end of a year, we’ve learned that sending partial payments, as well as small tokens of appreciation like Starbucks gift cards and company-branded tzotchkes (as appropriate) throughout the panel’s life, keep participants interested and happy.
The Center for Strategy Research, Inc. (CSR) is a research firm. The “Twist” to what we offer is this: We combine open-ended questioning with our proprietary technology to create quantifiable data. As a result our clients gain more actionable and valuable insights from their research efforts.
Secondary Sidebar Widget Area
This is the Secondary Sidebar Widget Area. You can add content to this area by visiting your Widgets Panel and adding new widgets to this area.
The Center for Strategy Research, Inc. 101 Federal Street · Suite 1900 Boston, MA 02110