The Tortoise Wins Again. Faster Data Collection Is Not Always Better.

At the risk of sounding like my grandfather, I’ve seen a lot of change in my career in terms of the speed at which we can deliver insights. Back in the day, we used to send tabs and reports to clients via snail mail. No, we didn’t use the Pony Express, but compared to today’s technology standards, it seems as though we weren’t far from that. ☺

I know I’m stating the obvious when I say that the advent of online interviewing brought a dramatic change in the speed at which we can collect data and deliver insights. And with lightning-fast capabilities sometimes comes the perception that we should collect vast amounts of data as fast as we can, maybe in just a couple of days at most. I understand that perception. And I understand the need for speed to insight in today’s world. But is faster always better?
The answer is a resounding “NO.” In fact, the results could be disastrous if you need a representative sampling of your population of interest (which I’m willing to bet is most of the time). Look at what happens when, after just 3 days of interviewing, we plot the age distribution of our respondents against the target population distribution for 7 consumer studies we recently conducted.

In all 7 cases, in 3 days’ time, we’ve under-sampled younger consumers and over-sampled older consumers. In fact, for Survey 7 we captured 20% more than the population target of the oldest age category. To rectify this we would need large weighting factors to bring our sample back in line with the general population. Or, we could continue to interview for one additional week and see the sampling distribution even out to reflect the natural population distribution more closely. Weighting would still be required in some cases, but the weighting factors would be much lower.

We have observed this same phenomenon on other variables beyond just age. For example, it has also been our experience that women tend to respond to online survey invitations more quickly than men.

At MSI, we have developed a stratification plan to insure consumers are screened proportional to their prevalence in the population. We do this because we understand that all online samples have the potential for some skews (i.e., age, employment status, etc.). By controlling the sampling frame via our stratification plan, we insure a representative sample with minimal weighting.

And as shown above, it takes a little bit of time for the plan to unfold since response urgency differs by respondent type.

So, while the pressures of today’s need for speed to insight are high, the risks associated with fielding too quickly are higher. Another way to look at it is that an additional week allotted for data collection is a whole lot better than having to refield a study to get a more representative sample.

Leave a Reply

Your email address will not be published. Required fields are marked *