There was a time when collecting quantitative data, more often than not, involved a conversation. And the conversation was a game where the respondent seemed to hold all of the cards. A lengthy interview, a poorly worded question, a missing response option – all represented excuses for the respondent to dismiss you by hanging up or otherwise ending the interview.
Can you tell I was an interviewer?
At that time, phone pretests often involved a painful process of listening to hours of rejections and hang-ups…but the result was invariably worth the price – a survey that more accurately collected the intended data.Additionally, pretests allowed us to identify production-related issues such as lower-than-expected incidences and survey issues which would cause respondents to hang up.
Today, with so much of our research being conducted quickly and efficiently through carefully managed online panels, many of the production-related issues have become less prevalent.
Unfortunately, it seems that as these issues have faded, so has the use of survey pretesting. Which raises questions…in the age of online research, has the ease with which we are able to field surveys caused us to become complacent with our survey design? Has pretesting become a lost art? I would argue that it has, while the need for survey pretesting is actually greater than ever.
One of the key advantages of one-on-one interaction, whether over the phone, in a living room or at an office, is that it allows the respondent and the interviewer to identify problems that exist with the survey design. Essentially, every interview represents an opportunity to recognize and fix design issues.
But with the currently prevalent paradigm of computer–to-person interviewing, to a large extent, this feedback loop no longer exists. Furthermore, the shift to computer-to-person interviewing has created an environment where surveys are expected to field very quickly – with shorter schedules and additional time pressures.
So, when running your next online panel survey, consider allowing us to build a formal pretest into your study schedule to make sure data is being collected in the manner intended. Yes, it may add a few days to your overall study schedule, but the quality improvement will be worth the time and effort.
This pretest can be conducted by recruiting qualified participants to complete the online interview at a central location (e.g. the researcher’s office), while you monitor remotely. While we always pretest our surveys with internal employees and with a small number of panelists, a more formal pretest allows you, the client, to get involved and provide your in-depth professional perspective.
MSI will be mindful of the sampling frame being used for your study, and be sure to recruit a “good mix” of pretest respondents. Each type of respondent brings a different set of perspectives to the survey, and you need to be sure you are capturing them in your pretest, just as you will in your final results.
MSI will offer an incentive to the individuals participating in the pretest, turning the exercise into a pleasant and productive win-win situation – a far cry from the long nights of rejections and hang-ups…
Using this pretesting approach will:
- Improve survey execution by confirming that respondents are able to complete the questions and tasks in the survey
- Improve comprehension by ensuring that respondents understand the context of the survey questions
- Improve response categories by checking to make sure we are capturing the full range of behaviors and/or attitudes expressed by the respondents
- Improve the respondents’ satisfaction and engagement with the survey, leading to better quality data and insights
So the next time we recommend a pretest, let’s discuss the ways it can help you maximize the return on your research budget. We are certain you will find the pretesting effort to be time and money well-spent!