We all receive waaaay too many emails and invitations to poorly-designed and poorly-executed questionnaires. This isn’t acceptable. Those who do research studies using questionnaires have a responsibility to those who fund, take, and learn from our surveys to do a good job.
In the interest of open research, here are a few of the techniques I’m using to respect the time and attention of my survey-takers in a large-scale online questionnaire. What do you do? Conversation welcome.
Background: Each month for the next three years I’ll be inviting recent corresponding authors from several dozen journals to take an online questionnaire as part of a research study on the impact of journal data archiving policies.
I started by sending emails to the journal editors, letting them know that I’d be inviting their corresponding authors to take part in this survey. To my happy surprise, all responses were positive: One editor asked me to clarify in my invitation that the study was not endorsed by the journal, another asked for more details, and many thanked me for letting them know and said they looked forward to hearing the results.
To create my contact list, I gather corresponding author email addresses from ISI Web of Science. Turns out it sometimes takes ISI 8 weeks to pull in new journal issues, but the data export is relatively fast and clean.
Using some custom python code, I filter the email addresses to those for just the desired month(s) and eliminate those to which I’ve already sent the survey invitation, to reduce survey fatigue. I send one initial invitation email and one reminder email about a week later. Unfortunately, the reminder email is sent to many people who’ve already taken the survey… a necessary byproduct of anonymous surveys. The initial email includes an option to unsubscribe. Less than 1% of email recipients have unsubscribed. Those who unsubscribe are automatically eliminated from the reminder email.
I’m keeping track of how long it takes respondents to complete the questionnaire. Pilot tests suggested 7-10 minutes so that is what my invitation says, but it is now looking more like 7-14 minutes, so I’ll rephrase my invitation and FAQs to be less misleading.
I plan to compare publicly-available information for the whole corresponding author population with responses to demographic questions in my sample to understand who the sample population is and isn’t. Because the response rate is pretty low for this sort of online survey (so far about 15%), this step is really important…. just knowing the demographics of the respondents without knowing how they compare to the larger sample can make for misleading generalizations.
Other than carefully designing the questionnaire to be as short, straight-forward, and on-topic as possible, those have been my main steps for respecting the time of my research subjects. Anyone have other tips and best practices?