Improving web panel respondent behaviour: The effect of encouragement messages throughout the course of the survey


Description:Respondent motivation during the survey response process can differ across individuals and situations, leading to suboptimal responding behaviours. Survey methodologists have been searching for strategies to identify and reduce this kind of behaviour among online respondents. Motivating, praising or telling respondents off might increase the willingness to answers, however, if these answers are given without appropriate thinking, this can lead to a decrease of the data quality. Past research has found little evidence of an association between displaying motivational messages and a reduction of break-offs or an increase of data quality. However, more approaches have to be tested This paper analyses an experiment conducted in waves 2, 4 and 6 of the CROss- National Online Survey (CRONOS) panel in Estonia, Slovenia and Great Britain. Thus, respondents were randomly assigned into a control group and two treatment groups in Wave 2 and were kept in the same group for the remaining two waves. Four to five other messages (depending on wave) were placed throughout the questionnaire to test the effect of introducing more than one message across the survey. Moreover, messages emphasized the positive or the negative consequences of their behaviours depending on the group, to compare between both approaches. In order to establish the impact on data quality, we compare control and treatment groups in terms of break-offs, item nonresponse, non-differentiation, completion time, efforts reported and evaluation of the survey experience. Our results suggest that motivational messages have little impact on data quality. Moreover, positive and negative messages have different impacts depending on the country. Thus, strategies to motivate respondents should be country-specific. Moreover, significant impacts, positive or negative, on data quality were mainly found on the first sets early in the questionnaire. Our results point that using standard motivational messages to all respondents is not the correct approach.

Download the presentation