Editor's note: Philip Derham is director, Derham Marketing Research Pty. Ltd., Melbourne, Australia.
Matrix questions are commonly used in online surveys when several statements, products or services are to be measured on the same scale. For visual economy and for ease in answering, the statement and scale questions are shown as one question. This format has led some researchers to require statement lists in matrix questions to be rotated, to avoid a hypothesized list-order bias. The list-order bias concern is that online survey participants may be more likely to answer the first listed statements and be less likely to answer statements lower on the list.
If the list-order bias hypothesis were valid, it would lessen the information collected, when answering each statement is not compulsory for participants to be able to advance to the next question or screen.
A solution available in online survey software is to rotate the list order, so that all statements on a list are equally advantaged or disadvantaged. This practice assumes the list-order bias hypothesis is valid.
Results from a recent online survey encouraged us to review the list-order bias hypothesis and this article summarizes recent online survey findings about the list-order bias. The initial findings (with caveats) suggest that there is not a list-order bias. Others may wish to review their findings to see if this initial study’s conclusions hold more widely.
The case studies discussed in this article are from online surveys with business, shopping center store and bank customers, recently undertaken by Derham Marketing Research in Australia. As a matter of practice, participants in our online surveys can move from screen to screen without being required to give answers to each question. We use skips, where needed for sequence logic, if a question or statement is not answered.
This practice replicates the freedom to answ...