Editor's note: Pete Cape is global knowledge director at SSI, a Shelton, Conn., research company. He can be reached at email@example.com. This article appeared in the August 27, 2013, edition of Quirk's e-newsletter.
Today, survey respondents are in control. They choose not only if or when to take a survey but also how to take it. An online survey-taker may choose to take a survey on a tablet while riding on a train today, on a smartphone while waiting in a queue at the bank tomorrow and then on a PC or laptop at home over the weekend. For multiwave or diary studies, a respondent may take parts one, two and four on a mobile device and parts three and five on a PC. To maintain a broad frame that is as representative as possible of our target populations, we must include these on-the-go survey-takers and offer the right survey design to fit their preference and lifestyle.
Always discuss device self-selection with your research and/or sample provider and consider how mobile-friendly your questionnaire is or will be. Once this is understood, you can take appropriate action to ensure you are reaching your target population appropriately and providing the right survey-taking experience for every segment.
SSI has identified four categories of questionnaire mobile-readiness: mobile-optimized, mobile-friendly, mobile-unfriendly and mobile-incompatible. Each has its own characteristics and requires a customized sample plan and an understanding of the study's cost and timing implications.
A mobile-optimized survey has been:
designed especially for a mobile device;
- scaled up for larger screen sizes; and
- tested on mobile devices.
We allow all our respondents to access mobile-optimized surveys from their device of choice. There are likely no sample or cost implications for our customers when surveys are mobile optimized.
A mobile-friendly survey contains:
nothing that would prevent it rendering properly on a mobile device;
- no long or wide scales; and
- no long brand (or similar) lists.
As with mobile-optimized questionnaires, we allow respondents to access mobile-friendly surveys from their device of choice but monitor dropout by device to ensure the questionnaire has no hidden "unfriendly" parts.
There could be sampling implications if mobile access needs to be switched off at any point, as well as project cost implications if many respondents cannot complete the study.
A mobile-unfriendly survey contains such things as:
- long answer lists; and
- long or wide scales.
These items will not render well on a mobile device, leading to respondent fatigue, frustration and higher a dropout rate. We prevent respondents from accessing these surveys on a mobile device but will allow respondents on tablets to take the survey.
There are some real consequences for the researcher here. The sample is now biased away from mobile-only and mobile-preferred respondents. This bias may get worse over time. Feasibility is impacted and therefore there could be cost implications for the project.
The survey has features such as:
- elements written in an unsupported language (e.g., Flash);
- requirement to see images in reasonable size; and
- items to be compared on one screen (e.g., conjoint studies).
We prevent mobile device and tablet respondents from accessing these surveys. Again, the sample is now biased away from mobile-/tablet-only and mobile-/tablet-preferred respondents. Feasibility could be significantly impacted, depending on the target population, and there will likely be project cost implications.
Discussing these issues in the early project-planning phase (i.e., before a questionnaire is programmed or even designed) will prevent unexpected cost and feasibility issues later.
Up the mobile-friendly spectrum
Very often, the goal is to make an incompatible or unfriendly questionnaire into one that is compatible. If you want to move a questionnaire up the mobile-friendly spectrum, consider the following adjustments.
1. Present grids as individual questions.
This will likely result in the questionnaire being longer than it was previously in a PC environment, impacting project cost and feasibility.
2. Scales longer than 11 points should be redesigned with broader ranges to reduce the number of points.
The resulting data will be less granular. Scales too wide for a small phone screen can be converted from horizontal to vertical design. This change should also be made across all versions of the questionnaire to prevent differences arising from processing effects in responding to the scale.
3. Answer lists longer than 12 items should be broken into separate questions.
Again, this change should be made across all versions of the questionnaire.
4. Questions with comparisons between side-by-side graphics must be redesigned.
Creative options can be used, such as showing thumbnail-size graphics with zoom-in options.
5. Conjoint exercises must be redesigned.
Researchers often have questions about how to incorporate mobile users in their sample. These are some commonly-asked questions regarding mobile readiness.
Can you help me get my survey from mobile-unfriendly to mobile-friendly?
Yes. The vast majority of surveys can be transformed successfully. Very often, the changes required are minor compared to the risk of excluding mobile survey-takers from your sample frame.
I know my survey is mobile-unfriendly but I really need the full sample coverage and I don't want to change my survey design. Can we field the questionnaire as-is?
We strongly advise against this. It is bad for the data and bad for the respondent. Dropout rates will likely be quite high.
Can you force the respondents to use their desktop or laptop PCs for my survey?
The shift away from desktop and laptop computers toward tablets and mobile devices is a social change; it is not something we can control. To maintain a representative sample frame as communication habits change, we need to allow respondents to access surveys through their device of choice.
Questionnaire consistency is more important to me than the possibility of sample biases so I cannot change my survey design. Can you guarantee that my data will remain unchanged going forward?
It's impossible to make that guarantee. The sample biases may not become apparent for many years but the effects of the small screen on a mobile device and how the survey renders will become apparent very soon. Now would be a good time to understand those effects and make any changes necessary to continue to get consistent data.
Improve the breadth and quality
Because the future potential of mobile technology for new research techniques and in-the-moment opinion-gathering is so captivating, basic knowledge about best practices for mobile survey-taking today has been little discussed. This is unfortunate. By using the right recruitment and engagement techniques for mobile survey-takers and providing them with a good survey-taking experience by making some relatively simple adjustments to questionnaires, we can greatly improve the breadth and quality of our sample frame and achieve better data today.