Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Mobile sample size: How much is enough?



Article ID:
20140725-1
Published:
July 2014
Author:
Jacob Tucker

Article Abstract

Analysis of mobile consumer research uncovers what using a smaller sample size means to a study's statistical significance.

Editor’s note: Jacob Tucker is the senior analyst of insights and strategy for uSamp.

Insights professionals have long followed the consumer research guideline that recommends a sample size ranging from 300 to 400 to help ensure a representative and statistically reliable sampling. In many instances, that’s a large number of participants to identify, screen and qualify. It's an especially large number when you're on a tight deadline or trying to target a hard-to-reach audience. Not to mention that larger study groups translate into higher costs. Suppose, however, that you could involve fewer participants in your mobile consumer research project and still achieve the same results. At uSamp we recently considered this question by reviewing data from previous studies.

Mobile research lists instant insights and quick turnarounds among its key strengths. A common challenge mobile researchers face, however, is harvesting enough completes to satisfy questions about how representative and reliable the results truly are. Given the weight of this challenge, we decided to pose a question of our own: Might a mobile-based study comprised of a large number of respondents produce the same research conclusions as one with far fewer?

Testing data metrics

To test the question, we revisited a range of research topics, various types of survey queries and an assortment of data metrics from mobile and online studies we have conducted in the past. Through our analysis of the data we hoped to understand at what sample size the response data essentially remained the same even as we enlarged the sample.

The following tables show the minimum number of completes at which the data stopped changing in significant ways for each question type.

Each table covers five distinct studies. We sorted the data according to varying point scales and question types. Some studies included multiple questions of the same type. The numbers shown for each study reflect the point at which we discerned no significant differences in the results with a larger sample size. For example, in Study 1 of Table A, a sample larger than 30 yielded the same results for a top two-box question rated on a five-point scale. So even though the survey netted 289 completes, for this question we could have in theory stopped at 30 and drawn the same conclusions. 

As a result, we observed no statistically meaningful changes in the data for samples exceeding approximately 75 participants. In many cases, it was far fewer. Overall, for the set of mobile-based studies we reviewed, the highest threshold number we encountered was 74, while others were considerably lower. Total sample sizes for these studies varied from 114 to 377. For the online category of studies, the thresholds were higher, but still much lower than the conventional recommendation for consumer research.

Conclusions on sample size

Considering that sample size has a dramatic effect on project timelines, there are some fascinating takeaways for the research community:

  • When running up against a tight deadline on a project with narrow demographic requirements, check the results you’ve collected up to that point against any incoming data and look for significant change. Depending on the project objectives, you may be able to close the study early if you have enough data to establish trends.
  • If your research objectives include comparison across sub-groups, sample sizes of 30 to 50 per subgroup may actually suffice for decision-making and analytical purposes.
  • If you’re working with a tight budget, conducting a study with fewer respondents can also pay off through reduced field time, costs and analysis labor (particularly in the case of rich media and open-ended coding).

It's important to note that our intention was not to overturn decades of research regarding statistical significance. It is not suggesting that a sample size of 75 is necessarily representative of the population. Rather, we hoped to give researchers confidence that when project specifications make obtaining large sample sizes difficult, they can still draw meaningful conclusions from the data. We believe these observations suggest interesting implications for researchers pursuing certain kinds of studies, including mobile-based research.

1The meaning of statistical significance in consumer research is a topic of lively debate. See, for example, What Every Researcher Should Know About Statistical Significance, DataStar, Inc. (GreenBook, The Guide for Buyers of Market Research)

 

 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

6TH CONFERENCE OF THE EUROPEAN SURVEY RESEARCH ASSOCIATION
July 13-17, 2015
ESRA will hold its 6th Conference of the European Survey Research Association on July 13-17 in Reykjavik, Iceland.
SENTIMENT ANALYSIS SYMPOSIUM
July 15-16, 2015
Alta Plana will hold an event, themed 'Sentiment Analysis Symposium,' on July 15-16 at The New York Academy of Sciences in New York.

View more Related Events...

Related Articles

There are 2411 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

The focus group bill of rights
Focus groups involve multiple players: client, moderator, facility and respondent. This article lists the rights of each of these roles when participating in a focus group.
New strategies are needed to restart energy efficiency efforts
The author explores research that uncovered reasons why energy efficiency efforts have stalled and offers strategies to adapt them to consumers' mind-sets and economic situations.
Trade Talk: Up close and personal
Quirk's Editor Joseph Rydholm interviews researcher Allison Cohen to understand the intimacy benefits of face-to-face, on-site and one-on-one research techniques.
Focus group gripes
Focus groups can have their challenges for different people involved in the research. This article shares the thoughts of a marketing research director, a focus group facility owner and a moderator.
Using morphological content analysis to mine insights from qualitative interviews
The author details the business-to-business application of morphological content analysis, which provides a systematic approach to analyzing interview transcripts. An example in the consumer electronics industry is provided to illustrate the main concepts.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Industries

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

TURF excel-based simulator
03/06/2015 by Nicky Turche
TURF excel-based simulator
12/16/2014 by Joseph O. Fayese
Hi Giovanni
10/17/2014 by Dohyun Kim
request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri

View More