Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Mobile sample size: How much is enough?



Article ID:
20140725-1
Published:
July 2014
Author:
Jacob Tucker

Article Abstract

Analysis of mobile consumer research uncovers what using a smaller sample size means to a study's statistical significance.

Editor’s note: Jacob Tucker is the senior analyst of insights and strategy for uSamp.

Insights professionals have long followed the consumer research guideline that recommends a sample size ranging from 300 to 400 to help ensure a representative and statistically reliable sampling. In many instances, that’s a large number of participants to identify, screen and qualify. It's an especially large number when you're on a tight deadline or trying to target a hard-to-reach audience. Not to mention that larger study groups translate into higher costs. Suppose, however, that you could involve fewer participants in your mobile consumer research project and still achieve the same results. At uSamp we recently considered this question by reviewing data from previous studies.

Mobile research lists instant insights and quick turnarounds among its key strengths. A common challenge mobile researchers face, however, is harvesting enough completes to satisfy questions about how representative and reliable the results truly are. Given the weight of this challenge, we decided to pose a question of our own: Might a mobile-based study comprised of a large number of respondents produce the same research conclusions as one with far fewer?

Testing data metrics

To test the question, we revisited a range of research topics, various types of survey queries and an assortment of data metrics from mobile and online studies we have conducted in the past. Through our analysis of the data we hoped to understand at what sample size the response data essentially remained the same even as we enlarged the sample.

The following tables show the minimum number of completes at which the data stopped changing in significant ways for each question type.

Each table covers five distinct studies. We sorted the data according to varying point scales and question types. Some studies included multiple questions of the same type. The numbers shown for each study reflect the point at which we discerned no significant differences in the results with a larger sample size. For example, in Study 1 of Table A, a sample larger than 30 yielded the same results for a top two-box question rated on a five-point scale. So even though the survey netted 289 completes, for this question we could have in theory stopped at 30 and drawn the same conclusions. 

As a result, we observed no statistically meaningful changes in the data for samples exceeding approximately 75 participants. In many cases, it was far fewer. Overall, for the set of mobile-based studies we reviewed, the highest threshold number we encountered was 74, while others were considerably lower. Total sample sizes for these studies varied from 114 to 377. For the online category of studies, the thresholds were higher, but still much lower than the conventional recommendation for consumer research.

Conclusions on sample size

Considering that sample size has a dramatic effect on project timelines, there are some fascinating takeaways for the research community:

  • When running up against a tight deadline on a project with narrow demographic requirements, check the results you’ve collected up to that point against any incoming data and look for significant change. Depending on the project objectives, you may be able to close the study early if you have enough data to establish trends.
  • If your research objectives include comparison across sub-groups, sample sizes of 30 to 50 per subgroup may actually suffice for decision-making and analytical purposes.
  • If you’re working with a tight budget, conducting a study with fewer respondents can also pay off through reduced field time, costs and analysis labor (particularly in the case of rich media and open-ended coding).

It's important to note that our intention was not to overturn decades of research regarding statistical significance. It is not suggesting that a sample size of 75 is necessarily representative of the population. Rather, we hoped to give researchers confidence that when project specifications make obtaining large sample sizes difficult, they can still draw meaningful conclusions from the data. We believe these observations suggest interesting implications for researchers pursuing certain kinds of studies, including mobile-based research.

1The meaning of statistical significance in consumer research is a topic of lively debate. See, for example, What Every Researcher Should Know About Statistical Significance, DataStar, Inc. (GreenBook, The Guide for Buyers of Market Research)

 

 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 201: FUNDAMENTALS OF MODERATING
January 14 - 16, 2015
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on January 14 - 16 in Rockville, Md.
RIVA COURSE 201: FUNDAMENTALS OF MODERATING
January 26-28, 2014
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on January 26-28 in Rockville, Md.

View more Related Events...

Related Articles

There are 2313 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Trade Talk: Another round of belt tightening
A report on a study of how middle America is faring under the current economic conditions and which retailers are doing the best jobs of meeting their needs.
Research defines market for CheckFree
CheckFree Systems used tracking studies to determine market demand and develop marketing strategies in its efforts to expand beyond its initial success in the health club market.
Insurance company combines methods for Web site usability research
Using an insurance company that was researching adding a feature to its Web site as an example, the article shows how site usability research can have both a qualitative and a quantitative component.
The myths of ethnic marketing research
The author debunks four ethnic marketing research myths demonstrated by companies that market themselves nationally.
A focus group confession: "What's so bad about bean quantitative?"
Quantitative and qualitative research long have been incompatible. This article discusses and advocates the "Bean Test," a quantitative technique for use in focus groups.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Industries

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

TURF excel-based simulator
12/16/2014 by Joseph O. Fayese
Hi Giovanni
10/17/2014 by Dohyun Kim
request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer

View More