Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Mobile sample size: How much is enough?



Article ID:
20140725-1
Published:
July 2014
Author:
Jacob Tucker

Article Abstract

Analysis of mobile consumer research uncovers what using a smaller sample size means to a study's statistical significance.

Editor’s note: Jacob Tucker is the senior analyst of insights and strategy for uSamp.

Insights professionals have long followed the consumer research guideline that recommends a sample size ranging from 300 to 400 to help ensure a representative and statistically reliable sampling. In many instances, that’s a large number of participants to identify, screen and qualify. It's an especially large number when you're on a tight deadline or trying to target a hard-to-reach audience. Not to mention that larger study groups translate into higher costs. Suppose, however, that you could involve fewer participants in your mobile consumer research project and still achieve the same results. At uSamp we recently considered this question by reviewing data from previous studies.

Mobile research lists instant insights and quick turnarounds among its key strengths. A common challenge mobile researchers face, however, is harvesting enough completes to satisfy questions about how representative and reliable the results truly are. Given the weight of this challenge, we decided to pose a question of our own: Might a mobile-based study comprised of a large number of respondents produce the same research conclusions as one with far fewer?

Testing data metrics

To test the question, we revisited a range of research topics, various types of survey queries and an assortment of data metrics from mobile and online studies we have conducted in the past. Through our analysis of the data we hoped to understand at what sample size the response data essentially remained the same even as we enlarged the sample.

The following tables show the minimum number of completes at which the data stopped changing in significant ways for each question type.

Each table covers five distinct studies. We sorted the data according to varying point scales and question types. Some studies included multiple questions of the same type. The numbers shown for each study reflect the point at which we discerned no significant differences in the results with a larger sample size. For example, in Study 1 of Table A, a sample larger than 30 yielded the same results for a top two-box question rated on a five-point scale. So even though the survey netted 289 completes, for this question we could have in theory stopped at 30 and drawn the same conclusions. 

As a result, we observed no statistically meaningful changes in the data for samples exceeding approximately 75 participants. In many cases, it was far fewer. Overall, for the set of mobile-based studies we reviewed, the highest threshold number we encountered was 74, while others were considerably lower. Total sample sizes for these studies varied from 114 to 377. For the online category of studies, the thresholds were higher, but still much lower than the conventional recommendation for consumer research.

Conclusions on sample size

Considering that sample size has a dramatic effect on project timelines, there are some fascinating takeaways for the research community:

  • When running up against a tight deadline on a project with narrow demographic requirements, check the results you’ve collected up to that point against any incoming data and look for significant change. Depending on the project objectives, you may be able to close the study early if you have enough data to establish trends.
  • If your research objectives include comparison across sub-groups, sample sizes of 30 to 50 per subgroup may actually suffice for decision-making and analytical purposes.
  • If you’re working with a tight budget, conducting a study with fewer respondents can also pay off through reduced field time, costs and analysis labor (particularly in the case of rich media and open-ended coding).

It's important to note that our intention was not to overturn decades of research regarding statistical significance. It is not suggesting that a sample size of 75 is necessarily representative of the population. Rather, we hoped to give researchers confidence that when project specifications make obtaining large sample sizes difficult, they can still draw meaningful conclusions from the data. We believe these observations suggest interesting implications for researchers pursuing certain kinds of studies, including mobile-based research.

1The meaning of statistical significance in consumer research is a topic of lively debate. See, for example, What Every Researcher Should Know About Statistical Significance, DataStar, Inc. (GreenBook, The Guide for Buyers of Market Research)

 

 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

MRIA 2015 NATIONAL CONFERENCE- STORYTELLING: FROM INSIGHTS TO IMPACT
May 24-26, 2015
The Marketing Research and Intelligence Association will hold its annual conference, themed 'Storytelling: From Insights to Impact' on May 24-26 in Toronto, Ontario.
IS YOUR RESEARCH STUCK IN THE PHONE AGE?
May 27 at 1 p.m. CDT, 2015
Research Now will be sponsoring a one hour Webinar titled, ‘Is your research stuck in the phone age?’ on May 27 at 1 p.m. CDT.

View more Related Events...

Related Articles

There are 2392 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

5 marketing trends and how MR must respond
Insights in Marketing LLC conducted research on research to create a list of potential MR trends for 2014. The author outlines five predicted industry trends and discusses how researchers can make the most of them.
Trade Talk: Holding up a mirror
Drawing from a Q&A piece asking researchers to address questions about MR in 2014 and what's ahead in 2015, Quirk's Editor Joseph Rydholm provides a sampling of what researchers said is being talked about too much or not enough.
From the Publisher October 1986
Publisher Tom Quirk offers a welcome to the first issue of Quirk's Marketing Research Review magazine.
Qualities of a Master Moderator
A key element in focus group success is the moderator. This article focuses on qualities of “Master Moderators”—those who have mastered the key skills and techniques that lead to effective group interactions.
Trade Talk: Online pharma MR course gets an update
An interview with Don Marek, executive director of the Marketing Research Institute International, detailing the changes to an online pharma market research course.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Industries

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

TURF excel-based simulator
03/06/2015 by Nicky Turche
TURF excel-based simulator
12/16/2014 by Joseph O. Fayese
Hi Giovanni
10/17/2014 by Dohyun Kim
request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri

View More