Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Mobile sample size: How much is enough?



Article ID:
20140725-1
Published:
July 2014
Author:
Jacob Tucker

Article Abstract

Analysis of mobile consumer research uncovers what using a smaller sample size means to a study's statistical significance.

Editor’s note: Jacob Tucker is the senior analyst of insights and strategy for uSamp.

Insights professionals have long followed the consumer research guideline that recommends a sample size ranging from 300 to 400 to help ensure a representative and statistically reliable sampling. In many instances, that’s a large number of participants to identify, screen and qualify. It's an especially large number when you're on a tight deadline or trying to target a hard-to-reach audience. Not to mention that larger study groups translate into higher costs. Suppose, however, that you could involve fewer participants in your mobile consumer research project and still achieve the same results. At uSamp we recently considered this question by reviewing data from previous studies.

Mobile research lists instant insights and quick turnarounds among its key strengths. A common challenge mobile researchers face, however, is harvesting enough completes to satisfy questions about how representative and reliable the results truly are. Given the weight of this challenge, we decided to pose a question of our own: Might a mobile-based study comprised of a large number of respondents produce the same research conclusions as one with far fewer?

Testing data metrics

To test the question, we revisited a range of research topics, various types of survey queries and an assortment of data metrics from mobile and online studies we have conducted in the past. Through our analysis of the data we hoped to understand at what sample size the response data essentially remained the same even as we enlarged the sample.

The following tables show the minimum number of completes at which the data stopped changing in significant ways for each question type.

Each table covers five distinct studies. We sorted the data according to varying point scales and question types. Some studies included multiple questions of the same type. The numbers shown for each study reflect the point at which we discerned no significant differences in the results with a larger sample size. For example, in Study 1 of Table A, a sample larger than 30 yielded the same results for a top two-box question rated on a five-point scale. So even though the survey netted 289 completes, for this question we could have in theory stopped at 30 and drawn the same conclusions. 

As a result, we observed no statistically meaningful changes in the data for samples exceeding approximately 75 participants. In many cases, it was far fewer. Overall, for the set of mobile-based studies we reviewed, the highest threshold number we encountered was 74, while others were considerably lower. Total sample sizes for these studies varied from 114 to 377. For the online category of studies, the thresholds were higher, but still much lower than the conventional recommendation for consumer research.

Conclusions on sample size

Considering that sample size has a dramatic effect on project timelines, there are some fascinating takeaways for the research community:

  • When running up against a tight deadline on a project with narrow demographic requirements, check the results you’ve collected up to that point against any incoming data and look for significant change. Depending on the project objectives, you may be able to close the study early if you have enough data to establish trends.
  • If your research objectives include comparison across sub-groups, sample sizes of 30 to 50 per subgroup may actually suffice for decision-making and analytical purposes.
  • If you’re working with a tight budget, conducting a study with fewer respondents can also pay off through reduced field time, costs and analysis labor (particularly in the case of rich media and open-ended coding).

It's important to note that our intention was not to overturn decades of research regarding statistical significance. It is not suggesting that a sample size of 75 is necessarily representative of the population. Rather, we hoped to give researchers confidence that when project specifications make obtaining large sample sizes difficult, they can still draw meaningful conclusions from the data. We believe these observations suggest interesting implications for researchers pursuing certain kinds of studies, including mobile-based research.

1The meaning of statistical significance in consumer research is a topic of lively debate. See, for example, What Every Researcher Should Know About Statistical Significance, DataStar, Inc. (GreenBook, The Guide for Buyers of Market Research)

 

 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

NETWORKING EVENT BY THE RESEARCH CLUB
September 24th, 2014
The Research Club will host a networking event in conjunction with the MRMW conference on September 24th at the Riva Bar in Berlin, Germany.
RIVA COURSE 201: FUNDAMENTALS OF MODERATING
Sept. 29 - Oct. 1, 2014
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on Sept. 29 - Oct. 1 in Rockville, Md.

View more Related Events...

Related Articles

There are 2259 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Hidden barriers to new product acceptance: preference inertia
Preference inertia, which prevents a target customer population from trying new things, can blunt sales by up to 40 percent. The factors behind it, and how to measure them, are the subjects of this article.
Using recognition-based tracking to compare the ROI of print, radio and TV
Ad campaigns often include a variety of media, which often leads to questions about effectiveness. This article is a case history from a continuing recognition-based tracking study by BlueCross BlueShield of Minnesota that shows a way to answer all the questions that arise regarding which medium is best.
Analysis of business-to-business survey database yields insights on creating competitive advantage
Priority Metrics Group compiled customer survey results into a proprietary database, the results of which were delineated into 17 categories and used to better understand the process of creating competitive advantage and the role of customer surveys in providing critical data necessary to achieving that understanding.
Hamilton Beach talks to consumers before introducing small appliance lines in Mexico and Europe
Hamilton Beach used focus groups in Europe and both qualitative and quantitative research in Mexico before expanding into these markets.
A guide to using PowerPoint to present research findings
To avoid the dreaded death-by-PowerPoint, presenters of research findings can follow these several helpful tips and guidelines.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Industries

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer
TURF excel-based simulator
03/25/2014 by Werner Mueller
I would like Turf Macro too!
03/06/2014 by Neelam Hinduja

View More