Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Mobile sample size: How much is enough?



Article ID:
20140725-1
Published:
July 2014
Author:
Jacob Tucker

Article Abstract

Analysis of mobile consumer research uncovers what using a smaller sample size means to a study's statistical significance.

Editor’s note: Jacob Tucker is the senior analyst of insights and strategy for uSamp.

Insights professionals have long followed the consumer research guideline that recommends a sample size ranging from 300 to 400 to help ensure a representative and statistically reliable sampling. In many instances, that’s a large number of participants to identify, screen and qualify. It's an especially large number when you're on a tight deadline or trying to target a hard-to-reach audience. Not to mention that larger study groups translate into higher costs. Suppose, however, that you could involve fewer participants in your mobile consumer research project and still achieve the same results. At uSamp we recently considered this question by reviewing data from previous studies.

Mobile research lists instant insights and quick turnarounds among its key strengths. A common challenge mobile researchers face, however, is harvesting enough completes to satisfy questions about how representative and reliable the results truly are. Given the weight of this challenge, we decided to pose a question of our own: Might a mobile-based study comprised of a large number of respondents produce the same research conclusions as one with far fewer?

Testing data metrics

To test the question, we revisited a range of research topics, various types of survey queries and an assortment of data metrics from mobile and online studies we have conducted in the past. Through our analysis of the data we hoped to understand at what sample size the response data essentially remained the same even as we enlarged the sample.

The following tables show the minimum number of completes at which the data stopped changing in significant ways for each question type.

Each table covers five distinct studies. We sorted the data according to varying point scales and question types. Some studies included multiple questions of the same type. The numbers shown for each study reflect the point at which we discerned no significant differences in the results with a larger sample size. For example, in Study 1 of Table A, a sample larger than 30 yielded the same results for a top two-box question rated on a five-point scale. So even though the survey netted 289 completes, for this question we could have in theory stopped at 30 and drawn the same conclusions. 

As a result, we observed no statistically meaningful changes in the data for samples exceeding approximately 75 participants. In many cases, it was far fewer. Overall, for the set of mobile-based studies we reviewed, the highest threshold number we encountered was 74, while others were considerably lower. Total sample sizes for these studies varied from 114 to 377. For the online category of studies, the thresholds were higher, but still much lower than the conventional recommendation for consumer research.

Conclusions on sample size

Considering that sample size has a dramatic effect on project timelines, there are some fascinating takeaways for the research community:

  • When running up against a tight deadline on a project with narrow demographic requirements, check the results you’ve collected up to that point against any incoming data and look for significant change. Depending on the project objectives, you may be able to close the study early if you have enough data to establish trends.
  • If your research objectives include comparison across sub-groups, sample sizes of 30 to 50 per subgroup may actually suffice for decision-making and analytical purposes.
  • If you’re working with a tight budget, conducting a study with fewer respondents can also pay off through reduced field time, costs and analysis labor (particularly in the case of rich media and open-ended coding).

It's important to note that our intention was not to overturn decades of research regarding statistical significance. It is not suggesting that a sample size of 75 is necessarily representative of the population. Rather, we hoped to give researchers confidence that when project specifications make obtaining large sample sizes difficult, they can still draw meaningful conclusions from the data. We believe these observations suggest interesting implications for researchers pursuing certain kinds of studies, including mobile-based research.

1The meaning of statistical significance in consumer research is a topic of lively debate. See, for example, What Every Researcher Should Know About Statistical Significance, DataStar, Inc. (GreenBook, The Guide for Buyers of Market Research)

 

 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

2015 PREDICTIVE ANALYTICS WORLD FOR BUSINESS
March 29-April 2, 2015
The 2015 Predictive Analytics World conference for business will be held on March 29-April 2 in San Francisco.
RIVA COURSE 303: ADVANCED MODERATING
May 4-6, 2015
RIVA Training Institute will hold a course, themed 'Advanced Moderating,' on May 4-6 in Rockville, Md.

View more Related Events...

Related Articles

There are 2363 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Hidden barriers to new product acceptance: preference inertia
Preference inertia, which prevents a target customer population from trying new things, can blunt sales by up to 40 percent. The factors behind it, and how to measure them, are the subjects of this article.
Is the market research industry failing its TQM clients?
Total quality management (TQM) and continuous quality improvement (CQI) emerged in the 1980s. Clients increasingly ask for more info more often with less budget, which begs the question, is the market research industry failing to meet clients' demands? This article describes the principles of TQM and CQI that researchers need to understand in order to win and keep quality-driven clients.
How to keep undesirable respondents out of your next online study
In research-on-research, executives from e-Rewards and Maritz Research detail their firms’ efforts to identify and control the effects of various types of undesirable online research respondents. A comparison of results from good and bad respondents is also analyzed.
Qualitative technique pits opposing groups to create insights
In the Rotated Opposed View method, a product or service’s users and non-users watch, listen and learn from each other to help observers develop more effective marketing strategies.
A Colorado HMO ties patient satisfaction to physician incentives
TakeCare HMO conducted a telephone satisfaction survey to gauge its primary-care physicians' performance in three crucial areas: access to care, physician care, and the office staff and environment. Rather than using a research firm to conduct these phone surveys, the HMO used specially trained employees.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Industries

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

TURF excel-based simulator
03/06/2015 by Nicky Turche
TURF excel-based simulator
12/16/2014 by Joseph O. Fayese
Hi Giovanni
10/17/2014 by Dohyun Kim
request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri

View More