Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Meta-analysis offers research on research for MR

Article ID:
May 2014
Kevin Gray

Article Abstract

Meta-analysis is a relatively new research-on-research tool that marketing researchers can use to examine a collection of results across multiple primary studies. This article covers the basics of meta-analysis and addresses some criticism toward it.

Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing science and analytics consultancy. This article appeared in the May 12, 2014, edition of Quirk's e-newsletter.


Meta-analysis is a set of procedures for statistically synthesizing  the results of several primary studies. It is research on research and is used when we want examine the body of evidence about a topic, rather than relying on the results of a single study. Even in rigorous disciplines such as medicine, studies often do not agree in their central conclusions and meta-analysis provides a systematic way to examine a collection of results, typically effect sizes, across multiple primary studies. It also sheds light on factors potentially underlying dissimilarities in findings and helps us make sense of the patterns of effects. It began to take root in the '90s and is applied in medicine, ecology, psychology, education, business and other fields as well as for the evaluation of government programs. It is still quite new to marketing research and arguably underutilized.


Estimate the true size


Meta-analysis is increasingly employed in place of the traditional narrative literature review, which has been criticized by its detractors for being subjective, unsystematic and as "arithmetic with words." Meta-analysis is appropriate, provided the primary studies are not too disparate. Since real effects cannot be reliably established through one study alone, meta-analysis helps us better estimate the true size of effects and is better suited when we wish to generalize to a broad population.


Effect sizes are typically measured by means, proportions, risk ratios, odds ratios or correlations. However, meta-analysis is not simply a matter of adding up effect sizes in each study and dividing by the number of studies we are examining, nor is it just counting the number of significant differences favoring one hypothesis over another and choosing a winner. Instead, it uses statistical techniques to combine the results of multiple studies. Ideally, data from individual subjects (individual participant data) in each study would be used but more typically this is not obtainable and aggregate data must be analyzed instead.


Weighted by precision


In meta-analysis, the individual studies are weighted by precision, which is derived from the standard errors of the individual studies. The term "inverse variance" is often used to describe this weighting mechanism, which is primarily a function of sample size. Larger studies (i.e., those with larger sample sizes) will be more influential in meta-analysis. It is not necessary that all primary studies in a meta-analysis measure effect in the same way and formulas exist to convert among effect sizes based on dissimilar metrics. There are other methods besides the inverse variance scheme. For details, readers can refer to standard references such as Introduction to Meta-Analysis (Borenstein et al.) and Modern Epidemiology (Rothman et al.).


Forest plots


Results of meta-analyses are normally summarized with forest plots, a handy graphical device that efficiently conveys the highlights of the analysis. (Forest stands for "forest of lines" and we are interested in both the trees and the forest.)


Figure 1 shows a simple example, taken from Wikipedia, based on five studies. In practice, sometimes dozens or even hundreds of primary studies are used. In the plot, the size of a square is proportional to the corresponding study's weight in the meta-analysis and the horizontal line represents the confidence interval of the study's odds ratio (OR), an effect measure that is standard output from logistic regression and equals 1.0 when there is no effect. The diamond at the bottom of the plot reports an overall odds ratio of 2.2, with the real effect size estimated to lie between 1.9 and 2.4.



Variations of meta-analysis


Fixed-effect versus random-effects


Statisticians will be familiar with fixed-effects and random-effects models. In meta-analysis, fixed-effect (singular form) has a different meaning and Borenstein et al. suggest "common-effect" might be a more suitable term. The fixed-effect model assumes the true effect is the same in all studies and that differences we observe among studies are the result of sampling error, while random-effects models allow that the true effect may vary from study to study.


When fixed-effect models are used, studies with larger sample sizes will receive relatively more weight and smaller studies relatively less weight, though this by itself should not determine whether a fixed-effect or random-effects model is chosen. Confidence intervals under random-effects models will be wider because both within and between (across studies) sources of variation are taken into account. Random-effects meta-analysis tends to be preferred because the primary studies as a rule are heterogeneous in participant (i.e., respondent) characteristics, in the implementation of treatments themselves or in other important operational aspects. It is generally, though not always, more realistic to assume that the true effect could vary from one study to the next. A new method, the quality-effects model, has been advocated as an alternative to random-effects models.




Regression is widely used in primary research and also has an important place in meta-analysis. Rather than subjects in a primary study, however, observations in the meta-regression data file are the results of the primary studies and both the dependent and independent variable(s) are at the study level, not the subject (i.e., respondent) level. As with standard regression, independent variables may be continuous, categorical or a combination of the two. Fixed- and random-effects models are both possible, as are variations of regression (e.g., polynomial). Analogous to regression in primary studies, the number of studies should be sufficiently large for the analysis to be meaningful and a minimum of 10 studies per independent variable is one rule of thumb, though this is not set in stone.


Psychometric meta-analysis


Any study has methodological faults that affect its results, sometimes substantially. Psychometric meta-analysis, also known as validity generalization or Hunter-Schmidt meta-analysis, attempts to adjust effect estimates for methodological limitations of the studies and perform a meta-analysis of the corrected results. The quality-effects model shares some of the aims of these methods.


Psychometric meta-analysis uses procedures based on psychometrics theory to make corrections for attenuation of effect size due to measurement error and other flaws. Effects will generally be larger and their variation smaller after corrections have been made and the distribution of true effects is emphasized, rather than the overall mean effect. Another distinction is that raw correlations are favored as the metric of effect and sample size is preferred for weighting.


Take into account


Researchers must take into account many factors when selecting primary studies. Obvious ones include the topic, the sample composition, how the sample was drawn and from what population and when and where the research was conducted. Less obvious, perhaps, are details of the study design (e.g., whether it was experimental or observational). (See my article, "Forget exact science: Drawing conclusions from observational research," in the July 22, 2013, edition of Quirk's e-newsletter, for an overview of experimental versus observational research.)  


Other considerations include which potential confounders were accounted for, how they were measured, the analytic methods employed and, of course, whether sufficient information regarding the study has been published or can be obtained. To a degree, heterogeneity is desirable since we will have more confidence that the findings of the meta-analysis are generalizable to real-world circumstances, though this is not to say anything goes.


Publication bias is another concern. This is a complex topic but essentially refers to the fact that many studies go unpublished, not because of poor quality, but because no statistically-significant effects were detected. Effects that achieve statistical significance will, in general, be larger than those that have not. Publication bias reflects editorial decisions but researchers must first decide on their own whether to submit results for publication. Statistically-significant findings are also more likely to be published quickly, to be published more than once, to be cited frequently and to be published in English and thus easier to locate. Meta-analysis cannot not magically make these potential biases disappear but statistical methods have been developed (e.g., funnel plots) to help assess whether they are a serious problem in the set of studies we have tentatively selected. In short, judgment plays a large part in study selection but it must be remembered that science is not merely plugging numbers into formulas in mechanical fashion. Whatever approach we take and regardless of our topic, it's garbage in, garbage out.


Somewhat contentious


Despite having advantages over alternatives, meta-analysis remains somewhat contentious. There are several reasons for this. Because the combined sample sizes can be very large, very small effects can be depicted as significant, even when effect sizes in all the individual studies are not statistically different from zero. Larger studies, moreover, are often operationally more challenging and can actually be more error-prone than ones that are smaller-scale and more carefully supervised. But larger studies will have more weight in meta-analysis because of their bigger sample size. Also, as noted, inclusion criteria for primary studies will have a major impact on the results and another critique is that meta-analysis can make it easier for the investigator to "cook" the results by cherry-picking "ingredients" that support a favored hypothesis. In other words, it can be used for advocacy instead of science.


Proponents of meta-analysis often respond to criticisms such as these by admitting that many of them do indeed have merit but that they apply even more to traditional narrative literature reviews or simply counting significant differences (vote-counting). In meta-analysis, it is claimed that these dangers are more transparent. My own view is that, while properly-conducted meta-analyses have important benefits, we do need to be vigilant against potential malpractice, perhaps more so than in the case of narrative literature reviews, which, in my experience, tend to be circumspect in their conclusions.


Who cares?


To this point, meta-analysis has been little-used in marketing research and some might wonder "Who cares? I'm not a doctor or a psychologist, so why should I incorporate these methods into mine?" My first response would be that, as researchers, we need to be attentive to what researchers in other fields are doing. After all, nearly all our methods and research protocols have been borrowed from other disciplines!


At a more day-to-day level, marketing researchers often undertake desk research as part of a project, especially at the design phase, and when doing so may occasionally encounter meta-analyses. It is therefore helpful to have some background on what it is and its strengths and limitations. Furthermore, we can conduct it ourselves when designing new research. Lastly, we can use meta-analysis for R&D on new methodologies, for examining questionnaire design issues, in ROI analysis, in pricing research and for testing product claims, to name just a few.


In a tiny nutshell


The foregoing is meta-analysis in a tiny nutshell. For those who'd like to study it in depth, there are numerous online sources and papers in addition to textbooks. Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Shadish et al.) is a widely-cited textbook that offers a comprehensive perspective on research, including meta-analysis, as well as detailed guidelines on research design and how it affects interpretation. The Cochrane Collaboration provides a very large repository of research on health care and health policy and is a good source for examples of meta-analysis. The Campbell Collaboration is a similar organization for social, behavioral and educational research.

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

August 30-31, 2016
Burke Institute will hold a seminar, themed 'Unlocking Consumer Insights: Neuroscience, Non-Conscious Measurement, Measurement of Emotion,' on August 30-31 in Chicago.
September 12-14, 2016
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on September 12-14 in Rockville, Md.

View more Related Events...

Related Articles

There are 2297 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

General Mills marketing research decides cookbook cover
"Betty Crocker's Cookbook" has sold over 22 million copies, but as the flagship of their publishing line, General Mills Marketing experts needed to figure out a cover that could keep the book selling strong. A variety of techniques were used to figure out what book cover would sell best.
JCPenney pinpoints its customers
In order to fully understand the needs of their customers, JCPenney has initiated a series of studies called Consumer Feedback. These studies give JCPenney a clear picture of the needs, attitudes and behaviors of their customers.
Rating scales can influence results
A summarized excerpt of a U.S. Department of Commerce study testing the merits of a seven-point rating scale versus a 10-point rating scale.
Quest research pays off for United Way
In the past, marketing research was too expensive for many United Way organizations. But all that has changed, thanks to a new research program called Quest. By utilizing innovative survey techniques and technology, Quest allows United Way organizations to improve communications, identify key services and improve fundraising easily and inexpensively.
Continuing analysis of shopping habits in San Diego
The "Continuing Analysis of Shopping Habits in San Diego," or CASH, is a clearinghouse for consumer information in San Diego County. The data is a service of The San Diego Union and The Tribune, a San Diego newspaper, and employs telephone interviews to collect extensive information on consumers throughout the area.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries



Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

Confidence interval Definition
03/03/2016 by Alex Hales
TURF Simulator with Shapley Value
02/10/2016 by Amit Zaveri
the best graduate degree for quantitative marketing research
10/31/2015 by Alex Hales
About Healthcare Research & Reports
10/28/2015 by Alex Hales
Compared to what?
09/24/2015 by Alex Hales

View More