Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing science and analytics consultancy. This article appeared in the May 12, 2014, edition of Quirk's e-newsletter.

 

Meta-analysis is a set of procedures for statistically synthesizing  the results of several primary studies. It is research on research and is used when we want examine the body of evidence about a topic, rather than relying on the results of a single study. Even in rigorous disciplines such as medicine, studies often do not agree in their central conclusions and meta-analysis provides a systematic way to examine a collection of results, typically effect sizes, across multiple primary studies. It also sheds light on factors potentially underlying dissimilarities in findings and helps us make sense of the patterns of effects. It began to take root in the '90s and is applied in medicine, ecology, psychology, education, business and other fields as well as for the evaluation of government programs. It is still quite new to marketing research and arguably underutilized.

 

 

Meta-analysis is increasingly employed in place of the traditional narrative literature review, which has been criticized by its detractors for being subjective, unsystematic and as "arithmetic with words." Meta-analysis is appropriate, provided the primary studies are not too disparate. Since real effects cannot be reliably established through one study alone, meta-analysis helps us better estimate the true size of effects and is better suited when we wish to generalize to a broad population.

 

Effect sizes are typically measured by means, proportions, risk ratios, odds ratios or correlations. However, meta-analysis is not simply a matter of adding up effect sizes in each study and dividing by the number of studies we are examining, nor is it just counting the number of significant differences favoring one hypothesis over anothe...