Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Forget exact science: Drawing conclusions from observational research



Article ID:
20130726-2
Published:
July 2013
Author:
Kevin Gray

Article Abstract

While most marketing research is observational in nature, we also conduct experimental research. Each has important advantages and disadvantages that are frequently overlooked and this article addresses them.

Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing science and analytics consultancy. This article appeared in the July 22, 2013, edition of Quirk's e-newsletter.  

Say, for example, we observe that some beer brands are much more popular in certain parts of the country than in others. Or perhaps we find that some fashion brands are strongly preferred by younger women and other brands by older women.

By contrast, let's consider an agricultural experiment in which two kinds of fertilizer are applied to two varieties of soybeans under three degrees of soil compaction and three amounts of watering. The plants are raised under controlled greenhouse conditions and the main purpose of the research is to learn whether one fertilizer will produce higher yields than the other.

Now back to our consumer examples. Why the connection between beer brand and region? Climate? Tradition? Or simply distribution? Some combination of the three, plus other factors? In our fashion example, income could be a factor underlying the age effect we observe. Or our result may merely stem from the fact that different brands are designed for and marketed to women of different ages.

These are examples of non-experimental research, also known as observational research. Some marketing researchers may associate observational research with ethnography but its meaning is actually broader. 

Our agricultural experiment, on the other hand, is just that - an experiment. Experiments employ statistical designs in which subjects (e.g., soybean seeds) are experimentally assigned to one or more treatment conditions (e.g., fertilizer, soil compaction). The experimental design is created when the research is being planned. The laboratory-like conditions of the greenhouse in our illustration are intended to minimize the effects of other variables, such as temperature and soil composition, that influence soybean yields.

In plain statisticspeak, yield is the dependent variable and type of fertilizer, soybean variety, soil compaction and watering are the independent variables. We are trying to explain or predict yield with these four independent variables. However, our principal focus is the effects of the fertilizers. If we find that plants given one of the fertilizers have higher average yields, we can be confident the fertilizer is truly more effective than had we simply visited farms and took note of how each farmer was growing his crop and measured yield after the fact. The latter would have been observational, non-experimental research. We would have had no control over variables that affect yield and would be less confident that any differences in yield we observe resulted from the type of fertilizer applied.

Experiments among consumers 

Why don't we conduct experiments among consumers? We do! A taste test is an example. Another is conjoint analysis in which a choice experiment is administered to a sample of consumers. In conjoint, the treatments are product features shown to respondents in experimentally-designed combinations and sequences. Respondents choose which product they prefer in each set of combinations (tasks) they are shown.

Experimental research can be expensive, however, and this does reduce how often it is used. Experiments have another limitation: They are artificial and it may not be clear how well our conclusions can be generalized to real-world conditions. Furthermore, rough directional implications may be all we need to make our decisions and it might not be necessary to make scientific inferences regarding causation. This is one reason a sizable chunk of marketing research is purely qualitative.

Conclusions about causation

However, there are situations in which we do draw conclusions about causation, at least implicitly, and these conclusions play a central role in our decision-making. As noted, most often we do this with observational data, not with experiments. Examples abound but include crosstabulations of selected questions with respondent demographics, preferred brand, purchase frequency and attitudes. We do this when only knowing the "what" is not enough and we want to understand why consumers are behaving as they do.

We make our causal deductions based upon associations, in other words. But there are risks. "Correlation does not imply causation" is an admonition drilled into future statisticians in the classroom. We are cautioned about the post hoc ergo propter hoc fallacy (e.g., Does the crowing of the rooster really cause the sun to rise?).

Four main reasons can be responsible for an association between one variable and another: causation, chance, bias and confounding.

Causation

As the word suggests, causation means that one variable causes or influences another. There is a cause-and-effect relationship between two variables. We may claim explicitly, for example, that we believe some consumers are buying Brand X because they trust its quality. Or we may only imply that such a causal relationship exists.

Chance

Chance associations are flukes (i.e., they occur by chance alone). Significance testing provides guidelines we can use to diminish the risk of making a causal connection that is actually the result of sampling. Inferential statistics, as many readers know, is a lengthy subject and several assumptions come into play. Even when these assumptions, such as probability sampling and measurement without error, are met, sample size has a major impact on our calculations. Trivial differences may be flagged as statistically significant if our sample size is very large. Conversely, with small samples, large and substantive differences may not meet conventional cutoffs (e.g., 5 percent and be deemed insignificant). Inferential statistics can only reduce the risk of being fooled by chance. They are also an integral component of experimental research.

Bias

This is a thorny topic and can influence both experimental and observational research. Put simply, our respondents may differ substantially and systematically from our target population in ways that distort conclusions we make about them. Bias can be a very serious problem and safeguards must be put in place to reduce the possibility that bias is contaminating our research.

Confounding

Confounding is often very hard to spot. A confounder is associated with the true cause of another variable but does not itself actually cause or influence this second variable. As an example, imagine a (hypothetical) correlation between pizza consumption and traffic accidents. How could eating pizza cause traffic accidents?

One plausible explanation is that pizza is frequently consumed in tandem with alcohol. A variable we hadn't thought of (alcohol) is correlated with pizza consumption and is the true cause of the increased risk of accidents. Pizza is guilty by association!

Admittedly, the foregoing is a silly example but hopefully will demonstrate how badly we can be led astray by mere associations. When experimentation is not possible or required, statistical control is often used as a compromise. Statistical control employs multivariate analysis to simultaneously adjust for the possible effects of exogenous variables such as respondent demographics and prior category usage. Propensity score matching is an extension of this idea that is gaining popularity in marketing research.

Interactions and multicollinearity

These topics fill many textbooks. Interactions and multicollinearity are two other subjects related to our discussion so I'd like to briefly introduce them.

An interaction is present when the relationship between two variables depends on a third variable. For instance, we may observe that category usage declines with age but much more so among women than men. This result would suggest an age-by-gender interaction is present.

Multicollinearity, highly-correlated predictors (independent variables), can lead to invalid or nonsensical results. When the correlations are very high it isn't mathematically possible to isolate the separate effects of the predictors and any number of solutions is possible. Multicollinearity can be a serious complication in key driver analysis, such as in customer satisfaction research, where we try to uncover the aspects that most impact overall satisfaction with a company.

Requires trade-offs

At first these topics can be difficult to grasp but a basic understanding is essential to sound research. "Sound" does not mean "perfect," however. Any research in any field will have flaws. Research, like most things in life, requires trade-offs, and we should define our objectives concretely and realistically during the planning phase.

In marketing research it usually is not obligatory to prove a causal relationship. And it can be argued this seldom is feasible. Often it will be enough to treat the results as exploratory findings that may suggest some marketing action. On the other hand, we should temper our conclusions and not fall into the trap of making important decisions based on flimsy grounds. 

Though our discussion has highlighted quantitative consumer survey research, the fundamental issues we've covered apply to any research. It's vital that we appreciate the strengths and limitations of observational research versus experimentation when we are designing our research or interpreting results of studies already completed.

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

PREDICTIVE ANALYTICS AND BUSINESS INSIGHTS 2014
September 23-24, 2014
Gateway Analytics Network will hold a conference, themed 'Predictive Analytics and Business Insights 2014,' on September 23-24 in Philadelphia.
NETWORKING EVENT BY THE RESEARCH CLUB
September 24th, 2014
The Research Club will host a networking event in conjunction with the MRMW conference on September 24th at the Riva Bar in Berlin, Germany.

View more Related Events...

Related Articles

There are 2494 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Regression-based satisfaction analyses: proceed with caution
This article warns against mathematical and philosophical dangers in modeling overall supplier satisfaction from attribute performance or satisfaction ratings.
Best practices for online research
While online research is faster and easier to implement than its traditional counterparts, that doesn’t mean that it’s as easy as it looks. This article outlines five of the most important factors for conducing online research.
Data Use: (Sub-)optimal test designs for multivariable marketing testing
Multivariable tests are valuable when used to their fullest advantage. Guidelines for getting the most from these tests are offered.
The differences between mystery shopping and marketing research
Both mystery shopping and marketing research are long-established tools to help businesses and organizations operate more effectively. This article compares mystery shopping and marketing research services and offers some insights into how mystery shopping can be used effectively to augment marketing research efforts.
Four steps to improve customer satisfaction and loyalty
The author explains how to interpret customer satisfaction scores and how to use these scores to improve customer satisfaction in four steps.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer
TURF excel-based simulator
03/25/2014 by Werner Mueller
I would like Turf Macro too!
03/06/2014 by Neelam Hinduja

View More