Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Forget exact science: Drawing conclusions from observational research



Article ID:
20130726-2
Published:
July 2013
Author:
Kevin Gray

Article Abstract

While most marketing research is observational in nature, we also conduct experimental research. Each has important advantages and disadvantages that are frequently overlooked and this article addresses them.

Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing science and analytics consultancy. This article appeared in the July 22, 2013, edition of Quirk's e-newsletter.  

Say, for example, we observe that some beer brands are much more popular in certain parts of the country than in others. Or perhaps we find that some fashion brands are strongly preferred by younger women and other brands by older women.

By contrast, let's consider an agricultural experiment in which two kinds of fertilizer are applied to two varieties of soybeans under three degrees of soil compaction and three amounts of watering. The plants are raised under controlled greenhouse conditions and the main purpose of the research is to learn whether one fertilizer will produce higher yields than the other.

Now back to our consumer examples. Why the connection between beer brand and region? Climate? Tradition? Or simply distribution? Some combination of the three, plus other factors? In our fashion example, income could be a factor underlying the age effect we observe. Or our result may merely stem from the fact that different brands are designed for and marketed to women of different ages.

These are examples of non-experimental research, also known as observational research. Some marketing researchers may associate observational research with ethnography but its meaning is actually broader. 

Our agricultural experiment, on the other hand, is just that - an experiment. Experiments employ statistical designs in which subjects (e.g., soybean seeds) are experimentally assigned to one or more treatment conditions (e.g., fertilizer, soil compaction). The experimental design is created when the research is being planned. The laboratory-like conditions of the greenhouse in our illustration are intended to minimize the effects of other variables, such as temperature and soil composition, that influence soybean yields.

In plain statisticspeak, yield is the dependent variable and type of fertilizer, soybean variety, soil compaction and watering are the independent variables. We are trying to explain or predict yield with these four independent variables. However, our principal focus is the effects of the fertilizers. If we find that plants given one of the fertilizers have higher average yields, we can be confident the fertilizer is truly more effective than had we simply visited farms and took note of how each farmer was growing his crop and measured yield after the fact. The latter would have been observational, non-experimental research. We would have had no control over variables that affect yield and would be less confident that any differences in yield we observe resulted from the type of fertilizer applied.

Experiments among consumers 

Why don't we conduct experiments among consumers? We do! A taste test is an example. Another is conjoint analysis in which a choice experiment is administered to a sample of consumers. In conjoint, the treatments are product features shown to respondents in experimentally-designed combinations and sequences. Respondents choose which product they prefer in each set of combinations (tasks) they are shown.

Experimental research can be expensive, however, and this does reduce how often it is used. Experiments have another limitation: They are artificial and it may not be clear how well our conclusions can be generalized to real-world conditions. Furthermore, rough directional implications may be all we need to make our decisions and it might not be necessary to make scientific inferences regarding causation. This is one reason a sizable chunk of marketing research is purely qualitative.

Conclusions about causation

However, there are situations in which we do draw conclusions about causation, at least implicitly, and these conclusions play a central role in our decision-making. As noted, most often we do this with observational data, not with experiments. Examples abound but include crosstabulations of selected questions with respondent demographics, preferred brand, purchase frequency and attitudes. We do this when only knowing the "what" is not enough and we want to understand why consumers are behaving as they do.

We make our causal deductions based upon associations, in other words. But there are risks. "Correlation does not imply causation" is an admonition drilled into future statisticians in the classroom. We are cautioned about the post hoc ergo propter hoc fallacy (e.g., Does the crowing of the rooster really cause the sun to rise?).

Four main reasons can be responsible for an association between one variable and another: causation, chance, bias and confounding.

Causation

As the word suggests, causation means that one variable causes or influences another. There is a cause-and-effect relationship between two variables. We may claim explicitly, for example, that we believe some consumers are buying Brand X because they trust its quality. Or we may only imply that such a causal relationship exists.

Chance

Chance associations are flukes (i.e., they occur by chance alone). Significance testing provides guidelines we can use to diminish the risk of making a causal connection that is actually the result of sampling. Inferential statistics, as many readers know, is a lengthy subject and several assumptions come into play. Even when these assumptions, such as probability sampling and measurement without error, are met, sample size has a major impact on our calculations. Trivial differences may be flagged as statistically significant if our sample size is very large. Conversely, with small samples, large and substantive differences may not meet conventional cutoffs (e.g., 5 percent and be deemed insignificant). Inferential statistics can only reduce the risk of being fooled by chance. They are also an integral component of experimental research.

Bias

This is a thorny topic and can influence both experimental and observational research. Put simply, our respondents may differ substantially and systematically from our target population in ways that distort conclusions we make about them. Bias can be a very serious problem and safeguards must be put in place to reduce the possibility that bias is contaminating our research.

Confounding

Confounding is often very hard to spot. A confounder is associated with the true cause of another variable but does not itself actually cause or influence this second variable. As an example, imagine a (hypothetical) correlation between pizza consumption and traffic accidents. How could eating pizza cause traffic accidents?

One plausible explanation is that pizza is frequently consumed in tandem with alcohol. A variable we hadn't thought of (alcohol) is correlated with pizza consumption and is the true cause of the increased risk of accidents. Pizza is guilty by association!

Admittedly, the foregoing is a silly example but hopefully will demonstrate how badly we can be led astray by mere associations. When experimentation is not possible or required, statistical control is often used as a compromise. Statistical control employs multivariate analysis to simultaneously adjust for the possible effects of exogenous variables such as respondent demographics and prior category usage. Propensity score matching is an extension of this idea that is gaining popularity in marketing research.

Interactions and multicollinearity

These topics fill many textbooks. Interactions and multicollinearity are two other subjects related to our discussion so I'd like to briefly introduce them.

An interaction is present when the relationship between two variables depends on a third variable. For instance, we may observe that category usage declines with age but much more so among women than men. This result would suggest an age-by-gender interaction is present.

Multicollinearity, highly-correlated predictors (independent variables), can lead to invalid or nonsensical results. When the correlations are very high it isn't mathematically possible to isolate the separate effects of the predictors and any number of solutions is possible. Multicollinearity can be a serious complication in key driver analysis, such as in customer satisfaction research, where we try to uncover the aspects that most impact overall satisfaction with a company.

Requires trade-offs

At first these topics can be difficult to grasp but a basic understanding is essential to sound research. "Sound" does not mean "perfect," however. Any research in any field will have flaws. Research, like most things in life, requires trade-offs, and we should define our objectives concretely and realistically during the planning phase.

In marketing research it usually is not obligatory to prove a causal relationship. And it can be argued this seldom is feasible. Often it will be enough to treat the results as exploratory findings that may suggest some marketing action. On the other hand, we should temper our conclusions and not fall into the trap of making important decisions based on flimsy grounds. 

Though our discussion has highlighted quantitative consumer survey research, the fundamental issues we've covered apply to any research. It's vital that we appreciate the strengths and limitations of observational research versus experimentation when we are designing our research or interpreting results of studies already completed.

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 201: FUNDAMENTALS OF MODERATING
January 14 - 16, 2015
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on January 14 - 16 in Rockville, Md.
RIVA COURSE 201: FUNDAMENTALS OF MODERATING
January 26-28, 2014
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on January 26-28 in Rockville, Md.

View more Related Events...

Related Articles

There are 2544 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Evolving customer satisfaction through brand authenticity
If the goal is to deliver an authentic brand experience, rather than measuring the customer experience based solely on internal organizational metrics and procedures, firms must also understand how customers perceive it. The authors use case studies to examine what happens when brands do and do not align with customer perceptions and expectations.
In-flight satisfaction research keeps United Airlines customer oriented
United Airlines uses ongoing quantitative research to determine in-flight customer satisfaction. Data from its complex in-flight survey program are used to examine competitive performance, track performance and measure success at achieving goals.
The impact of gender in e-mailed survey invitations
In response to a similar exercise in Germany, the author fielded a test in Australia to see if prospective respondents would respond differently to an e-mailed survey invitation sent by a man versus one from a woman. While gender appears not to be a factor, some interesting findings arose from including location-specific information in the subject line.
QRCA group offers tips on conducting qualitative research with Latinos
Members of a group within the Qualitative Research Consultants Association focused on researching the Latino market present their insights on conducting research with and marketing to Latino consumers.
Exploring criterion variables for business-to-business customer satisfaction programs
The authors feel (and provide supporting evidence) that the relationship between share-of-wallet and satisfaction is non-linear and argue for managers to undertake satisfaction efforts that work in concert with share-of-wallet-enhancing efforts.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

TURF excel-based simulator
12/16/2014 by Joseph O. Fayese
Hi Giovanni
10/17/2014 by Dohyun Kim
request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer

View More