Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

Forget exact science: Drawing conclusions from observational research



Article ID:
20130726-2
Published:
July 2013
Author:
Kevin Gray

Article Abstract

While most marketing research is observational in nature, we also conduct experimental research. Each has important advantages and disadvantages that are frequently overlooked and this article addresses them.

Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing science and analytics consultancy. This article appeared in the July 22, 2013, edition of Quirk's e-newsletter.  

Say, for example, we observe that some beer brands are much more popular in certain parts of the country than in others. Or perhaps we find that some fashion brands are strongly preferred by younger women and other brands by older women.

By contrast, let's consider an agricultural experiment in which two kinds of fertilizer are applied to two varieties of soybeans under three degrees of soil compaction and three amounts of watering. The plants are raised under controlled greenhouse conditions and the main purpose of the research is to learn whether one fertilizer will produce higher yields than the other.

Now back to our consumer examples. Why the connection between beer brand and region? Climate? Tradition? Or simply distribution? Some combination of the three, plus other factors? In our fashion example, income could be a factor underlying the age effect we observe. Or our result may merely stem from the fact that different brands are designed for and marketed to women of different ages.

These are examples of non-experimental research, also known as observational research. Some marketing researchers may associate observational research with ethnography but its meaning is actually broader. 

Our agricultural experiment, on the other hand, is just that - an experiment. Experiments employ statistical designs in which subjects (e.g., soybean seeds) are experimentally assigned to one or more treatment conditions (e.g., fertilizer, soil compaction). The experimental design is created when the research is being planned. The laboratory-like conditions of the greenhouse in our illustration are intended to minimize the effects of other variables, such as temperature and soil composition, that influence soybean yields.

In plain statisticspeak, yield is the dependent variable and type of fertilizer, soybean variety, soil compaction and watering are the independent variables. We are trying to explain or predict yield with these four independent variables. However, our principal focus is the effects of the fertilizers. If we find that plants given one of the fertilizers have higher average yields, we can be confident the fertilizer is truly more effective than had we simply visited farms and took note of how each farmer was growing his crop and measured yield after the fact. The latter would have been observational, non-experimental research. We would have had no control over variables that affect yield and would be less confident that any differences in yield we observe resulted from the type of fertilizer applied.

Experiments among consumers 

Why don't we conduct experiments among consumers? We do! A taste test is an example. Another is conjoint analysis in which a choice experiment is administered to a sample of consumers. In conjoint, the treatments are product features shown to respondents in experimentally-designed combinations and sequences. Respondents choose which product they prefer in each set of combinations (tasks) they are shown.

Experimental research can be expensive, however, and this does reduce how often it is used. Experiments have another limitation: They are artificial and it may not be clear how well our conclusions can be generalized to real-world conditions. Furthermore, rough directional implications may be all we need to make our decisions and it might not be necessary to make scientific inferences regarding causation. This is one reason a sizable chunk of marketing research is purely qualitative.

Conclusions about causation

However, there are situations in which we do draw conclusions about causation, at least implicitly, and these conclusions play a central role in our decision-making. As noted, most often we do this with observational data, not with experiments. Examples abound but include crosstabulations of selected questions with respondent demographics, preferred brand, purchase frequency and attitudes. We do this when only knowing the "what" is not enough and we want to understand why consumers are behaving as they do.

We make our causal deductions based upon associations, in other words. But there are risks. "Correlation does not imply causation" is an admonition drilled into future statisticians in the classroom. We are cautioned about the post hoc ergo propter hoc fallacy (e.g., Does the crowing of the rooster really cause the sun to rise?).

Four main reasons can be responsible for an association between one variable and another: causation, chance, bias and confounding.

Causation

As the word suggests, causation means that one variable causes or influences another. There is a cause-and-effect relationship between two variables. We may claim explicitly, for example, that we believe some consumers are buying Brand X because they trust its quality. Or we may only imply that such a causal relationship exists.

Chance

Chance associations are flukes (i.e., they occur by chance alone). Significance testing provides guidelines we can use to diminish the risk of making a causal connection that is actually the result of sampling. Inferential statistics, as many readers know, is a lengthy subject and several assumptions come into play. Even when these assumptions, such as probability sampling and measurement without error, are met, sample size has a major impact on our calculations. Trivial differences may be flagged as statistically significant if our sample size is very large. Conversely, with small samples, large and substantive differences may not meet conventional cutoffs (e.g., 5 percent and be deemed insignificant). Inferential statistics can only reduce the risk of being fooled by chance. They are also an integral component of experimental research.

Bias

This is a thorny topic and can influence both experimental and observational research. Put simply, our respondents may differ substantially and systematically from our target population in ways that distort conclusions we make about them. Bias can be a very serious problem and safeguards must be put in place to reduce the possibility that bias is contaminating our research.

Confounding

Confounding is often very hard to spot. A confounder is associated with the true cause of another variable but does not itself actually cause or influence this second variable. As an example, imagine a (hypothetical) correlation between pizza consumption and traffic accidents. How could eating pizza cause traffic accidents?

One plausible explanation is that pizza is frequently consumed in tandem with alcohol. A variable we hadn't thought of (alcohol) is correlated with pizza consumption and is the true cause of the increased risk of accidents. Pizza is guilty by association!

Admittedly, the foregoing is a silly example but hopefully will demonstrate how badly we can be led astray by mere associations. When experimentation is not possible or required, statistical control is often used as a compromise. Statistical control employs multivariate analysis to simultaneously adjust for the possible effects of exogenous variables such as respondent demographics and prior category usage. Propensity score matching is an extension of this idea that is gaining popularity in marketing research.

Interactions and multicollinearity

These topics fill many textbooks. Interactions and multicollinearity are two other subjects related to our discussion so I'd like to briefly introduce them.

An interaction is present when the relationship between two variables depends on a third variable. For instance, we may observe that category usage declines with age but much more so among women than men. This result would suggest an age-by-gender interaction is present.

Multicollinearity, highly-correlated predictors (independent variables), can lead to invalid or nonsensical results. When the correlations are very high it isn't mathematically possible to isolate the separate effects of the predictors and any number of solutions is possible. Multicollinearity can be a serious complication in key driver analysis, such as in customer satisfaction research, where we try to uncover the aspects that most impact overall satisfaction with a company.

Requires trade-offs

At first these topics can be difficult to grasp but a basic understanding is essential to sound research. "Sound" does not mean "perfect," however. Any research in any field will have flaws. Research, like most things in life, requires trade-offs, and we should define our objectives concretely and realistically during the planning phase.

In marketing research it usually is not obligatory to prove a causal relationship. And it can be argued this seldom is feasible. Often it will be enough to treat the results as exploratory findings that may suggest some marketing action. On the other hand, we should temper our conclusions and not fall into the trap of making important decisions based on flimsy grounds. 

Though our discussion has highlighted quantitative consumer survey research, the fundamental issues we've covered apply to any research. It's vital that we appreciate the strengths and limitations of observational research versus experimentation when we are designing our research or interpreting results of studies already completed.

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 501: FACILITATION - PRACTICAL TOOLS, TIPS AND TECHNIQUES
August 11-13, 2014
RIVA Training Institute will hold a course, themed 'Facilitation - Practical Tools, Tips, and Techniques,' on August 11-13 in Rockville, Md.
RIVA COURSE 202: SKILL ACCELERATION
September 8-10, 2014
RIVA Training Institute will hold a course, themed 'Skill Acceleration' on September 8-10 in Rockville, Md.

View more Related Events...

Related Articles

There are 2470 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Measuring consumer attitudes: what is your scale really telling you?
One of the fundamental purposes of market research is to understand a targeted audience. This article discusses how several alternative attitudinal scales were tested on more than 1,000 respondents, measuring the same attribute yet providing radically different empirical results.
Are you collecting too much information in your 'voice of the customer' process?
Building on the January 1999 article “’Voice of the Customer’ Disconnects Still Exist in Most Companies,’” this article addresses fundamental shortcomings in the design of the VOC process.
Data mining and usage for corporate profit
Many American companies collect customer satisfaction data but one has to wonder what effect that customer information has on their businesses. Only companies that know their customers’ needs will thrive. This article shows how placing customer information front and center in the planning process can produce dramatic results and transform the way a company does business.
Qualitatively Speaking: Thoughts on Gladwell’s Blink
A researcher responds to some of Malcolm Gladwell’s anti-research viewpoints as expressed in his book Blink.
Where will your company’s next great idea come from? From your customers.
Collaborating with customers, through online communities, offers a new and exciting way to develop new product ideas. The authors present examples from Del Monte Foods and its research with pet owners and from a toy manufacturer and snack-food maker.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

request
06/06/2014 by Monika Kunkowska
TURF excel-based simulator
04/17/2014 by Giovanni Olivieri
XLSTAT Turf
04/10/2014 by Felix Schaefer
TURF excel-based simulator
03/25/2014 by Werner Mueller
I would like Turf Macro too!
03/06/2014 by Neelam Hinduja

View More