Editor's note: Aaron Reid is chief behavioral scientist at Sentient Decision Science, a Portsmouth, N.H., consulting firm.

For years, the market research industry has been dominated by methods that rely on explicitly-stated information from consumers on what they want and why they want it. This surface-level data cannot avoid the tragic “can’t say/won’t say” problem (i.e., that consumers are often unable or unwilling to self-report the true drivers of their behavior). Subsequently, our models of consumer behavior that rely on this explicitly-stated data are fundamentally flawed and even our most elegant estimation procedures (e.g., hierarchical Bayesian utility estimation, structural equation modeling, etc.) cannot cover up those flaws.

It is as though we are trying to use a complex combination of circles to describe what is really an ellipse. If we have the wrong data inputs for our models, our description – and therefore our insights – will miss the mark.

Our industry is in need of descriptive models of behavior that not only provide better prediction of consumer behavior but also provide a fundamental truth about the whys behind consumer behavior. This is where true insight lies and by uncovering that insight we can reveal a reservoir of innovative ideas for business. When we know the true nature of how people reach a decision, we can then provide products and services that better meet their needs. And when companies better meet consumers’ needs, both companies and consumers prosper.

Constant search for innovation

In order to achieve growth, business is in constant search for innovation. To find true innovation we need novel insight into human needs and motivations that are not currently being met in the marketplace. As market researchers, we may feel that truly deep insights are hard to find. I believe this feeling has arisen because historically, as an industry, we have been inspecting the surface level for reasons.

The good news is that deep insights on the whys behind behavior have already been unearthed. Within the behavioral sciences there is a vast reservoir of knowledge on human needs and motivations that is largely untapped by business. The behavioral science literature, including the sub-disciplines of psychology, neuroscience, sociology, behavioral economics and cultural anthropology, possesses deep knowledge on the whys behind behavior that has yet to be applied on a broad scale by businesses. This has led to a lot of buzz around the promise of new methods, including: neuromarketing, behavioral economics, implicit associations, eye-tracking, the quantification of emotions, social media analysis and choice architecture. Each of these disciplines and research tools holds great promise for revealing new insights for business.

Let’s take the case of the behavioral economics principle of hedonic bundling as an example of how behavioral science can provide a deeper understanding of the why behind behavior and thereby provide a foundation for replicable successful marketing.

Imagine you walk into a grocery store and you see the following promotion on a gallon of milk and a bag of cookies:

Promotion A: Buy together and get $2.00 off!

It sounds like a good deal and you might consider buying both the milk and the cookies. However, imagine you walked into the same grocery store and you saw the following promotion on a gallon of milk and a bag of cookies:

Promotion B: Buy together and get $2.00 off the cookies!

Which of these two promotions sounds better to you? Promotion A or Promotion B? You’ve probably noticed that fundamentally these promotions are the same: you have to buy both products and if you do you’ll save $2.00. However, one of these promotions is much more effective at increasing bundle sales.

Kahn and Dhar (2010) dubbed Promotion B an hedonic bundle and showed that in a bundled product offering (i.e., buy two items and a get a discount), placing the discount on the more “hedonic” item in the pair (e.g., the cookies) can increase bundle adoption up to 20 percent!

Hedonic bundling is an effect that results from two key behavioral economic principles of framing (Tversky and Kahneman, 1981) and mental accounting (Thaler and Sunstein, 2008). The mechanism behind hedonic bundling (the why) works as follows: Shoppers create different mental buckets for their indulgences. Indulgences come with the specific cost of guilt of having indulged. Therefore, offering discounts on the more indulgent item in a product bundle effectively reduces the pain of the guilt following from the indulgence. Reducing the pain reduces a choice obstacle and this simple framing of the discount results in greater bundle sales.

Understanding the true why

As a researcher, understanding the true why behind this behavior (rather than relying on what consumers would state explicitly) places you at a distinct advantage in providing guidance for your client. Revealing the qualifying conditions and the mechanisms at work within each principle provides the foundation for generating applications of the principle to the marketplace. Gaining this deeper level of explanation allows for the principle to be generalized for broad strategic and tactical executions that a more surface-level explanation cannot achieve.

A surface-level explanation of why I’m more likely to buy the milk and cookies when you give me the discount off the cookies would likely have provided a non-replicable insight at best or an erroneous understanding of the why behind the behavior, leading to wasted marketing or innovation resources.

Understanding that the effect works through the mechanisms of active shopper mental accounting and “guilt mitigation” provides key insight for marketing communication, new product innovation, product-pairing ideas and social media promotion strategy, as well as balanced product offerings across the product and brand suite and inventory management, to name a few. The description of the behavior is more fundamental and therefore the insights more powerful for driving business growth.

Hundreds of principles

It is important to note that this is the application of one behavioral economics effect to one business application (promotion optimization) within one vertical market. In the reservoir of current behavioral science insights, there are hundreds of behavioral economics principles that can be applied to multiple business issues across all verticals. And that’s just behavioral economics! Similar depth and breadth of behavioral insight can be found in social psychology, neuroscience, sociology and cultural anthropology, among other disciplines.

This brings us to an innovation imperative as an industry. The best research firms will construct themselves around these disciplines and, with the addition of keen insight on how to apply the insights to business in a practical and accessible format, will make a tangible impact on their clients’ businesses.

While the reservoir of insights is deep and the methods for extracting insights are advancing every day, the market researcher still needs to assess when to use which advanced method to answer the fundamental business question at hand. With a strong dose of humility and a fierce commitment to delineating when specific behavioral science methods are most appropriate for which business applications, we can make the promise of behavioral science the future of business.