Editor's note:Rajan Sambandam is chief research officer at TRC Market Research. He can be reached at rsambandam@trchome.com.
The subtitle of Julia Galef’s recent book, “The Scout Mindset,” is “Why Some People See Things Clearly and Others Don’t.” It is based on the idea that there is a lot of uncertainty in the world and it is impossible to account for everything. So, changing our minds based on new information is a virtue that allows us to constantly update our maps of reality – the scout mind-set.
This is in contrast to the soldier mind-set, which is more rigid and unwilling to change and falls prey to various biases such as motivated reasoning that allow one to rest comfortably inside a perception that could diverge significantly from reality.
This view is very appealing as it reflects a perspective that I used when I taught market research to MBA students. The purpose of research is not to obtain perfect information – that’s impossible. The purpose is to go from a state of higher uncertainty to one of lower uncertainty. Of course, we do this by systematically gathering data, analyzing it for insights and providing results to help business decision-makers.
Where do Mr. Spock of TV’s “Star Trek” and famed naturalist Charles Darwin, referenced in my article’s title, fit in? In the way we gather and use information to come to the right conclusion. Before we get to them, let’s first look at what we mean by uncertainty.
Two kinds of uncertainty
Galef talks about two kinds of uncertainty – uncertainty in you and uncertainty in the world. Though they are conflated in practice, it is the former that is problematic. For example, she cites studies that show that patients are unhappy with doctors who seem uncertain – making them wonder if a better doctor should be consulted. But when doctors clearly explain the complexity of the case and provide context for their uncertainty, the patients are much happier.
In market research we are, of course, plagued by uncertainty, while at the same time, business decision-makers clamor for certainty. Saying “I don’t know” may be taken as an inadequate response because the decision-maker may question the competence of the researcher. This could lead to less-desirable options such as trusting the decision maker’s gut – or some other source that feigns certainty.
The better option is to show that uncertainty is justified, i.e., it is “in the world” not “in us.”
The simplest example, of course, is margin of error in surveys. Results come shrouded in uncertainty and that rarely has to do with the competence of the researcher (though it can). Providing context and explaining why the numbers may be closer to directional than precise can show that uncertainty should be respected not wished away.
This is true even when using more advanced methods like conjoint analysis. Their sophistication can help temper uncertainty but does not eliminate it. And their complexity can make them seem artificially precise, lulling an executive into thinking uncertainty has been squeezed out of the results.
Not so.
By providing the context for how the data were collected, and explicitly acknowledging uncertainty, we can convey the caution needed in interpreting the results.
With that in mind, let’s move on to Spock and Darwin.
The Spock problem
Galef watched and noted every prediction Spock ever made and evaluated their accuracy. It turns out that Spock is really quite bad at predicting things. For example, when he thinks something is impossible, it happens 83% of the time, while something that he thinks is more than 99.5% likely happens just 17% of the time.
If Spock had been from the planet Bayesia rather than Vulcan, perhaps he wouldn’t be so bad at making predictions. Rather than blithely believing in the strength of his logical thinking, he may have correlated his predictions with actual outcomes and, over time, improved his ability to make predictions.
This should have particular resonance for market researchers. While Bayesian updating is the analytic foundation of conjoint estimation, we don’t always apply that to thinking about research problems in general.
Going back to the beginning, if our purpose is to move from a point of greater to lesser uncertainty, it is unlikely to happen in one shot (or one study). It is more often an incremental process that includes triangulated information from varied sources, in order to reduce uncertainty over time.
A handy tool in this process is thinking in bets. “If I were to bet money on an outcome, what would I say?” is qualitatively different from simply summarizing the results from a study. Such a process can be helpful in calibrating our confidence level and providing more realistic predictions.
Spock’s problem is not just unwarranted confidence in his thinking but also an inability to learn from one situation to the next.
Darwin, though, is quite different.
Darwin‘s process
No, it’s not about the theory of evolution, natural selection or survival of the fittest. It’s about Darwin’s process for getting to those paradigm-shifting ideas. Darwin followed what he called a “golden rule” to combat motivated reasoning:
“...whenever a published fact, a new observation or thought came across me, which was opposed to my general results, (I would) make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favorable ones.”
Compare this to (the admittedly fictitious) Spock’s reaction to events. Unlike Spock, Darwin is essentially following a Bayesian updating process, going the extra mile to ensure that unfavorable outcomes don’t get underweighted, by specifically noting them down. And, as Galef observes, in the long run this proved tremendously useful as seeming anomalies (such as the gaudy tails of peacocks) actually ended up making his theory stronger.
Open to changing their minds
As researchers, we are privileged to constantly work with data and help executives make decisions. The right way to do that is by having a scout mind-set. Scouts understand that their map of reality is inaccurate (often wildly so) and strive to make it incrementally more accurate. Their beliefs are never threatened as they are always open to changing their minds in response to new information. They don’t minimize information that’s counter to their perspective and are willing to make proportionate changes.
It’s a process that takes time, effort, appropriately calibrated confidence and constant incremental updating. In other words, it’s the process of doing research well and conveying results to decision-makers.
Research strong and prosper.