Editor’s note: Stephen J. Hellebusch is senior account executive with Directions Research Inc., Cincinnati.

In the book Predictably Irrational by Dan Ariely, a study is described which reveals something very interesting about the effect of context on how we make decisions. Knowledgeable and very intelligent people who were fully aware that certain numbers were chosen in a completely arbitrary, random fashion were still influenced by those numbers when making decisions about what they would pay for certain items. This is described as arbitrary coherence, a term found in the behavioral economics literature. This is so illogical that I replicated it to see for myself, as described in this article.

This influence on bright people who “knew the story” has implications for marketing researchers. If, as has been demonstrated, irrelevant context can have an impact on responses even when the responder knows it is irrelevant, it suggests that marketing researchers must work harder to compensate for this factor.

Many marketing researchers are familiar with the standard “AB-BA” balancing pattern. In taste-tests between two products, for example, half of the respondents taste A first, B second, and half taste B first, A second. We look at the total, which is balanced, and usually look at the two cells (A1-B2 and A2-B1) to see if order effects exist. (They often do!) Arbitrary coherence suggests it would be wise to employ the same logic more widely. This possibility is discussed at the end of this article.

Method

Basically, in the original study, MIT students were asked three questions about several items that might be purchased in a store. However, the first step they were asked to take was irrelevant: to write the last two digits of their Social Security number (SSN) in dollars next to the item. So, if their SSN ended in 12, they wrote $12. If it ended in 87, they wrote $87. Next, they were asked to indicate whether they would pay that amount for the item (yes/no). Finally, they were told to pretend the item was being auctioned and to write in the amount they would be willing to pay.

Replicating this research required a call for volunteers, and 71 of the author’s associates at Directions Research Inc. responded. This replicates the reported study in that the respondents were, for the most part, highly educated professionals.

The first step was to select items. Four were randomly selected from the Sunday newspaper FSIs: a 12-bar value-pack of Irish Spring soap; one pair of Dr. Scholl’s hand-stitched leather loafers; a 750ml bottle of Baileys Irish cream; and a Teleflora Spring Pitcher floral display.

The respondents filled out a self-administered questionnaire that asked them to write down the last two digits of their SSN as if they were dollars, then asked whether they would buy each of the items for that amount, and for the amount they would be willing to pay if the item were being auctioned.

Results

The results from Predictably Irrational are quite uniform. Six varied items - computer elements (trackball, keyboard), a book, edibles - are reported, and the average price willing to pay is categorized by the last two digits of the SSN. The results show that having the arbitrarily-chosen SSN “price” near at hand leads respondents to bid higher and higher in the auction. As the SSN price rises, so does the average amount people are willing to pay. The correlations between the SSN price and the price respondents would pay ranged from a low of +0.32 to a high of +0.52. Those are the type of results expected for the four products used in this replication attempt.

Clearly, the results shown in Table 1 are a little different than expected. For the Teleflora display and the Dr. Scholl’s loafers, results are very consistent with what has been reported. With these two items, the earlier research is successfully replicated.

With the Baileys Irish cream, not so much. A much weaker pattern of average price increase with increasing SSN price is found, and the correlation is much weaker than any previously reported. For the Irish Spring 12-pack, the phenomenon breaks down completely, and the correlation is close enough to zero to say there is no correlation.

One of the questions about the summary report described in Predictably Irrational is “What percent were willing to buy the items for the SSN dollar price?” That finding is not reported. In this case, however, many were still willing to buy Teleflora at the highest prices, and some stayed with Dr. Scholl’s loafers to the highest level. But no one was willing to pay over $50 for 750ml of Baileys, and no one was willing to pay $26 or more for 12 bars of soap. At least at Directions Research Inc., people are not irrational.

Generally speaking, as prices go up, a smaller and smaller percent is willing to buy, as expected. Some stayed with Teleflora and with Dr. Scholl’s loafers, but the percentages who did so drop sharply after the $50 price point (Table 2).

Finally, there was speculation concerning what the initial results (Table 1) would look like if the data were restricted to only those willing to pay for the item at the SSN price. This required new (and small) base sizes for each item in each price class.

Interestingly, the phenomenon is still seen for Teleflora and Dr. Scholl’s when this restriction is made, even though the numbers in each group get very, very small. Also interesting is the fact that, with items that most would believe are lesser values, no one was willing to buy at the highest prices, so there is no “average price” respondents are willing to pay (Table 3).

Arbitrary coherence exists

Overall, the replication demonstrated that arbitrary coherence exists. The research was replicated, to a degree. For the items of lowest interest (Baileys Irish cream and Irish Spring 12-pack), it did not work quite so well, suggesting that there are other factors at work. For an inexpensive item that everyone knows is inexpensive (i.e., soap), it did not work at all. So, the hypothesis to be tested is that arbitrary coherence works when we have a high degree of uncertainty about the cost of an item, but does not work so well - we are not influenced by the random number - when we are assessing an item that has an approximately known price.

If a known, irrelevant context has a definite effect on responses, at the very least those who write questionnaires should be made aware that this is so. There may be ways to deal with the concern. One that comes to mind is to balance, rotate or randomize sections of related questions within a questionnaire, as well as the continuing the standard use of these methods within appropriate sections. Obviously, not every section of a questionnaire is movable, since the flow cannot be reduced to gibberish, but there may be many interchangeable sections that the designer would not think to interchange, seeing no reason to do so.

Similarly, at least for the most important projects, multiple interviewing modalities may be a good idea - conduct some interviews online, some by phone, some in person. If the results are similar, it removes or reduces the concern that some unknowable context effect is at work. Once, this would have been considered ridiculous, just because of added expense. Now, we know there is a reason: Context known to be completely irrelevant can have an effect!