Heard the one about the diabetic alligator?

Editor’s note: Fred Phillips is a research professor at Oregon Health & Science University, Beaverton, Ore., and a former market research executive.

The research firm that employed me from 1976-89 measured consumers’ purchase behavior in the packaged goods and dry goods categories sold by the largest companies in the U.S. Our clients compared our purchase volume numbers to what they thought they had sold - and sometimes, to what they wanted to believe they had sold. When there was a discrepancy, they called on yours truly, the firm’s hapless but not totally unprepared quality assurance director.

Discrepancy, thy name is coverage

Our Census-balanced consumer panels were weighted upward so we could give clients population estimates of sales volumes, penetrations, shares, repeat buying and so on. Because our sales volume estimates were readily compared to clients’ factory shipment numbers, clients didn’t hesitate to complain about any gap between the two. We called these complaints “coverage problems.”

Coverage complaints were frequently a cover (excuse the pun) for price negotiations or demands for free analyses. However, our estimates were careful and usually extremely good, given the sample sizes clients were willing to pay for.

Dealing with coverage problems involved interesting travel and intriguing mysteries. Their solutions, as you will see, were of varied kinds.

Accounting mysteries

Some were not that mysterious. One client sold jewelry door to door, in those days when “housewife” was an acceptable and meaningful name for a customer segment. Their sales figures were consistently higher than our purchase figures, and they called us on the carpet for it. I flew to the client’s New York headquarters, not having an answer in my pocket, but hoping that by talking with them I’d be enlightened. As they spoke, I thought about the sales process and the “conversations” that might ensue when one spouse says to the other, “Honey, look what I bought today!”

Snapping back from my reverie I blurted out, “Have you removed returned merchandise from the shipments you’re comparing to our estimates?”

“We’re not sure,” was the response. “We’ll check.”

A few minutes later, an executive returned to the room and said, no, they had been using gross shipment numbers. A quick calculation showed that our estimates fit their returns-adjusted shipments more than reasonably well. I flew home.

Logistics mysteries

A food-industry client claimed to know how much of their product was on store shelves at any given moment. How did they know this? Their drivers, they told us, shelved the merchandise themselves, under agreements between the major supermarket chains and the manufacturer. It was these store volume numbers to which they compared our estimates. They claimed to notice a gap, and your QA director was mobilized.

I should say that my role model in coverage investigation was the protagonist of a story told by Colorado School of Mines Professor Gene Woolsey. In his story of missing orange juice, our protagonist begins his quest in the Florida warehouse where slurry (partially concentrated juice) is pumped into tanker trucks. The warehouse abuts a swamp, and he notices a number of dead alligators near the edge of the water. When one truck is sealed, our hero puts his own crimp in the seal, and follows the truck to a New York warehouse, where he finds the seal intact. The juice is drained into cans, and the cans into cases. Hiding in the warehouse during a shift change, he notices a few loading-dock workers putting cases of OJ into their cars. Punching his calculator under the light of a pocket flashlight, he determines that continued theft of this sort could amount only to a tiny fraction of his coverage problem. He follows the truck back to Florida, where its interior is steamed out and the process begun again.

Wait…steamed out? Spotting another returning deadhead tanker, he climbs inside and thrusts a ruler toward the interior wall, which proves to have a six-inch layer of concentrated orange juice adhering to it. Now, what’s the formula for the volume between two cylinders? The trusty calculator shows that this is the answer to the coverage problem: a steam machine is dumping thousands of gallons of sugary orange concentrate into the swamp. Our hero gets his employer to fund an autopsy on one of the alligators, which proves to have died of diabetes.

With Woolsey’s inspiring example in mind, and spotting one of my own client’s trucks tooling down a Texas highway, what could I do but follow it? It left the highway and entered a Safeway parking lot. I watched the driver dolly a load of goods into the back door of the supermarket, and I planted myself near the interior door between the back and front sections of the store. The driver never appeared. I waited, waited longer, then looked outside to see him driving the truck away.

I mentioned this to the client, who replied, “Oh, I forgot to tell you, Safeway doesn’t let our drivers shelve the goods.”

I said, “Safeway is only the largest supermarket chain in the country (as it was then), and you forgot to tell me this?!”

The matter more or less ended there, demonstrating that this “coverage problem” had more to do with the client’s need to posture than with actual concern about the estimates. However, solving coverage problems can involve having to know a lot about the physical distribution of the product. (Not only logistics and channel behavior, but how much end-user product goes to dormitories, school cafeterias, restaurants, hotels, prisons, military messes, cruise ships, government warehouses, fraternal organizations, foreign sales and foreign aid, and other destinations that are not U.S. households.) It’s also true that with a nationally distributed brand, shrinkage (the minor theft, loss and damage that occurs during distribution - not what happens to Seinfeld’s friend George Costanza) is rarely material to a coverage gap.

Sample composition mysteries

We received a call from a client that made pancake syrup. We were, the client accused, suddenly and drastically over-covering syrup shipments, by as much as 100 percent.

I checked the consumer panel purchase database for the time period in question. A fairly low sample penetration of pancake syrup for the period; lots of people buy it, but not very frequently. That meant a smaller current buying sample size and hence potentially bigger sampling variation. And one sample family in Mississippi seemed to be varying big time; they were buying a lot of syrup.

Our panel relations manager phoned them. They lived on a large ranch in Mississippi, they told us. Why so much pancake syrup? A few times a week, after morning chores, they invited all the ranch hands to the house for a big pancake breakfast! We expected our sample families to be as hospitable as anyone else, but this was ridiculous. The panel relations manager asked whether the family would mind being excused from further panel duties - to which they replied, “Sure, hon, fine.” - and syrup market estimates fell back in line with shipments.

Trademark mysteries

Another instance of over-coverage had nothing to do with sample aberrations. We were over-covering a major brand of pantyhose, and there was no apparent reason for it. In a brainstorming session with the client, one of my colleagues asked whether there would be any reason that panel members would not be identifying the brand correctly. “Well, now that you mention it,” the client allowed, “we have some legal proceedings underway against an Asian company that’s making a look-alike knock-off.” We could not see their lawyer’s discovery files, but the client later said estimated sales of the Asian substitute neatly filled the coverage gap.

Data entry mysteries

In an old but instructive story, an assertive customer is poking through a bin of smallish frozen turkeys. “Don’t these things get any bigger?” he complains. “No,” the butcher replies, “They’re dead.”

I frankly don’t know why three bytes were allocated for the weight field in our database’s frozen turkey records; did we expect turkeys to get that much bigger? A question arose in a month that was not November - meaning that few turkeys were purchased, and small sample variations had a big impact on reported volumes. When we discovered that a key operator had typed “270” pounds instead of “027” for one purchased turkey, that was the end of that particular over-coverage problem.

Math, statistics, and client relations mysteries

When coverage stays at a constant level, say 85 percent of client shipments, the gap doesn’t affect the measurement of the trends which greatly concern clients. If sales volume increases 10 percent from one quarter to the next, the panel trend accurately reflects the shipment trend. Similarly, goods take time to travel along the distribution chain. A regular lag between panel volumes and shipment volumes is easily detected and corrected.

Items that do not reach the home are less likely to be entered in panel diaries. Snacks and smaller packages, things that are eaten outside the home, generally mean lower but still steady coverage. A higher “weighting factor” for snack foods is helpful and easily accepted by clients.

Other events can make coverage anything but constant, regular and steady. A wholesaler or retailer who is expecting a price increase will buy ahead as much as his cash allows. However, no one likes to keep cash tied up in inventory, so store promotions proliferate. They are attempts to reduce inventories. Much of marketing is a three-way game among consumers, retailers and wholesalers, each trying to get the others to carry the inventories. Retailers also execute manufacturers’ promotions. If the promotion works, consumer sales increase. The retailer must order enough stock in time to cover the expected demand. This introduces irregularities into the distribution lag. You can imagine the impact on coverage.

One client, a major snack manufacturer, had been increasing prices steadily, to the point where its products were expensive relative to other brands. They had also begun to promote much more intensively than in the past. Small wonder that coverage was destabilized. This was an important account, and an assistant and I worked full time for two weeks to devise a mathematical correction that transparently captured all the known price, promotion and lag effects. The model succeeded in bringing panel numbers and shipment numbers back into synch.

At the critical meeting, an executive of the client company argued with my new numbers. I knew (and I think he knew I knew) that he had no scientific grounds for criticizing my method. Even worse, one of his employees had leaked the fact that it was the time of year for brand managers’ bonuses! This executive’s objective was not to fix the coverage problem in the most accurate way, but to fix it in a way that would maximize his people’s bonuses. It was within his power to terminate our account, though, and I had to deal with him.

During the meeting, he tried repeatedly to tear my method apart, and finally demanded loudly, “What if I still don’t believe your numbers are any good?” Naïvely irritated that he had twisted the scientific question for his own (not even his company’s) gain, I injudiciously told him that we could step outside and settle it there. To the surprise of all, he backed down, mumbling, “Well, you have to understand, I even doubt numbers my mother gives me.”

We arranged to provide the client with two concurrent market assessments: “uncorrected” and “corrected.” They eventually decided to purchase only the “uncorrected” reports, which, with no technical footnotes, were easier to understand.

I do not recommend violence, or even the threat of it, as a negotiating tactic. Thinking back, I believe this is what happened: standing my ground on what had become an ethical question, I was helping the client save face by not placing the ethical issue explicitly on the table. I was fortunate to have the support of an ethical top management at the research firm. We reported our best take on what the market was doing, and never distorted estimates to suit a client’s wish to look good. I could admire the executive’s desire to take care of his people, but could not let that affect our reputation for unbiased reporting.

Trading area mysteries

A premium brand of packaged meats was sold only in a four-state area, and the client paid only for consumer data from those four states. We consistently under-covered sales, and the client complained.

“Your brand is pretty well-known,” we ventured during a meeting.

“Well, yes, of course it is.” How could a client say otherwise?

“Here is a map of your trading area and the surrounding states. Lots of roads in and out of the four-state area, and quite a few big population centers nearby but outside the area. Are you thinking what we’re thinking?”

As it happened, this client’s neighbor was a big fan of Coors beer when Coors was a regional brand available only in the far west. The neighbor would often drive to Colorado and return with his car trunk full of Coors. Our client was more than ready to believe that consumers were crossing state lines just to buy his meats, and doing so in the volume described by the coverage gap.

“We can easily test this by sampling our panel members in adjoining states. Here’s what it will cost. In the future, you can buy nationwide data, or just continue with the four-state data and guesstimate the ‘export’ sales. After all, as you’ve shown us, you already know how much you’re selling. You buy our data for our detailed numbers on penetrations, shares, per-purchase volumes, promotion response and purchase frequencies - not for total sales numbers.”

This solution had the virtue of feeding the client’s sense of worth (people were driving long distances to buy his product!) while also further educating him about the use of our data. He continued with the four-state data for a while, and then moved to full reporting when the brand went national.

Third-party sources

In some cases, coverage questions stemmed from clients comparing our reports to another data source: their own shipments or store deliveries, or reports from a third party. More to the point, it was a comparison of our estimates to what the client thought we should have reported. These other sources naturally had their own measurement problems. We had to mention the latter gently; attacking a client’s credibility is not wise, even when he is attacking ours.

A client’s first thought was always that undercoverage resulted from panel under-reporting. This was often true (though after 50 years in business, our company had a good handle on its magnitude). Panel households with other concerns or less free time - new-baby households, households with both spouses working outside the home; young, single-male households - do tend to neglect their purchase diaries more than others. Different product categories, and categories mentioned in different positions in the diary also correlated with different levels of under-reporting.

Our sample balance, and hence the basic weighting factors, were based on the U.S. Census, which is conducted every 10 years. Government estimates of intercensal populations are based on its Current Population Survey, which itself is sample-based and subject to measurement error.

So coverage problems were often real; the real ones could nearly always be corrected by math modeling; some irreducible uncertainty always remained; and much useful marketing analysis could be done with our data regardless of the coverage level.

As these stories show, though, coverage complaints could also mask a variety of human problems. A coverage complaint was often simply the client’s way of saying, “I don’t think you have been paying much attention to me lately.”