The project from hell: a cautionary tale

Editor’s note: Deborah C. Sawyer is president and director of research at Information Plus, a Buffalo, N.Y., research firm.

It’s more usual to write about successes than failures; the following story about a project that was more failure than success offers a cautionary tale for anyone about to embark on research, be they client or research supplier. For my company, it was simply - and always will be - TPFH: The Project From Hell.

To fully convey the horror of it all, it is first necessary to backtrack and fill in the reader about our company, our client’s goals and the project. Information Plus specializes in qualitative research, notably one-on-one interviews over the phone. We do not use structured questionnaires but rather scripts which allow for open-ended questions. Our main focus is business-to-business research. In October 1997, we were approached by a company we’ll refer to as VC. A management consulting firm, VC had used our services in the 1986-1990 period but we had not heard from them since. Boy, how we wished things had stayed that way!

VC was working for a banking industry association in Canada and wanted to demonstrate that Canadian legislation preventing banks from participating in car leasing had a detrimental effect on lease rates. VC had just done a quick survey of its own (calling a total of 50 car dealers in Canada and the U.S.) and had decided, based on its method and this small sampling, that it could prove the absence of bank financing for leases in Canada created a spread between loan and lease rates closer to 2 percent, whereas in the U.S., where the banks do participate in leasing, there was only a 1 percent difference.

So they wanted to conduct 600 interviews, 300 in each country. They turned to us, they said, because of our ability to do in-depth interviews.

In January 1998, when TPFH finally began, VC showed up with the full specs. Our discussions in 1997 had given us the skeleton of the project; here was the flesh. Research was to focus on three particular auto makers: Chevrolet, Dodge and Ford. For each, a particular vehicle had been selected. Of the 300 calls in each country, 100 were to be made to each dealer. Next came the weighting, which had to account for where dealers are located; for example, if a third of all the car dealers in Canada were in the province of Ontario, then a third of the sample had to be in Ontario. Sounds easy enough so far, right?

The next catch was that the sample was also to be allocated to cities in three size ranges: those of 250,000+ population, those in the 100,000-249,999 range and those of 99,999 and under, with 150 calls to the 250,000+ group and 150 covering the 249,999 and under cities. Oh, and a 50-mile radius had to be imposed after every trio of dealers had been called. So when we had called a Chevrolet, Dodge and Ford dealer in one population center, we would have to eliminate calling any more within a 50-mile radius of those dealers.

Enough, you think? VC wasn’t done yet. We also had to avoid the subvented rate - the low interest rates such as 0.9 percent or 1.9 percent - advertised by car dealers to attract sales. All loan and lease rates had to be for three-year terms. And, all calls had to be on a mystery shopper basis, with no identification of our company.

With those specifications out of the way, they introduced us to their computer program. Data obtained - the manufacturer’s suggested retail price, residual value, capitalized cost, cap cost reduction, lease term, stated interest rate, monthly payment  - had to be entered into a computer program which would then check the stated interest rate (which the dealer had verbally provided) against the calculated interest rate derived from the other data (buried in the monthly payment, residual value, etc.). This would reveal what the dealer was really charging. The idea was that the stated interest rate and the calculated interest rate should be within 10 basis points of one another. In this way, VC hoped the 300 calls in Canada would neatly show their presupposed 2 percent spread while those in the U.S. would calculate closer to 1 percent. (We also had to source loan rates.) They called it breakthrough research. “Breakdown” research might be a better term.

TPFH was to be conducted over a six-week period. By the time my company had finished the work, we had had to: abandon all our other client work; take on extra staff; work evenings - something we don’t usually do; and tear out most of our hair. And, as a company which usually designs its own research strategies, we had had to wrestle with a very inflexible design that VC refused to adapt to market realities. Here are some of the things that went wrong and the issues other research suppliers and their clients need to consider so they never end up with such a nightmare on their hands.

• Be upfront from Day One, Minute One, as to what the project really involves.

Had we known about the computer program back in October 1997 we would never have quoted on the work. The kind of in-depth interviews we do involve discussing issues, strategies, plans, and trends and when we do obtain statistical information, we do not run it through a computer program. We accept that the data provided to us is usually provided in good faith, and catching people out - which is what this program attempted to do with the car dealers, who often mask exorbitant interest rates in their quotes - is not really part of what we do.

• Is your sample realistic?

One of the biggest hurdles we had to face in getting this project off the ground was getting the sample to shape up the way VC wanted it, namely, to slice the pie vertically, horizontally and obliquely. The first problem was that their cuts, in terms of city sizes, did not match with how list brokers cut their lists; few were able to provide lists segmented by a population size of 100,000-249,999. The next problem was the weighting. No matter how many ways we tried to get the world to come out to suit VC, it was not possible. When weighted by size, for dealers in population centers of 250,000+:<250,000, was="was" 1:6="1:6" Canada,="Canada," with="with" ratio="ratio" closer="closer" 1:15="1:15" in="in" United="United" States,="States," meaning="meaning" 150:150="150:150" split="split" not="not" accommodated.="accommodated." And="And" when="when" we="we" tried="tried" weight="weight" sample="sample" to="to" where="where" dealers="dealers" were="were" geographically="geographically" concentrated,="concentrated," none="none" of="of" the="the" other="other" criteria="criteria" could="could" be="be" met.="met."

And this didn’t even begin to get at the issue of the 50-mile radius; for example, the area known as the GTA (greater Toronto area) and Golden Horseshoe (the area west of Toronto, through Hamilton, down the Niagara Peninsula to the U.S. border) is where nearly 40 percent of the population of Canada lives and — surprise, surprise — is where most of the car dealers have chosen to locate. Once the 50-mile radius was imposed on this part of the country, a sizeable number of Canadian car dealers were knocked out of consideration.

On the U.S. side of the fence, even greater problems emerged in trying to distribute a sample of 300 dealers across 50 states, weighted to the three size-bands, the two population breaks and geographic dealer concentrations while allowing for the 50-mile radius limitation. It’s also debatable whether 300 is really a realistic sample for the U.S. While statistical reliability may be inferred from a number such as 300, the U.S. market is far too large and far too diverse to interpret broader issues such as financing patterns with such a small sample. As it was, we had to persuade VC to change its grand design so we interviewed only in the top 10 states, based on where the largest numbers of Chevrolet, Dodge and Ford dealers were located.

• Accept that the world is not designed to meet your objectives.

Developing specs without a reality check also creates ongoing problems in a research project. For TPFH, one of these concerned the number of actual dealers for the three automakers chosen. There are far fewer Dodge dealers in both countries; if a realistic approach had been used, a weighting closer to 60:30:60 (for each 150 calls) would have been used, instead of 50:50:50. The lack of Dodge dealers often made it difficult to come up with the trio required in any population center selected for interview. When asked, one of the VC consultants explained their rationale for picking these three makes of car: there was one of each dealer, within a short drive from his home in Collingwood, Ontario! He viewed this as typical of small towns in Canada and, by extrapolation, the U.S. The fact that Collingwood — a haven of Yuppie chalets and stratospheric real estate values — is probably the most atypical small town in Canada, didn’t seem to enter his thinking!

Another “real world” reality which clouded the picture is that, in parts of the U.S. and Canada, leasing is unheard of. As one dealer in Minnesota explained: “Lady, no one leases cars in these parts! I’ve been here for 20 years and have never done one!” This made it very difficult to come up with both loan and lease data.

• Are there other ways to achieve the objectives?

As much as researchers of all stripes like to make their work precise, the fact is, some adjustment is necessary; even scientists working in the lab have to change their experiments as they go along. One of the other obstacles which came up is that the Dodge dealers, in particular, would not quote loan or lease rates other than their own subvented finance rates. VC had placed a very firm parameter on the work that subvented rates were not acceptable, as these would not prove their point. The trouble was, dealers at all three of the auto makers often did not know what the comparable bank loan rate would have been. (And why should they? It wasn’t necessarily a requirement of their job.)

Whenever we tried to raise this issue, VC simply faxed back the original specs to us. This was generally true about any problems we raised and, as we were a subcontractor, not the research director, we could not make changes ourselves. For example, it would have been perfectly possible, if they had been willing to ease up on their specs, to phone banks in the various communities where we had interviewed dealers unwilling to give non-subvented loan rates and find out what the community standard was from banks operating in each locality.

Similar inflexibility on the part of VC was also a problem when we encountered sizeable geographic areas where every single dealer refused to discuss pricing and rates over the phone. No doubt this was either a dealer policy or a manufacturer policy, handed down from higher levels; in parts of Canada and the U.S. where the population was sparse — the 50-mile radius often pushed us out into the hinterlands to conduct interviews — it meant there were no further dealers to interview. There were also similar problems if we had interviewed one or two Chevrolet, Dodge or Ford dealers and then the third in that area turned us down. Again, in many parts of both Canada and the U.S., there were no dealers left to turn to when this happened. It also meant perfectly acceptable, completed work had to be discarded.

• Ranges versus rigidity.

VC also was very rigid on a number of other issues. Some dealers would only quote ranges for lease rates or loan rates and no amount of re-calling or skillful probing would persuade them to narrow this down to a set amount. When we drew this problem to VC’s attention, again, their response was to fax back their original specs — as if we hadn’t read them a hundred times already! Their computer program had also been set up so that only fixed lease rates, rather than ranges, could be accommodated in the cells in the spread sheet.

Then there was the issue of terms: many dealers refused to quote for 36-month terms. They were only willing to cite 24-month or 30-month terms. Researchers tried every tactic in the book to tease 36-month rates from these folks but they were unmoved. And who can blame them? Many were just following directives from higher up. Why should someone jeopardize their job for VC? (There were also cases where dealers were likely on probation, meaning they suspected our calls were from the head office, which was checking up on them. No wonder they wanted to play it by the book.)

We also hit a snag with the 10-basis-point tolerance imposed. Car dealers are not known for their precise quoting over the phone and so rarely did the two interest rates come this close.

We eventually told VC it would have to be at least 50 basis points, if they wanted the work done on time.

The key learning here is that, no matter how skilled the researchers or how much effort is expended, no respondent to a survey is obligated to give all the information you want nor to provide it exactly as you need.

What eventually happened?

We did, with Herculean efforts from everyone at both of our locations, manage to get the results to VC on time; another problem with their grand design was that they already had a set deadline for presenting the results of their own analysis (for which they needed our work), to a parliamentary committee. Not such a good idea; sometimes it is better to allow a longer lead-time rather than set the presentation date and then go out to sign up a subcontractor. Interestingly enough, not only did VC manage to present its work on schedule, as studies published by its banking industry association client and reported in the press revealed, they ended up cherry-picking the findings that best met their objectives of “proving” that the loan/lease rate spread is closer to 2 percent in Canada and to 1 percent in the U.S.

When we did our own analysis, we discovered that there really is not much difference and there are just as many wide gaps between loan and lease rates quoted by U.S. car dealers as there are by those in Canada. We also doubt if VC’s research methodology, although it went overboard on rigor, really reflects marketplace realities. Interest rates and dealer pricing are not driven by the number of financiers available. Most financial institutions and others who provide financing are subject to fluctuations in the central bank rate and global trends in interest rates, meaning there are forces at work which are bigger than any one single country.

And, conventional wisdom suggests that the way dealers price may have very little to do with what is going on in the larger financial marketplace. One suspects that the way it really works is that, when the auto makers realize they have a lot of vehicles coming off lease in the not-too-distant future, the word goes out to all the dealers to make loan rates look more attractive than lease rates; conversely, when the auto makers feel that they are not doing enough financing - most provide their own funds and do not need indirect lending from other financial parties - then the word goes out to lower the spread between loan and lease rates.

It would be interesting to hear from other research companies - assuming they can follow the convoluted research design for this project - as to what they would have done if approached with TPFH. And did VC really demonstrate valid findings or was the whole exercise really what we felt it was - statistical contortionism - to prove the client’s point?