Editor’s note: Howard Waddell is president of Decision Resource, Inc., a Miami research firm.

The salesman at the dealership where I recently purchased a car told me that I would be receiving a call from “the home office” during which I would be asked about my overall experience with the dealership and about him in particular. “If you don’t give me the highest marks...” he said, and he proceeded to draw his outstretched fingers across his throat, execution-style.

Only a few weeks later, a factory-authorized appliance repairman, after finishing his work at my home, said I would be receiving a questionnaire in the mail about the service call. He said that getting the highest ratings was important to his career. I told him I would take care of him.

One of my clients, a senior executive at a local cruise line, once expressed some puzzlement about why past cruisers don’t take cruises more often. He explained that surveys conducted among disembarking passengers showed that virtually everyone rated the cruise as superior in every way. It wasn’t until I took a cruise myself in 1999 that I was able to shed some light on the issue.

Near the end of the cruise, there was a meeting in the ship’s theater to explain to the passengers what they needed to know about U.S. customs and immigration, tagging baggage, finding their baggage on the pier, and so on. The cruise director also explained that we would all receive a questionnaire about our cruise experience. He simply said that if we enjoyed the cruise, we should check off the highest ratings for each of the several attributes listed. Later in the day I spoke with our dining room tablemates and asked them how they rated the cruise on their questionnaires. They had followed the cruise director’s instructions precisely.

Only wonder

Based on the data that is being collected in the name of research, one can only wonder about the quality of the marketing decisions that are being made in the executive offices of corporate America. When senior managers are not fully aware of the true levels of customer service their businesses are providing, it is no surprise that good customer service is becoming increasingly difficult to find. But the problems of research are not limited to measuring customer satisfaction and they are not limited to poor methodology.

Too often, the questionnaires themselves are poorly written. It is apparent that there are more than a few professional market researchers who have totally unrealistic expectations of what people can remember about the routine things they do in their daily lives. Consider the following questions I have been asked on mail and telephone surveys. (And yes, I lie when asked if I work in the marketing research industry before the “real” questions begin. I know I am not alone in this regard.)

Question in a mail survey: In the past three months, how much have you spent on movies you saw advertised in the newspaper?

Did the researcher who wrote this question really believe people could provide anything other than a wild guess? Not only do I not know how much I spent in total on movies in the past three months, I certainly don’t remember which of the movies were advertised where. (And, by the way, should I include tickets I bought for me alone or for me and my family?)

Question on a telephone survey: Of your last 10 drinks of scotch, how many were at home? At a friend’s? At a restaurant? At a bar or tavern? Elsewhere?

Sorry, but my last 10 drinks of scotch span a period of no less than two years and I’m afraid I neglected to fill out all the columns in my alcohol intake logbook. Scotch will do that.

There are also the merely dumb questions. In a recent telephone survey I was asked: Would you confirm your age? My response: Yes, of course. How old do you think I am?

Then there are the carelessly constructed questionnaires. One mail questionnaire I received presented a potential new product from a well-known consumer packaged goods manufacturer. After the product was described in some detail, a question asked about the likelihood that I would purchase the product. In my case the likelihood was low (“probably would not buy”). However, later in the questionnaire I was asked how I would use the product, when I would use it, and at what price I felt the product was so inexpensive that I would not purchase it because of quality concerns. One might think the author of the questionnaire had never heard of skip instructions.

And there are the questions that are written by people who are either very careless or who live lives that are very different from my own. I offer two examples from a mail panel questionnaire. I did not make these up.

Question: How many times in an average day do you apply your usual underarm product? One to two times per day? Three to four times per day? Five to six times per day? More than six times per day?

Question: How many times in an average day do you shower/bathe? One time per day? Two times per day? Three times per day? Four times per day? Five or more times per day?

I believe in good grooming but this is ridiculous.

There are also the problems associated with the task of data collection. During one phone survey, I was asked about my awareness of several companies in a particular industry. The industry was one with which I had little familiarity and I didn’t recognize the first several company names. Finally, I heard a name I knew but, before I could answer, the interviewer said to me, “Surely you’ve heard of them!” Talk about intimidating the respondent!

More recently, I was listening in on telephone interviews that were being conducted by a large phone center. To my dismay, the interviewer (who knew I was listening) was paraphrasing some of the questions.

Variety of problems

We read from time to time the laments of corporate market researchers: they don’t get the respect they feel they are due; or their research findings are not getting the attention they warrant. But given the frequency and variety of problems with survey methodology, questionnaire design, and data collection that I have seen recently (and I do not participate in that many surveys), I am not surprised that the research function in corporate America is not always held in high esteem.

So, what is the solution and how is it achieved? Clearly, part of the solution is to ensure that researchers have the necessary skills. First, those who are given the responsibility of writing questionnaires must have a superior command of the language. They must be able to write simply, clearly, and precisely. They must also be able to proofread.

Researchers must be able to place themselves in the role of the respondent when writing questionnaires. They must have the ability to comprehend how someone else might interpret the language they use in a questionnaire. They must be able to anticipate how people might respond to a question. They must understand the difference between a reasonable and an unreasonable question. They must understand what respondent fatigue is. Of course, questionnaires should be pre-tested even if only among others in the office.

Researchers must be able to anticipate what can invalidate their findings and take corrective action. How many car salesmen, appliance repairmen, and cruise directors are instructing customers how to complete survey questionnaires? If the risk of outside influence exists, it might not be unreasonable for the questionnaire to ask: “Has anyone in the [company] organization talked with you in any way about your possible answers to this questionnaire?” The questionnaires where the response is “yes” can either not be counted at all or segregated from all other responses in the data processing.

Researchers must be able to tell the CEO that the questions he or she wants to ask may not be good ones or will make the questionnaire too long. How often have we heard the phrase “As long as we’re talking with these people…”?

Finally, there are the roles of the phone center and the mail-panel company. In the case of the questionnaire that asked about underarm products and bathing frequency, I had to ask myself if the project directors in this well-known mail-panel company blindly printed the questions their client provided to them. If they did, shame on them. And if phone centers don’t review the questions before the interviewing starts, they should at least ensure that the questions are being read as written. Paraphrasing and editorializing by interviewers is not acceptable.

Have an obligation

We have all read about New Coke and other marketing gaffes that were the result of flawed marketing research. It would seem, though, that flawed research is not that uncommon. As members of a research community that is seeking greater recognition and appreciation, we have an obligation to our employers and our customers to provide the best market information that we can while not wasting money in the process. Ultimately, we must take personal responsibility for the quality of every research project we undertake. Those who take that responsibility seriously will themselves be taken seriously.