Delivering the desired results
Editor's note: Dan Wiese is president of Dan Wiese Marketing Research, Cedar Rapids, Iowa.
A course in statistical methods was part of my statistics major program at Iowa State College. Mail surveys were covered in the course – for about six seconds. “We won’t bother with that since you can’t get a decent return percentage,” the professor said. Imagine my surprise when I went to work for Meredith Publishing a few years later only to learn how wrong the professor was.
We did mail surveys with subscribers of Better Homes & Gardens and Successful Farming and achieved response rates of 80 to 85 percent. The last one I did for Successful Farming several years later got 75 percent. These days, employing all the tricks, I can still get 50 percent. One approach is the use of an incentive – typically a dollar bill – with the survey letter. The dollar adds little weight to the postage but carries a lot of weight toward the success of the survey. It’s too small for people to consider it payment but it gets their attention so they will read the cover letter – and that’s paramount because the magic is in the letter.
Recently I was hired to do some research by the city of North Liberty, Iowa, and my first recommendation was to do a mail survey. North Liberty is just north of Iowa City and is one of the fastest-growing municipalities in the state. The city had some specific needs regarding demand for public transit and had tried to do its own surveying previously, without success.
Dean Wheatley, city planner for North Liberty, explains the purpose of the project: “We needed a good unbiased understanding of how residents feel about specific existing and potential city services. In these times of lean budgets and ever-increasing demands, it is important to focus attention where it is most needed and not to try to be the provider of everything every person or group is interested in. For this survey, we were expecting to provide assistance to a transit committee and to a parks and recreation committee by helping to narrow and define public interests. We also asked questions to help staff and the city council understand the level of satisfaction with existing and potential services and projects.”
On a volunteer basis
An earlier North Liberty survey only covered public transit and people were invited to complete it on a volunteer basis through various avenues – the city’s Web site; by picking them up at city hall; and by copying it out of the local paper. As you might expect, only people with an interest in public transit bothered to take the survey - 104 out of a population of about 10,000.
I knew that a mail survey could certainly do better than that but doing mail surveys for municipalities precludes the use of the dollar bill. Taxpayers don’t look favorably upon their elected representatives and city officials wasting tax dollars on that kind of foolishness. It kind of starts the survey cooperation process off on the wrong foot. So in order to increase response rates, my advice was to field a survey covering a range of subject matter, so there is something of interest to everyone.
What response can you expect from citizens when their leaders want their opinion? My experience has been that 35 to 40 percent is the best you can achieve, assuming ideal conditions - good questions people are glad you asked; citizen respect (or strong disrespect) for leaders; willingness by the leaders to pay for the mail survey tricks. Make no mistake, cheaper is not the reason to recommend mail surveys. Proper mail techniques are more costly than people expect and doing things the right way can put mail survey expenditures on par with those of telephone surveys.
But mail bests telephone in its ability to cover lists of items to be rated. While telephone interviews can turn into drudgery having to read lists and reread rating scales, everything is clearly laid out in front of the respondent in a mail questionnaire. Another plus for mail surveys is that everyone has an address. This gives me confidence that I’m dealing with random sampling. I’m not as confident as time goes by with telephone interviewing. Not everyone has a landline anymore and private numbers have always been a problem, although less so in my part of the country. A large and increasing percentage of households, especially younger people, only have a cell phone.
To obtain a random sample it is necessary to employ hybrid methods that combine landline and cell phone samples. While no marketing research call is heartily welcomed these days, this kind of call coming to a cell phone is actively resented, according to anecdotal evidence. This certainly calls into question the quality of what you can manage to obtain in the interview. Web-based surveys, although quick and inexpensive, are definitely out for a citizen survey such as this one. Too many people have no access to the Internet and it precludes random sampling.
I like Web surveys a lot when I have a list that properly represents a universe, such as an employee roster, but that’s definitely not the case for municipal citizen surveys. Trying to develop a hybrid method for these kinds of surveys is definitely overkill, adding unnecessary complication and cost to what can be a very straightforward solution.
Built-in biases
Every method has built-in biases. Web and cell phone surveys skew toward younger people. Mail survey response is more likely to be from older people, the more thoughtful and those who tend to be more patient. Interestingly, these characteristics tend to describe those with a higher likelihood of voting. It follows then that mail surveys will tend to do a better job of measuring the attitudes and predicting likely behaviors of voters. Like it or not, it’s important to know what voters think. People with a problem and those who are otherwise negative will always be much more vocal than those who are happy. It’s pretty easy to get knocked off balance by all the noise from these folks and wonder whether their views are valid and merit some sort of action. On the other hand, inaction on an issue a lot of folks are worked up on has its risks as well. Make no mistake; we’re talking about getting reelected here.
I’m not in love with mail surveys in spite of how it may seem. As a consultant who only charges for his time, I really don’t care what method I recommend, only that it is best for the job at hand. Mail surveys have some downsides and are not good for every situation. They are unsuitable for measuring awareness, for example, since there’s no way to prevent people from looking things up so they can appear to be “smart” on the survey. The recipient of the survey is also able to inspect the entire questionnaire before deciding to answer it or not. They also take longer to execute than telephone or Web surveys.
Existing alternatives
In the best interest of the client I usually look for existing alternatives that might meet their needs and I did so in this case. There are at least two firms that I am aware of that specialize in these kinds of surveys for municipalities. I’ve seen their work and they do an exceptional job. They pretty much pull out all the stops from a technical standpoint. Their reports are clearly-written and well-presented with many excellent and useful features. They definitely cover the waterfront of concerns for any municipality and they each have a long list of clients. Their questions are largely standardized, so they are able to show trends through time for yearly or every-other-year waves of interviewing and can show a municipality’s scores versus all other cities or versus cities of their size. This is a highly useful feature. I would not hesitate to recommend either to a client whose need fit their services.
However, their one drawback is affordability. Large municipalities obviously feel they can afford the services, based on the firms’ client lists. While a few smaller cities and towns are customers, most would find it too expensive. North Liberty certainly fell in that category, so I designed a streamlined approach very similar to what the other firms do, as it turns out, but much more affordable. I charged North Liberty $2,600 for planning, several meetings, material development, advice on methodology, data entry, tabulation, a written report and a presentation. They took care of printing, assembly and postage and paid for those themselves. Take away the need for meetings and travel and handle everything by e-mail and my charge drops to $2,000.
The specialty firms employ mail surveys, too, but I did a simple random sample of households, while they use stratified random samples. They make fairly extensive efforts to do follow-ups; I only did the original mailing with no follow-ups. I can’t disagree with what they do, but the results bear out my streamlining of the process. In the North Liberty project, we mailed out to a net of 1,005 after undeliverables. We received responses from 334 households at the cutoff, a 33.2 percent return, with no incentives used.
Worked well
For North Liberty, the mail survey worked well, Wheatley says. “Using the survey results documentation we are able to honestly and factually report and act on the priorities among the survey questions posed. It helps us respond to anecdotal accounts and claims by enabling us to cite statistically valid resident interests.”
As he points out, when municipalities endeavor to learn their citizens’ opinions, it is critical that this learning is achieved through scientific, projectable surveying. The survey must settle the issues and not leave room for continued argument. I love it when people say, “Well, it wasn’t a scientific survey.” Then why did you bother? Nothing you could count on was learned. The object is not to use the latest trendy method; it is to learn truth you can trust.
Further, working with a professional researcher gets a third party involved, so that it’s not city officials who get attacked by those who don’t like the results. A professional researcher can provide a cogent and understandable defense of the findings.