Do some online interviewing of your own

Editor’s note: Janet Savoie is senior director, client development, at Online Survey Solution, a Nashville operating unit of M/A/R/C Research, Irving, Texas.

For four years I have purchased the same brand and model of running shoes, the New Balance 991. However, after my second knee surgery I decided I might be due for a change. I went shopping for a shoe that was a perfect fit for my running style and would give me the support I needed. I was amazed by all the choices I had. I didn’t know where to begin.

I often get the impression market researchers feel the same way when they are looking for the perfect online data collection vendor. In the last three years the number of companies offering a variety of services has tripled. Figuring out how to differentiate between the offerings and the quality of those offerings is no small task when researchers are being called on by so many different companies. Because I was wishing I had my shoe issue of Runner’s World with me as I was staring at the racks of running shoes, it dawned on me that it might be helpful to develop a checklist of questions to ask potential online partners. I came up with these questions by listening and asking - listening to my prospects as I spoke with them for the first time, and directly asking people in the industry, who do a lot of online, to tell me what is important to them. The same issues came up over and over again and it was not difficult to narrow it down to the following 10 areas of concern.

1. What services does your company provide?

This is the best place to start because there is great variation among what vendor companies are offering and what research companies need. There are service bureaus offering one-stop shopping. Deliver a final survey to them and they will deliver data to you and do everything in between. They supply sample or work with client lists, they program and host. Many also offer tables, coding and weighting. But not every company needs or wants this full-service option and there are many choices for just programming or just sample. Match the offerings to your needs and then ask the more specific questions.

2. How large is your panel?

This seems like a simple question, but panels actually count their panelists differently. The number can represent households or individuals. Panels counting households assume they have access to everyone living in that household for surveys through the individual who actually signs up for the panel. For instance if they have one million households they would assume perhaps 2.5 million potential respondents. Panels that count individuals may have more than one person in a household on their panel but they do not track that.

The size of the panel is especially important if you plan to do large tracking studies that require unique respondents for every wave or if you are looking for low-incidence groups. If you are doing research overseas, what is the panel’s international reach? If they do not have their own panel do they have reliable global partners?

3. What do you know about your panelists?

The more information known about respondents and their household members, the better it is for the health of the panel and for panel buyers. If consumer buying behavior, hobbies, occupation, etc., can be targeted, fewer e-mails have to be sent out - which cuts down on the number of invitations each panelist receives. It is good for researchers because it helps keep down the cost of low-incidence studies. Most panel companies can supply you with a list of their targets. Usually the list is long but they may have only screened a portion of the panel on each target (except for major demographics). What this means is that there may not be sufficient numbers to do the size study you need to do, especially for B2B. B2B is still difficult online but for some specific groups such as physicians, small-business owners and C-level executives, panel companies have invested in developing sub-panels. Find out what is available or if they offer a phone-to-Web option. Many panel companies regularly re-screen panelists for specific information, so a good follow-up question (if you have a group you routinely survey) is to ask if there is a way to get a question or two on soft exits or on screening questionnaires.

4. What steps do you take to make sure I have a representative group taking my survey?

Most panel companies can claim that their panels stack up well against the Census, except for the elderly, the very low-income and the minority segments. But the base representation of the panel is not the key issue. What is important is the number and variability of the sources for the panelists and the process used for pulling a representative sample. One issue to be aware of is that when sample is pulled, the pull is usually based on three variables - age, gender, geography - and it is not pulled by race unless it is specifically requested by the client. There is nothing wrong with this, but from my experience, clients aren’t always made aware of it, and they get frustrated when they get their data back they often have only 2-4 percent African-Americans and Hispanics. In the client’s mind a rep sample is representative on all demographics, not just three. Find out how many and which variables they are using. Also make sure to verify that when they are pulling sample that they take into account the response rates of the different segments of their panel.

5. How do I know if Jane Doe really filled out my survey?

This is a hard question for panel companies to answer as there is no equivalent to phone validation in the online world. Some panels have a system in place where they re-screen for demos that should not have changed, like gender and year of birth, and compare that data to what is in the respondent’s record. If there is a discrepancy they can delete the survey. This process needs to be combined with password protection so that Jane Doe cannot take the survey multiple times as different people and then send it to John Smith so he can take it. Passwords insure the survey assigned to Jane can only be taken once.

6. Do you limit the number of surveys respondents can receive?

Everyone is worried about professional respondents, and most panels claim to have a strict policy of survey management. Some companies limit the invitations sent to a respondent over a certain period of time; others rest panelists when they have taken a certain number of surveys. Find out what the policy is and then join the panel using a personal e-mail address. That is the best way to determine their commitment to their policy. In their defense, panel companies are finding it more and more difficult to control survey volume to their panelists because the demand for online research has dramatically increased over the last few years. The demand has grown faster than the panels have grown. Ask what their plan is to balance supply and demand for sample with their panel management policy.

7. What information will you need to give me pricing?

Although incidence, length in minutes and number of completes is what is minimally necessary, companies will usually require complexity level and programming length as well as respondent length and specific quotas. The more information you are asked for the more likely the price will be accurate. If you are asked all the right questions up front, there is lower likelihood of price issues down the road. Of course if your specs change before or during fielding, the price will also likely change. In general, incidence is the responsibility of the client, not the vendor. Most companies are willing to give pricing for several incidence levels if the client is unsure.

8. Will your programming platform meet my needs for complexity and timing?

The ability to program specific types of questions (discrete choice), to handle complex quotas and to offer cool tools (card sorting) is an important area to explore. Ask if they have a survey demo. You can get a good feel for their capabilities and a good look at how they present the questions to the respondent. This is also a good time to inquire about the experience level of the programmers on staff and where they are located. Are they in-house or overseas? Will you have access to them? Will they be on the kick-off call? And, since timing is often an issue, what is the typical turnaround time and can you accelerate a schedule if necessary?

9. Who will be managing my project?

Once a project hits the field, the day-to-day contact is with the project manager. If you are looking for a company to partner with, you probably want to develop a relationship with one or two project managers who will be a constant on all your jobs. First find out the experience level on the team in research and in managing research projects. If you do very complex work, ask if you can be matched with a project manager who can handle that level of complexity. This is a key relationship: You want someone who understands your requirements and can anticipate your needs. If you work well with a particular project manager, can they be assigned to all your projects? Will there be a back-up person assigned?

10. How do you price your services?

Does it always come down to price? Clients say no, but they want to understand how things will be priced and if it is a firm price. Are the charges per hour, per complete, per amount of sample used? Can costs be broken out so they can compare between vendors? Are there hidden costs? One universal complaint is being nickel-and-dimed to death. Clients expect price changes if the length and incidence change dramatically, but they don’t like surprises that knock their budgets out of the water.

The right fit

Asking the questions that reflect what is important to you is crucial when you are interviewing a potential online vendor. Does the company have the resources to do the job the way you need it done? Just like shopping for the right running shoe, you need to make your choice based on the right fit for your needs. But in either situation, the proof is ultimately in the pudding. Whether you are out on the road for the first time or partnering on that first project, what matters most is if promises are kept and it feels good. Hopefully, asking these questions will help you narrow the field down to those who are potentially the best options.  | Q

Â