Editor’s note: Dave Koch is vice president of Adapt Inc., a Minnetonka, Minn., research firm.
Technological advances have provided today’s researcher with many options for data collection. Both the old and the newer methods have their strengths and weaknesses. As a result, many researchers are utilizing several different data collection techniques to ensure they are reaching the right numbers and types of respondents. For example, 75 percent of the data collection on a survey project may be done using an Internet survey. The researcher may also choose to augment the sample with a mailing to insure representative response rates from a particular demographic that is less likely to be Internet-enabled. Additional phone interviews may also be done to round out the sample.
The end result is often data coming in from different vendors, at different times, and often in different formats. This can result in a compatibility nightmare for data processing if you are not careful.
Most of these types of problems can be avoided with good planning at the very beginning of a project. However this can be difficult in today’s work environment. Researchers are often so busy in the initial stages of getting the project ready to field that all back-end processing details are put on the back burner. This may seem to make sense as you rush to meet deadline pressures and deal with last-minute client changes. However the price can be very high in terms of your time, the use of company data processing resources, and even project quality.
Examples of problems we have witnessed include:
- The Internet vendor supplied the data in an Access format, the CATI vendor supplied the data in a delimited ASCII format, and the data entry vendor supplied data in a column binary format - the end result being needless headaches for the data processing department.
- The rating scales for a paper questionnaire were captured as most satisfied = 5 and least satisfied = 1, while the Internet supplier programmed the scales exactly opposite with most satisfied = 1 and least satisfied = 5. The error was not caught until the tables were run and the discrepancy became obvious. Once again, lots of wasted time in data processing, as well as a potential data quality problem.
- Open-ends from a paper study were coded using a code list developed by the vendor. On a later portion of the study, fielded via CATI, coding was done by an internal department. Not realizing the data needed to tie back to a prior segment, the coding department renumbered the code list completely so they could use two-digit rather than three-digit codes. The end result was that none of the coded data matched, and the open-ends had to essentially be data processed as two separate projects, taking twice the time.
As these examples demonstrate, the cost of failing to manage multiple data sources properly can be very expensive. The key to success is taking the time to carefully consider back-end processing requirements when you first begin setting up your project.
Here are some tips to keep in mind next time you are planning to field a multimedia study.
1) Make sure you have control of the data layout. Create a data map at the same time you finalize the questionnaire. All supplying departments and vendors should commit in advance to supplying collected data in a set format, and in the exact layout you specify. You should have the option (although you may not use it) of combining the files from all different suppliers into one file for data processing. Some key items to keep in mind are:
- Be sure to include specific rules on capturing data. For example, how will multiple answers be handled? Or how many codes will be allowed for each open-end that is coded.
- Make sure all rating scales are consistent. Don’t leave room for your scanning vendor to assign a 5 to extremely satisfied and a 1 to extremely dissatisfied, while your Internet provider does exactly the opposite.
- If open-ends will be captured, make sure they will be supplied in the same format by all vendors or supplying departments.
- Be specific on file formats. Your data processing people will not want to receive files from one vendor in fixed-field ASCII, and another in an Access database.
2) Bring the data processing people (if it is not you) in on your project in the initial planning stages. If your company has several programmers, one should be assigned to a project in the development phase rather than at the last minute. This way data processing can clearly define their data needs before the project goes to field.
3) Centralize back-end processing as much as possible. For example, have all the coding done by one vendor, or by your own coding department. This will lead to increased efficiency and better consistency in how data is handled. The same applies to data cleaning and tabulation. Once again, make sure all of your data suppliers are committed to a common format and layout.
4) Pay close attention to timing. Note that the fielding time for the mail or phone portion of your study will be different than that of the Internet portion. Make sure that the timing of the fielding of all methodologies is coordinated to meet the requirements of your study.
5) Be careful to proofread all survey documents very carefully, and reference the documents for other methodologies often. In other words, make sure that the exact same questions are asked on your mail study as your Internet portion. The same verbiage should be used whenever possible to avoid influencing the respondents’ answers in any way. Also, make sure that any concept or stimuli information that is presented to the respondent is the same.
Plan and document
The key to successfully managing the data from a multimedia study is to plan and document your data requirements upfront, before anything hits the field. Make sure that all supplying departments and vendors have a very clear picture of exactly what you are looking for as a deliverable. If they cannot commit to supplying it, you may want to consider another vendor. The end result will be more efficient use of the valuable data processing resources of your company and potentially shorter lead times to delivery of final reports.