Editor’s note: Cristina Mititelu is data processing manager at Ipsos-New Media Research, Bucharest, Romania.

Internet surveys are becoming more and more popular among market researchers. The benefits they offer can no longer be overlooked: cost efficiency, field speed and a variety of media testing options.

Of course, there still are debates about what works and what doesn’t work but some of the methods and techniques applied on the Internet have been tested and validated.

We know now that Internet surveys can be successfully used only on certain targets and in some specific types of studies where the research universe coincides with the Internet population.

Nonetheless, one of the greatest benefits of the Internet is its applicability in terms of multi-media research. Copy tests have practically been revolutionized. Several other kinds of studies are now run on the Internet, the most common of which are: employee studies, concept and product tests, customer satisfaction and business-to-business studies.

Data quality in Internet-based research

Issues like sampling representativeness and response rates are always raised when discussing Internet-based research.

There are two main sampling options one can choose when running Internet surveys:

  • quota samples, which are a form of non-probability samples;
  • probability samples - random, systematic samples in any sampling scheme, stratified or structured, as in offline research.

Most online surveys use quota samples, largely used because the Internet is a medium that can reach special targets where offline research proves to be expensive and less efficient: managers, telecom employees, early adopters, etc.

Quota samples are used when there is little known about the structure of the universe and when the researcher may want to interpret the findings in the light of certain different categories relevant to the study, purposely predefined by the researcher and of course under the subjectivity of the researcher.

Unfortunately, the Internet universe is not a good sampling framework for one reason: no one can have the full list of e-mail accounts in order to make a random selection. Even if this were possible, ethically this is out of question; there are very strict regulations toward data protection. This is why most Internet samples are a combination of availability sampling with quota sampling.

Probability samples are used when you want to apply the findings to a population whose characteristics you already know. Probability samples in any of their forms are the ones we question for representativeness. The issue always raised is “What is the sample representative of?” Is it the Internet population or is it the population of Sweden, for instance?

It can be easily said that the sample used in an Internet survey is representative of certain criteria but this is not enough to have reliable data.

Where there is a perfect match of offline population and Internet sample on background criteria it is very likely that the Internet population and therefore the Internet sample have a common trait that induce biased responses on several variables.

If the random samples are biased, weighting can be a balance option for many researchers. When the biases are great, weighting is a questionable procedure. Weighting cannot replace a good selection of sample and it cannot correct a sample if the selection was inappropriate.

What happens when response rates are low? In such cases the data are susceptible to invalidity. Data from both quota samples and random samples is at risk.

Key aspects

Here are a few key aspects to take into account in order to collect accurate data:

  • Work with good Internet panels. This is a rule that concerns both random samples and quota samples. What does “good Internet panel” actually mean?

a. The recruitment in a panel is very important. There are two types of panels: access panels and opt-in panels. Opt-in panels are those where the respondent did not reject the idea of being contacted in the future in order to take part in a survey or some other form of research. Access panels rely on respondents’ agreement to take part in future surveys on a regular basis. Access panels are more difficult to maintain but turn out higher response rates, 35-75 percent, while the opt-in panels result in only 5-10 percent response rates. It is always helpful to ask in the establishment survey for information about Internet connection: connection type, ability to view movies and other multimedia insertions with high fidelity. This is essential in copy tests.

b. Panels are a good resource if they have been randomly selected from a universe that we want to investigate.

- Use precise sampling selection and sampling methods of panels and samples.

- Send reminder invitations to increase response rate as much as possible.

- Apply correction tools in case of high non-response rates. Any inference from a biased sample to a population is susceptible to huge error.

Market researchers should constantly make the clients aware of the limitations of the Internet. The ultimate way to understand the comparability with the offline research is to run in parallel the same questionnaire using both offline and Internet surveys.

A few practical things to take into account when deciding to run online surveys:

  • Use the right software or a programming solution for the implementation. Some of the available software is limited, some is very flexible. The flexibility lies in the capability to handle multimedia insertions, multiple validation options, etc. The software must also support large databases.
  • Use a powerful provider. Simultaneous entries can crash the server that hosts the online survey. This is why you must know the capacity of the server and send the e-mail invitations in batches if needed.
  • Test the questionnaire in several browsers and resolutions.
  • Details are important in online surveys. The way the invitation letter, the reminder and the subject are phrased is essential. Information on how the information obtained in the surveys is handled makes the respondents confident and increases response rates.
  • Conduct a pretest. This is a stage of the research that few would skip in offline research. In many instances, the Internet is the last research option because the findings are needed as soon as possible and there is not time to conduct an offline survey. A pretest would disclose some of the programming errors.

Trends in Internet surveys

The latest developments in Internet research are very client-oriented and are technological rather than methodological. We have seen the emergence of many specialized companies in Internet-based research. Most of them have the logistic tools to deal in the Internet medium but have no knowledge in terms of methodology of marketing research.

When working with Internet research agencies it is essential that you not base your choices on purely technical services (questionnaire programming, hosting and server availability, panel availability and data processing capabilities). It is equally important to demand that the Internet research agency understands the purpose of the study and that they are able to select the sample rigorously and give you methodological details that help you understand if the data collected via Internet is valid.

Here are a few of the trends in Internet-based research:

  • Total transparency. The current tendency is towards total transparency to the client. This means that the client agency can monitor the time/times of the e-mail invitations and the questionnaire completion status. In addition, the drop-out rate, the number of completed interviews, pending interviews, interrupted interviews, the total e-mail invitations, etc., are all available in graphical representations from some Internet-based research agencies.
  • Less use of cookies. The use of cookies turned out to have a few disadvantages: respondents whose PCs were cookie-disabled could be interviewed twice. The respondents had to be warned about the insertion of cookies and the response rates dropped dramatically due the well-known reluctance towards cookies. The new respondent identification techniques do not rely on cookies, are much safer, and do not admit multiple interviewing of the same respondent.
  • History options. This new identification technique allows history tracking. What does identification mean here? This means that the respondents are associated to unique serial numbers. This helps the data processing analysts to check for possible multiple questionnaire completion by the same e-mail account holders. The history option permits a respondent who started the questionnaire but interrupted it for different reasons to restart the completion right at the point of interruption.
  • Real-time distribution of responses. This can be found as a separate module of the Web survey software or as a complementary service from the company that hosts and implements your survey. Client companies want more and more to have field control at any survey stage. Sometimes this is detrimental to the way the data collected online are analyzed and interpreted. Internet data need to be validated and only then reported and interpreted. Few such services or software programs apply corrections on data collected from biased samples or with high non-response rates.
  • Media protection. Since many copy tests are run on the Internet, it was compulsory to guarantee the security of the media shown. In pre-tests, run before campaigns are launched, one big demand of the advertising agencies but also of the campaigns’ beneficiary is that creative ideas and messages remain secret. Following the pressure of the clients, especially advertising agencies, there has been real progress in terms of preventing media download or media copying.
  • Reception accuracy. The size of the commercials was reduced for better reception and upload speed without diminishing the quality of the sound and image.

Much to be debated

With online research, there is still much to be evaluated and debated in terms of non-response analysis, sampling and research methodology. Nevertheless, the value of the Internet as a research tool is clear. Yet Internet research is still insufficiently exploited. It is important for market research agencies to make clients aware of the limits of online research in addition to the uses of the Internet as a complement to offline research.