Thoughts from the technicians

Editor’s note: Tim Macer is managing director of U.K. consulting firm meaning ltd.

The perils of online research in general and panels in particular featured heavily at this year’s conference of the Association for Survey Computing (ASC). Held in September at Southampton University in England and themed “The Challenges of a Changing World,” the conference offered 40-some presentations, many of which were refreshingly candid on just what responding to the challenges of speed, the changing technology of Web 2.0 and respondent engagement can involve. Unlike some of the more staged industry events, at the ASC conference speakers were happy to cover the rough and the smooth. It made me uncomfortably aware of just how much there is still to understand in the practice of online research and how many unintended consequences can flow - often invisibly - from seemingly innocuous choices at the design stage.

The ASC event is known for being much more direct and hands-on than other industry events. Technicians, after all, have to be practical, and if anyone is used to sorting things out when they go wrong, it is the people in data processing. It no doubt helps that ASC is not wholly focused on marketing research either - government statisticians and social researchers, who use many of the same tools and methodologies, are also major contributors, and many of the lessons learned are universally applicable.

In his keynote address, Mick Couper, research professor at the Survey Research Center, University of Michigan, drew a useful distinction between what he identifies as three distinct strands of Web 2.0: the social Web, the interactive Web and the mobile Web.

While acknowledging that, “for market researchers in particular, these social networks offer insight into how people think and what’s hot,” he warned of the danger of getting carried away with the size of the membership of such networks as Facebook and Second Life and ignoring the fact that these people are, by their makeup, often technological elites. The opportunity presented by the social Web “is not really survey research as we know it.” The interactive Web “has much more direct relevance for survey research as we know it.”

The key question, according to Couper,   is “...whether increasing interactivity and richness of the media improve the quality of survey data, or the survey experience in general. Thus far the answer may be sometimes yes and sometimes no.” He cited much contradictory research in the area and urged researchers to resist leaping to build ever more creative components into surveys as a way to imitate the top multimedia sites. Instead, he suggested “using the new features of the Web 2.0 world to improve survey measurement” such as to enhance the survey experience, improve flow and “grapple with the key challenges facing representative sample surveys.”

This was put to the test by Nicola Stanley of Silver Dialogue and Stephen Jenkins of Snap Surveys, both of the U.K., who shared the results of experimental work they had undertaken into asking rating-type questions using the kinds of interactive components Couper spoke of. This included such graphical feedback devices as sliders with faces that went from a cheery smile to a frosty frown, thermometers with draggable menisci and two-dimensional mapping using drag-and-drop.

The duo were interested in comparing these graphical questions in terms of respondent engagement, usability and the accuracy of the results compared to conventional surveys. A control group was given the same survey with conventional text-based grids and radio-button selections. Each group comprised the first 112 completes from a sample of 300 adults, matched for age and gender.

Not surprisingly, there were many differences to note. The graphical survey did appear to appeal more to the 24-35 age group than older respondents. By all measures, the graphical survey took a little longer to complete but respondents reported they enjoyed it more and appeared to be more engaged. Taking longer is not necessarily a bad thing. One side effect of the use of graphical components appears to be that it forces respondents to pay greater attention, and this focus gives rise to both “engagement” and more care over the responses. This was reflected in respondent comments too.

For accuracy, graphical scales seemed to accentuate the positive: respondents scored items significantly higher when dragging a thermometer than when using a conventional drop-down box. Brand rankings were similarly boosted when respondents dragged and dropped them into a 2-D contingency table, rather than clicking on wordy scales. Which score more accurately reflects the “truth” is impossible to say. Overall, it seems that it does make respondents happier and possibly more engaged, for the cost of the extra design and setup involved, but care is needed in interpreting the somewhat rosier findings that can result from the ratings.

Respondent trust

Earning and engendering respondent trust in your survey featured in a paper from Adam Joinson at the U.K.’s University of Bath, on privacy, trust and the factors surrounding respondents’ willingness to self-disclose in Web surveys. Based on research he carried out, Joinson reported that “survey organizations ignore issues of privacy and trust at their peril.”

Yet the factors that influence trust are simple and refreshingly inexpensive to improve. Reputation is a strong factor in establishing trust, but so too is making sure that the survey organization appears open, accessible and “real” to the respondent. A street address and a geographically-located phone number, pictures of buildings and even named people within the organization will all help build trust.

Joinson warned against what he had identified as triggers to lack of trust: spelling mistakes in the survey, poor graphics and designs, etc. As soon as respondents perceive a lack of competence, trust will fade. Bottom line: Survey companies need to demonstrate their competence to respondents or panelists in the surveys they present.

The factors around privacy are more complex, and respondents are often contradictory in their stated approach to privacy and what in practice they are willing to reveal. But the perceived reputation of the research company is critical here too: publishing a clear privacy policy and making statements about privacy do all help, it seems. 

Problems with panels

Less easy to fix, according to several speakers, is the potential damage to the whole industry wrought by problems with online panels. Reg Baker, from the U.S.-based research company Market Strategies International, gave a bleak assessment of respondent behavior in panels in a paper entitled “Separating the Wheat from the Chaff.” He identified three types of what he termed “new online panelist pathologies.” Starting with the “hyperactive” respondents who belong to multiple panels, he noted, “In our experience the levels of hyperactive respondents can vary substantially both among panel companies and even within surveys using the same panel company.”

“Inattentives” make up the second group: respondents who give less than full attention to the survey, and who are inclined to skim through the questions, “straightline” through grids and leave many questions unanswered.

It is a phenomenon Baker and others at the conference referred to as satisficing - the hybrid term combining satisfy and suffice, which effectively means doing just enough to get by - which in this case is the shortest distance from survey invite to points rewarded. The methods Stanley and Jenkins had earlier described for better engagement, in addition to simply adding timers and polite warning messages, were among the kinds of technical remedies Baker proposed for slowing the satisficing respondents down.

“Fraudulents” make up the third group, the “gamers…who assume false identities or simply misrepresent their qualifications” and with whom Baker considers the MR industry has become overly obsessed. As he pointed out, interview fraud is not exclusive to online research, but is easier to detect there. He applauded the emergence of industry best practices attending to verification and deduplication such as the ESOMAR 25, and associated ISO working group on ISO 20252, which will reduce the scope of fraud considerably.

“One cannot help wonder if the very success of online panel research is not now the cause of its potential undoing,” he mused, citing that, after deduplication, around 18 million adults are now members of online panels in the United States - around 9 percent of the population, of whom an even smaller core participates regularly in response to the invitations they receive.

He concluded: “The most obvious next step is to find ways to encourage a greater share of the population to join panels and participate regularly. Less obvious is some alternative means for getting the respondents we need…that not only increases the size of the respondent pool but also allows us to reduce the bias that has always been inherent in the volunteer panel design.”

“Waning star”

This notion of representativeness in panels was considered in more detail by Jelke Bethlehem of Statistics Netherlands and Ineke Stoop of SCP in the Netherlands in a paper entitled “Online Panels - A Paradigm Theft?” Bethlehem gave the figure 2.7 as being the number of online panels an online panelist appears to belong to in the Netherlands. He is concerned that the MR industry is staking its future on Internet research based on online panels, where the signs are already that this is a “waning star.”

Bethlehem and Stoop had researched the composition of panels and their success in predicting electoral outcomes in the Netherlands, and discussed this in the context of background assumptions about the very nature of sampling. Their conclusions are that the theoretical problems long identified in using either self-selecting or convenience samples as a proxy for random probability samples are now being borne out in the accuracy of the results being obtained from online surveys.

Without the gold standard of a random probability sample from the whole population behind these results, Bethlehem stated there is no theoretical basis on which to confer any measure of accuracy, or test that the results are statistically representative. “A large panel is no guarantee for accurate estimates: small groups are often very underrepresented,” Bethlehem said. “And while response from the panel may be high, response during recruitment was probably very low. It is response during the recruitment response that matters.”

He continued: “Where representativeness is not an issue, online panels may be a useful tool for exploratory studies,” but not for point estimates or the “size of relationships.” Otherwise, he stressed the need to address the population who are not on the Internet or not active on the Internet, and incorporate them through using mixed methodologies. He urged panel providers to be transparent with their clients and explain the limitations of self-selecting samples - and to educate the media and other transgressors to reserve the term “representative” for samples where sampling error can actually be measured.

These and other papers from the conference may be downloaded free of charge at www.asc.org.uk