Of the many lessons being learned in these heady early years of on-line research, one of the best I’ve come across is: just because you can do something doesn’t mean you should do it.


This maxim arose from the experiences of Atlanta-based Turner Entertainment Group and its research partner, Cincinnati-based Burke Interactive, a division of Burke, Incorporated, as they developed and implemented Burke’s Digital Dashboard™ method for Web site research. Candid interviews with two key participants illuminate the shaping of an ambitious idea, the efforts to get it off the ground, and the regrouping that occurred after some important realizations.

In 1997, Burke Interactive began conducting some individual research projects on the Web sites of some of the many entities in the Turner Entertainment Group, which includes cable networks such as TBS, TNT, Turner Classic Movies (TCM), and the Cartoon Network. The information from that research was valuable in helping the Webmasters in the various business units change their sites.

(Viewers visit the Turner sites for everything from obtaining inside information on their favorite TV shows to participating in interactive simultaneous TV/Web events such as NASCAR races during which fans can watch the race on TV and view the action through in-car cameras on the Web.)

In 1998, Burke Interactive pitched an idea it called the Digital Dashboard, which would, in real time and on an ongoing basis, answer a host of questions about the Web sites for several entities within the Turner Entertainment Group, from Webmasters to researchers to marketing people. Site visitors would be sampled on an nth basis to complete a questionnaire covering everything from satisfaction with the site they were visiting to their media consumption habits.

A main goal was to profile the site visitors, says Fred Spring, director of research for TCM. "Assembling an audience profile is easier in the TV industry because we have Nielsen ratings, but in the Web world it’s a little more difficult. You can get a sense of it from some of the feedback you get but we really wanted a more thorough profile," he says.

"There are a lot of sites out there that don’t pay attention to who’s consuming the content," says Dan Coates, vice president, Burke Interactive. "A media company like Turner is all about demographics. So for Turner, it all begins with looking at who is visiting their site and seeing if that aligns with their on-air market."

Spring also wanted to get satisfaction readings, understand why visitors came to the site, uncover any gaps between what site visitors wanted and what they were getting, and also examine functional, navigational and design issues.

Gauge interest

Adopting the role of champion of the Digital Dashboard idea within the Turner Entertainment Group, Spring began meeting with the various factions throughout the company to gauge their interest level and buy-in for such a comprehensive approach. "Digital Dashboard was a departure from the one-time studies we had conducted. It looked at collecting, on an ongoing basis, 5,200 surveys year [100 per week] for each site, rolling them up and presenting them against some benchmarks," Spring says.

"After collecting a ton of information," Coates says, "we would be able to give the particular audiences within Turner Entertainment Group the ability to drive the car, as it were. That’s why we used the digital dashboard metaphor. They could make changes to their content in week one and be able to see how that impacted their audience profile and site satisfaction ratings in week two."

Practical problems

As early iterations of the Dashboard idea were circulated around Turner Entertainment Group, some basic, technical problems cropped up. People didn’t have the latest versions of Netscape Navigator or Internet Explorer, which were needed to use the Digital Dashboard system. The interface didn’t work well on Macintosh computers.

Burke worked to overcome these hurdles but the problems raised early flags with people in the organization who were already skeptical, Spring says.

The Webmasters of the individual sites balked at the sheer amount of data the research would generate. Their perception was that having to monitor and react each day to information about their site would add more work to their already overloaded schedules.

In addition to being understaffed and overworked, the Web units were also underfunded and as they learned more about the cost structure for Digital Dashboard, they realized they didn’t have money in their budgets for a significant research function. "Anything higher than free was too high," Spring says. "I tried to marshal some resources and create some structures around it to make the costing more efficient but we couldn’t get it down to a level that would work for the Webmasters."

And without a push from each network’s research department, the Webmasters couldn’t be convinced that Digital Dashboard deserved their support, Spring says. "Early on, I didn’t get the buy-in from researchers at those respective networks to push the idea forward. The Webmasters didn’t see the value because they were so caught up in their day-to-day work."

In the end, the mountains of real-time data proved to be too much, Coates says. "I would say the epitaph on this experience was: overwhelming. The Turner people were on all fronts overwhelmed with all the data they didn’t need instead of receiving the nugget that they did need.

"One of our outcomes from this project was, just because you can incorporate all of this stuff and create one large on-line report that encompasses everything, doesn’t mean you should. You start linking together audience profile and satisfaction and tactical performance and media consumption and you end up with this big unwieldy thing. And from Turner’s point of view it became too complex too quickly."

Rather than a real-time monitoring of visitor opinions on their sites, the Webmasters said they would be content with an occasional research check-in, once a year or twice a year at most, to do some audience profiling and obtain feedback on critical issues.

"The Webmasters wanted the audience profile," Coates says. "The research department was certainly interested in some of the other stuff. Their executive management was interested in the consumption aspect. But to put all those things together and throw them into the Digital Dashboard initiative was certainly ambitious. What held us back in terms of realizing the potential of the real-time reporting was the fact that we were trying to report not on a single car but on a whole freight train."

Not soured

The trials of shepherding Digital Dashboard certainly haven’t soured Spring on the merits of on-line research. In his experience, it’s been "phenomenally successful."

Cooperation rates are high - tests for Digital Dashboard earned a 41 percent rate in some cases. And 20 percent of first-time visitors to the sites, who weren’t taken through the survey due to their lack of experience with the site, cooperated and took the survey later after being contacted by e-mail with a second chance to participate.

Participation levels also can depend on the reason people are visiting the site, Coates says, adding that the Turner sites benefit from their entertainment orientation. "We’ve done similar methodologies with Intel’s support site and with a situation like that, where people are having actual problems with their PC or chip set, they don’t want to complete a survey. The participation rates were lucky to climb into the double digits on that site. So a lot of the success in terms of participation has to do with whether people are actively engaged in a specific task that they really need to get through or, as is the case with the Turner sites, if they’re just surfing or planning their television viewing."

As the novelty of Web research wears off, incentives are going to play a bigger role in enticing participation. Spring says that the old standby, a cash sweepstakes, has worked well, but charitable contributions have also been popular. They make the most sense, of course, if they’re related to the subject of the research or are tied into the entity sponsoring the research. For example, the Turner Classic Movie network has conducted studies which give two dollars for every completed survey to a film preservation fund in the respondent’s name.

Spring has tried to gather feedback from potential site visitors as well as regular visitors. "You can use panels for that or e-mailed site feedback that you have saved or other areas of registration where you have asked people if they would participate in research...any resource that you can use to locate people who have the criteria of your site visitors but haven’t visited it yet. Be sure to parallel that with the research with regular visitors so that you’re not working in a vacuum. This way you have a potential to grow your site traffic rather than maintain it."

Needs change

A Web site’s research needs will change depending on who claims ownership of the site within the company or organization, Coates says. For those on the system administration side, research may mean analyzing the server log file and measuring and evaluating the hits and the click stream. "As the Web site becomes more a major part of the corporate entity, then the traditional marketing, advertising, and research forces begin to play a dominant role," he says.

And the amount of research you do may also depend on how much traffic your site gets and how often it changes. "If your Web site is growing by leaps and bounds with people visiting more often and staying longer, you might want to take on more frequent research. In our case, for some of the incrementally-growing sites, the perceived need was much less," Spring says.

Coates says watching media companies come to terms with the Internet has been a very interesting experience. "In theory, they should be the best-equipped to deal with the immediate implications of the Web because it’s just another medium, after all. Early on it was difficult to tell to what extent this new medium is going to upset traditional media and so the false hope that the Web was a simplistic secondary format to the main format was an understandable initial reaction."

Necessary tools

As the Digital Dashboard example shows, at this point the reach of real-time Web-based research may exceed its grasp. "You can report things in real time," Coates says, "but does that mean that the client will have the need or all the necessary tools to take advantage of it?"

The technology exists to capture and manipulate a huge amount of data from on-line research. But to truly take advantage of the data, plain old hardware issues will have to be overcome. In many cases, on the client side, computer systems need to approach state-of-the-art to take full advantage of the data processing and analysis. And on the respondent side, colorful graphics and streaming media will make a snazzy survey experience but not when they’re being forced through 28.8 modems into aging PCs.

"There’s a lot of potential surrounding what the Internet can do for our industry," Coates says. "As you develop your capabilities, you realize that there’s more that needs to be in place before the true promise of the Web is realized within our industry. There are a lot of research companies that have a strong functional capability on the data collection side with emerging on-line reporting capabilities, but in terms of really revolutionizing our industry and bypassing the traditional mechanisms with an end-to-end Web solution, there is more that needs to be in place."

Quick to react

Coates says that the research industry has been quick to react to the promise of the Web, contrary to what its members may feel. "In the research industry we’re often hard on ourselves but we’re very quick to react. While many industries are just coming to terms with the Web, I can’t name a single [research] company that hasn’t gotten involved at some level. Every major organization that you talk to has got something underway, or they’ve got a partnership that gives them the [Web research] capability."

But in the rush to embrace a new medium and meet client wants and needs, the industry can’t lose sight of the need to conduct methodologically-sound research. "Our industry is so client-driven that when the issues touch the Web we’re there and when real-time reporting is a necessity we’re trying to quickly develop the capability too. Along the way we have to ask ourselves whether all of the capabilities that we are developing are resulting in better research. While the rush to the Web has temporarily bypassed some important validation issues for some companies, I think that validation is a strong and recurring theme for this year. It seems like some of us early pioneers in the Web took giant strides forward and now we’re stepping back to make sure we’re heading in the right direction."

Still alive

The Digital Dashboard idea is still alive, but, at least in the case of Turner Entertainment Group, it may be used on a smaller scale. "With adaptation it could be valuable on a smaller scale, as a slimmer, toned down profiling tracker that is less ongoing and more opportunistic," Spring says.

"At some point we may in fact wind up with something that is very close to what we started off with," Coates says. "After people have made their way up the learning curve, after the Webmasters have gotten comfortable with seeing their demographic profile day-to-day and seeing it change and seeing how what they do impacts satisfaction, they might be interested in finding out more."

The real-time benefit didn’t really reach fruition, but it will, Coates says. "Overall, the data collection went smoothly, Digital Dashboard worked as a convenient reporting capability. But as for the notion of creating this real-time dashboard that would allow people to drive as they go and see how the decisions they’ve made have impacted things and then feed that back into the process, more work needs to be done by both clients and research vendors to refine this capability. But I think the industry is aiming towards that and when we get there it’s going to change the nature of research forever."