Money metrics

Editor’s note: John Burshek is chief research officer at NetRaker Corporation, a Sunnyvale, Calif., research firm.

These days, whether you’re a business-to-consumer marketer or a business-to-business firm, you need to have a presence online. Indeed, already over 90 percent of large businesses and 41 percent of small businesses have an online presence. Of course, presence alone does not mean these companies are achieving critical company goals, but more on that later.

Much has been said about Web site usability, navigational issues, empowerment of the customer, two-way communication, e-CRM, site satisfaction, retention factors, etc. However, we were unable to find empirical data demonstrating how measurable metrics truly relate to what most companies with an online presence crave: increased revenue (if an e-commerce site) or increased usage (if an information or e-community site).

We began pondering these issues while working with companies on their online presences. We wanted to provide a systematic method of identifying and verifying important and meaningful key metrics, metrics that could be applied quickly and easily to any company’s Web site. But of all the interactions with a site that could take place, which were really meaningful?

To answer these and other questions, we decided to undertake a self-funded study. Our goal was to find a research method or methods that would provide statistically-sound information against which advanced statistical procedures could be applied.

E-mail studies, whether conducted against a panel or another type of pre-recruited source, were subject to the biases inherent in the list or panel. They were also subject to low response rates. There was also the issue of handling complex questions and skip patterns.

Panel studies have biases of their own. While yielding extremely large sample sizes, the inherent response rate, as in an e-mail study, was typically in the 5 to 12 percent range. This left far too large a non-response bias to be able to apply statistical procedures to the datafile, regardless of how large that datafile might be.

Qualitative research methodologies, whether conducted online or offline, would not yield projectable results given the very nature of the procedures and the opportunity for respondent group-think to impact the findings. Strictly offline quantitative procedures such as telephone sampling did not provide a true read of the participant’s response since an interaction with the medium and the company’s site had to be manufactured as opposed to actually taking place.

Probability theory in real life

We conducted side-by-side studies using the aforementioned techniques and comparing the findings against probability sampling procedures, typically applied through offline telephone random-digit-dial techniques. In all cases, despite the generation of large online sample sizes, significant bias was determined in the findings of a strictly online sample using available techniques. In some cases, the findings were so dramatically different (in excess of 400 percent) as to render use of some of these procedures questionable except for the most exploratory of purposes.

For the most part, many of these processes couldn’t meet the basic elements of probability theory. A tenet of probability theory is that the sample represents as closely as possible the population from which it is drawn. We determined that the inherent biases of drawing a sample from panel or pre-recruited populations resulted in sampling frames that do not allow for equal representation of the online populations under consideration, despite containing large amounts of individual respondents. Hence our datafiles from these methodologies did not qualify as probability samples.

Another key statistic directly related to the quality of the data was the response rate. We discovered a response rate of around 5 percent was very typical. This leaves an enormous non-response bias, prohibiting the use of statistical interpretation of much more than just the sample statistics themselves. There was simply no way to draw any conclusions about the population under study from such a sample. As the response rate increased, there was a direct correlation to the repeatability of study findings, a key determinant of quality of findings.

We settled on developing a multiple methodological attack of the problem, based on the premises of experimental design techniques. In basic terms, we screened and recruited research participants using offline probability telephone sampling. In using this process, we were able to achieve a significant response rate among the qualified participants.

We then conducted the actual company site testing online, with the research participants coming to a specific test site. Response rates were calculated on the fly and steps taken to insure response rates necessary for meaningful data. We achieved a final study response rate of 46.6 percent. Given the preceding sampling frame and response rate, the study can be said to involve a representative sample of the online population at large.

The study

Three mega-sites, each striving to redefine the new category of “power site” and each offering shopping as either all or part of their online presence were selected: Amazon.com, America Online, and Yahoo!. Research participants were sent to each site independently and an in-depth interaction and questioning process was conducted.

A combination of usability techniques (task tests) and market research evaluation questions were asked for each site. In total, over 45 variables were presented for each site. The variables included in this study were those deemed important through initial market research and usability testing efforts, including brand variables such as company reliability and leadership; navigational and general usability variables; past interaction with the site; visual design variables; ease of use and perceived helpfulness; and dependent variables which included likability, meeting expectations, referral likelihood, site satisfaction, re-visitation and purchase likelihood. Testing took between 20 and 35 minutes for each site.

The presentation of the site was done through an online technique our firm offers that allows for split-window presentation of both the site and the questioning process. This allowed site interaction, testing, impressions and evaluations to happen at the same time. The questioning proceeded from an initial evaluation of the visual design of the site through two task-tests and ended with a battery of site and company branding evaluations.

Analysis protocol

The goal was to determine which independent variables had the most meaning and, if acted upon, would impact key dependent variables for a company. The variables themselves were designed to allow the highest dispersion possible by using an 11-point scale, 0 to 10. Through many tests, this scalar pattern has proven very successful in such endeavors for a couple of reasons: 1) it’s the most dispersed scale that respondents can quickly grasp and 2) it has been meaningfully correlated to actual results in the marketplace.

Several statistical techniques were used, including multiple regression and discriminant analyses. We were looking for relationships between and among the variables as well as explaining the variation surrounding key dependent variables.

Charts

Outcomes

The charts above highlight some of the consistent overall differences determined on key variables between the three companies.

The following are the top seven findings from the study:

1) If we had to name a winner in terms of doing the best job of driving visitor expectations, it would be Yahoo!. Yahoo! edged out Amazon decisively on several key variables.

2) AOL, on the other hand, was not in the running on any of the variables by which the sites were tested.

3) All the responses toward the three companies held true regardless of the past interaction or awareness of the site.

4) The usability portion of the study yielded very beneficial differences between the three companies and had a major impact on the range of evaluations between the three companies.

5) None of the scores from any of the three companies exceeded an 8.02 (on a 0-to-10 scale). For both Yahoo! and Amazon, the vast majority of scores were in the 7.0 to 7.8 range, basically, only “average” scores in the interpretation of such scales. By comparison, true industry leaders in brick-and-mortar (Nordstrom, Eddie Bauer) and even professional services (Arthur Andersen) achieve scores ranging from the high 8’s into the 9’s when similarly evaluated.

6) Site satisfaction and its components comprised the greatest drivers of revisit and purchase likelihood. Specifically, site satisfaction was found to be directly linked to the following individual variables: “guiding you through content and function,” “making it easy to search for what you’re looking for,” “portraying a very reliable company,” “providing easy to understand instructions,” and “the visual design makes you want to explore further.” If a company were to look at which variables need to be benchmarked and tracked to insure its online presence was moving in the right direction, these are where it needs to start.

7) Last but not least, one of the most important and significant findings was that there is practically a one-to-one relationship in the interaction between the important independent variables (see previous list) and the dependent variables of site satisfaction, re-visitation and purchase likelihood.

This last finding is most significant. The practical application being that nearly any changes that affect the previous independent variables of a company’s site will have an immediate and direct impact to those variables that drive the issues of most importance to a company: re-visitation and purchase probability.

Real-life application

A common criticism of marketing research is that it doesn’t relate findings to the real world. Business owners, senior management, boards of directors, and stockholders all eventually require that the numbers we derive be related to the numbers they see, typically revenue and profit figures. We knew the same challenge faced us regarding the results of this study.

Since Amazon.com was the mega-site with revenue numbers most easily related to its customer activity, we looked at the impact of “moving the needle” on the site variables our research found to be important and applied it to company data derived from Amazon’s annual report.

Amazon Chart

Not surprisingly, Amazon’s own information allowed us to see the great importance of customer retention. Armed with the preceding company information, several what-if scenarios were worked through and the information regarding the impact of positive changes to the independent variables on Amazon’s site was applied to the dependent variables of re-visitation and re-purchase likelihood. The results were then entered into a proprietary algorithm incorporating Amazon’s own company information to determine what real-world impact should be seen through improving the company’s site.

Amazon is forecast to grow 40 percent this year in revenue just doing business as usual. First-quarter results for this year bear out this estimate. If Amazon were to address and improve upon the important issues relative to its site, it should expect to be able to move the needle of average (mean) re-purchase likelihood among past customers from a 7.3 to a 7.9 or slightly higher. This correlates to slightly more than a 5 percent movement given the use of an 11-point, 0-to-10 scale as the means of measurement.

The impact of this move is a 15 percent increase in top-box likelihood of re-purchase percentage. Now, we all know that what people say and what they actually do are almost always two different things. However we also know from years of study on product and industry categories that correlations between expected and actual results can be quite accurately drawn. In this case, for the sake of being conservative, an actual increase of only 5 percent was applied for the rest of the calculations.

When this number was then applied to the current anticipated re-purchase patterns determined without any change to Amazon’s Web site, the result was a delta between the anticipated revenue without change and the anticipated revenue with changes. Conservative estimations are that Amazon would see an additional $158 million in revenue from this slight increase in meaningful functionality of its Web site. From a practical perspective, given how large Amazon is forecast to become, is this really a significant number over the course of the year?

It would mean an annualized growth rate of 49.5 percent as compared to the anticipated 40 percent. This equates to an increase in annual growth rate of 24 percent, a significant percentage even for a fast-growing company like Amazon.com!

Lessons learned

The lessons learned from this study have application to e-commerce sites of any size.

First, a company can and should know where it stands on Web site metrics that are meaningful to its site customers, visitors, or even employees and partners on an extranet. This doesn’t need to be a mystery anymore. This is knowledge that, upon application, is guaranteed to have a significant impact on a company’s market success.

Second, the world is not static in its expectation of a company’s online presence. Change in expectations caused by companies and elements that seem out of a specific company’s realm of control will have significant impact on how a company is viewed and branded by its marketplace. We’ve seen this in work conducted over the past several years that included Amazon.com. The evaluations and impressions of all three companies under consideration here are changing over time and Amazon and Yahoo!, it appears, have not only worked very hard to keep up with the change in expectations but have at least managed to stay within an acceptable range of the expectations of the online population.

Finally, it’s clear that companies, with the help of marketing research, that perform better on specific variables will reap real-world rewards and their efforts will more than repay the investment needed. It could easily mean the difference between continued existence and real-world success and dying a slow-but-sure death.