Editor’s note: Charles Young is president of Ameritest/CY Research, Albuquerque, N.M. Larry Cohen is president of Seminarts, Evanston, Ill.

Many advertising creative directors who distrust the value of copytesting research view winning creative award shows as an important validation of the quality of their work. The creator of an ad is more likely to accept the verdict of his/her peers over the performance measures devised by researchers. The feeling is that research all too often either “measures the wrong thing” or rewards formulaic advertising at the expense of more original executional ideas. Indeed, the director of one of the major award shows, whom we initially approached with an offer to conduct research on their award-winning ads, rejected the offer outright on the grounds that research was “antithetical” to what they were doing.

On the client side, many managers are suspicious of creative awards, viewing them as self-congratulation by the creative community, or as a public relations vehicle for new business pitches by agencies rather than an indicator of good performance on their accounts. In fact, when we approached several advertisers for permission to reproduce their award-winning ads in this article, we were more than once informed that the agencies that had created the advertisements had been fired because their advertising “wasn’t working.”

Undoubtedly the judgement of successful advertising practitioners, as summed up by the major award shows, must count for something. Indeed, it could be argued that creative award shows are themselves an early form of copytesting, developed long before the advent of modern copytesting methods as a way of informing advertisers what defines quality in creative work.

You could even argue that copytesting is a reasonable substitute for the combined opinions of a group of masters of the advertising craft. In making the decision about whether an ad is good enough to run, a hypothetical client could call up a friendly list of creative directors on the order of, say, Jay Chiat, Hal Riney and Steve Hayden and ask their opinions about the work. Or, that hypothetical client could, somewhat less expensively and more practically, simply commission a research study to find out the same thing. This view would lead one to expect that, on average, if the research is well-designed it should produce results that are the same as the judgments of creative award shows.

The contrarian position is that professionals, because they are professionals, look at advertising in a fundamentally different way than ordinary consumers do. Since the true test of advertising quality is whether or not a consumer audience will notice the ad and buy something because of having seen it, advertising research among consumers may frequently produce results that are not only different but are superior in some ways to the judgments of creative award shows.

We enter the debate by contributing a piece of research on research where we asked the question: Does the fact that a print ad won a major award show tell us anything about how that ad is likely to perform in research tests among consumers? And, as a corollary: What does winning an award show not tell us that may be critical to understanding the true potential of the ad to generate sales in the marketplace?

The sample of award-winning print advertisements we chose to include in this study were national winners of major award shows, with the majority of ads drawn from the pool of national winners of either a gold or silver medal from the One Show. Several additional ads were drawn from the winners of the Kelly Show, the Art Directors Show and the Athena, for a total of 37 award-winning ads.

All the ads were tested in Ameritest and compared to our norm base for categories corresponding to the award winners. In general, the average ad in our database is not an award winner, but rather is representative of “regular” advertising.

The three key copytest scores produced by Ameritest are attention, brand linkage and motivation. Attention is a measure of the stopping power of the advertisement in a clutter book with other ads competing for the reader’s attention. Brand linkage is measured by coding the verbatim responses to the attention question for the percentage of respondents who recall the ad with interest and use the brand name as the top-of-mind handle for describing the ad. Motivation is collected after a second exposure of the test ad by itself, without the clutter environment, and is measured by a simple five-point probability scale for rating consumer intentions. This scale varies depending on the category being advertised (from “definitely will buy/consider/visit” to “definitely will not buy/consider/visit”). Finally, other diagnostic measures were also included in the test.

More stopping power

How do award-winning ads perform on the copytest? The performance of the ads on the three key evaluative measures are shown in Figure 1. On average, we see that award-winning ads have more than twice the stopping power of regular or non-award-winning ads. However, award winners are significantly less well-branded compared to regular ads. And, award-winning ads are no more motivating than non-award-winning ads.

These results confirm that award shows are telling us something important about the potential performance of an advertisement: the judgments of creative award shows appear to be highly correlated with the ability of an ad to break through clutter and get noticed. Intuitively, this makes sense. One of the most important reasons that advertisers hire advertising agencies is to get someone who has the talent to create an execution that is particularly clever or entertaining or unique so as to capture the attention of a large audience.

The finding that award-winning ads are no more motivating than non-award-winning ads also has face validity. In general, identifying a selling proposition that has real strength in motivating consumers usually requires a deep understanding of the product category based on a substantial amount of strategic research into consumer’s attitudes, perceptions and current behavior. Typically, the information needed to make a critical judgment as to whether or not a particular message will be persuasive is proprietary and is not submitted along with the entries to creative award shows. As a result, the judges of creative award shows do not have access to relevant information to effectively discriminate among the various entries on the basis of motivational impact. Award shows are about executions, not strategies, so it should not be surprising that the key to winning award shows is the attention-getting power, not the motivational power, of the ads that are entered.

The weakness in brand linkage in award-winning ads is more of a problem. Strategic insight is not required to make a judgment about how strongly a particular execution is branded. One possible reason for this finding is that award shows tend to reward ads that break the rules, viewing this rule-breaking as an indicator of creativity. Certainly, clearly communicating the identity of the brand in an ad is a classic rule of advertising. Defining creativity in iconoclastic terms might lead some award show judges to implicitly prefer executions that challenge assumptions about traditional branding rules, as illustrated by this comment from the legendary advertising exec Bill Bernbach: “There will be a time when no headline is proper; there will be a time when a headline is proper. There will be a time when using a logo is the worst thing in the world you can do.”

It is likely that the finding of weak brand linkage reflects an underlying philosophy of how to create advertising that is effective in today’s cluttered advertising environment. Many creatives today appear to believe that to get past the psychological defenses that consumers have raised against the continuous bombardment of selling messages an ad should not look like an ad. With this line of creative reasoning an ad needs to lure a consumer in with intrigue, sleight of hand or entertaining dissimulation in order to get past their guard to deliver the sales message.

The logic of this creative philosophy is that for many types of executions there is an implicit trade-off between consumer attention and brand linkage. And for award-winning ads, this trade-off appears to be a good deal. On average, compared to regular ads, the substantial increase in stopping power more than makes up for the slight decrease in brand linkage.

Creative ads are more entertaining, but more confusing

Diagnostic ratings are shown in Figure 2. Award-winning ads are rated significantly higher than regular ads on being a lot of fun and being different from other ads. Consistent with the attention-getting philosophy of award shows, the focus of award-winning executions is on entertainment value and executional uniqueness.

Award-winning executions are also rated significantly higher in terms of confusion and irritation. This suggests that award-winning ads are likely to be the products of creative risk-taking, a willingness to break conventional rules of advertising communication and push the edge of reader sensibilities.

From a client perspective, however, award-winning ads might appear to be less effective. The messages conveyed by award-winning ads are rated less believable and the consumer audience has a hard time relating to the characters or people shown in the ad. Consistent with the weakness in brand linkage, the integration of the brand into the execution is not as clear with award-winning ads, as shown by the higher ratings on the statement “I couldn’t tell what the brand had to do with what was said and shown.”

In general, the diagnostic ratings confirm the results of the evaluative performance measures. Award shows such as the One Show appear to tell us a great deal about the kind of advertising executions that are likely to attract the attention of an audience. On the other hand, they appear to tell us nothing about the strategic effectiveness of the award-winning advertisements, either in terms of branding or their ability to clearly communicate motivating messages.

Small changes, big differences

Malcolm Gladwell’s book The Tipping Point presents an illuminating story about the importance of non-linear processes in marketing. A non-linear process is one where the size of an outcome can be drastically affected by relatively small changes in the inputs. Clearly advertising is such a process. Marketers recognize that getting creative executions right can sometimes magnify the effectiveness of a given level of media spending weight many fold.

As every good creative director knows, God is in the details. Our purpose in this part of the research is to demonstrate that advertising executions are indeed non-linear variables in the marketing process where the judgment of experienced creatives (particularly those immersed in the client’s business) regarding the details of executions is of paramount importance. By taking several award-winning ads from our test sample and making a relatively small change to the execution with Photoshop and re-testing the ads, we wanted to see how it would affect the overall performance of the ad in a significant way.

For the client perspective, we also wanted to demonstrate that winning a major creative award is not necessarily proof that the ad is optimized from an overall advertising effectiveness standpoint. In particular, we wished to show that there is not necessarily a trade-off between attention-getting power and other dimensions of advertising performance.

First case: improving an ad’s performance

The first case, shown in Figure 3, is an ad for Time magazine, a Consumer Arts winner. The ad shows a little boy wearing a camouflage uniform and holding a rifle, with the red rectangle of the Time logo outlining his face. There is no headline. The body copy says, “Make sense of anything. Almost.” This is a childhood picture of a future high school killer.

This ad scores above average in terms of stopping power, outperforming the average by a factor of two. But it is only average in terms of brand linkage, average in terms of motivation. The reason for this performance becomes apparent when we look at the diagnostics: Nearly half of respondents found this ad to be confusing. With probing, it became clear why. Seeing the Time box containing only the boy’s face but not the uniform or rifle, many respondents were left with a confusing interpretation thatTime leaves important information out of its stories!

Reflecting on Bernbach’s dictum we thought that this might be one of those times where the ad does need a headline. So in the revised version of the ad we added the headline, “Time tells the whole story.” As a hypothesis, we felt that this change might shift how the image would be perceived. By reassuring readers that the magazine does not leave important information out, we felt that respondents might look at the image in a more three-dimensional way - seeing the face on the cover with the implication that the rest of the information is “behind” the cover or inside the magazine.

The performance of the revised ad was even more positive than the original. The level of confusion was cut in half. Consequently, the stopping power, strong to begin with, doubled. Brand linkage increased. And motivation doubled to an above-average level. This was clearly a case where there was no trade-off operating between the attention-getting power of the advertisement and other key dimensions of performance.

Second case: hurting an ad’s performance

The second case, shown in Figure 4, is an ad for American Standard, another award winner. The ad depicts a bull with its horns stuck through the drain pipe holes in a double sink. The headline is “Introducing One Tough Sink.” The ad is an attention-getting execution, with a stopping power twice the average. It is also above average in terms of motivation.

In this case the striking visual was by far the dominant creative element in the ad and the headline appeared to play a supporting role, merely as a caption for the visual. In our revision, the headline was changed from the rather clever original to the more literal “Introducing a Sink as Tough as a Bull.” In other words, our revision was designed to represent the kind of change that many creatives fear - those a literal-minded client might ask for.

While it is unchanged in terms of stopping power and brand linkage, the revised ad is significantly weaker than the original in terms of motivation. The reason for this can again be found in the diagnostic information that was collected.

One of the key diagnostic measures of our firm’s methodology is a measure of stickiness, which is the construct of how long an ad will hold a reader’s interest once he or she has stopped to look at it. In this test stickiness is measured attitudinally after controlled time exposures of the ad when the respondent is asked to report interest in “continuing to look at or read the ad.” You’ll note in Exhibit 2 that the stickiness of the American Standard ad is significantly reduced as a result of the revision made to the original.

Evidently, the change we made to the headline makes the point too obvious! The original, award-winning ad is thought-provoking and engages the consumer’s mind in thinking about the durable properties of the sink. Our flat-footed revision, by making the point obvious and not worth spending time on, turns the claim of durability into a more expected, and less persuasive, piece of advertising puffery. Agencies may have a point, therefore, when they argue forcefully with clients for the creative elements in ads. Small changes in executions can make a big difference in advertising performance.

The lesson?

What is the lesson of this research? It teaches us two things. First, that agencies are right when they celebrate the importance of creative judgement about advertising. But second, that clients are also right when they express concerns about the requirements that advertising executions must fulfill in order to be effective at generating sales.

Winning creative awards is hard, but making effective advertising is harder still. This research shows us that creative awards do tell us something important about the power of advertising to break through a cluttered media environment. But awards do not teach us everything we need to know about how an ad will work. Effective advertising is more likely to come from the collaboration of creative and client, working with the discipline of research to produce breakthrough advertising that also builds brands and motivates sales. 

References

Feinberg, S. “Turnabout: Agency Creatives Evaluate Research,” Marketing Review, January 1994.

Gladwell, Malcolm, The Tipping Point: How Little Things Can Make A Big Difference, New York: Little, Brown and Co., 2000.

Young, Charles, “High Touch vs. High Tech,” Quirk’s Marketing Research Review, March 2002. (www.quirks.com )