Show your work
Editor's note: Dominique Romanowski is vice president of MMR Strategy Group, an Encino, Calif., research firm.
Are you considering making advertising claims that consumers think your brand is superior versus the competition? If so, you are likely aware that the Federal Trade Commission Act requires that advertisers have a reasonable basis for their claims prior to introducing the claims to the marketplace.1 This means that you must have research to substantiate the claim and it must be completed before running the advertising.
Perhaps you have already done some consumer testing that indicates your brand is preferred. You may be wondering if your existing testing could substantiate a claim and withstand a potential challenge from a competitor. Or perhaps you plan to conduct a new study but are not familiar with the requirements of research designed for claim substantiation.
Superiority claims, such as “Brand X is preferred to Brand Y” or “Brand X tastes better than Brand Y,” may be challenged by a competitor. In that case, the advertiser will likely rely on the research to help defend the claim before one of the regulatory bodies responsible for resolving disputes about deceptive advertising, such as a federal court or the National Advertising Division (NAD) of the Better Business Bureau.
These venues hold research for claim substantiation to specific standards that may be different than standards typically followed in research conducted for other purposes, such as during the development of a new product or improved formula. If you are considering making superiority claims about your product, such as a food or beverage brand, here are four things to keep in mind about tests for advertising claim substantiation.
1. Claim substantiation research should survey only those respondents relevant to the claim.
Consider the claim you want to make. Does the claim refer to a specific brand, the category as a whole or to a specific type of consumer? The claim will define the correct population for your survey.
For example, if the claim is about how consumers of Brand A prefer Brand B, then the survey should include only consumers of Brand A. If the claim is about how consumers of auto insurance think one brand is easier to buy than another, then the survey should include consumers of auto insurance involved in the buying process in their household.
Including the correct universe is important as claim substantiation surveys can be criticized for including respondents who don’t represent the relevant population (i.e., over-inclusive) or for only including respondents who represent certain segments of the relevant population (i.e., under-inclusive).2 For example, there have been challenges regarding pasta sauce claims and cereal claims where the NAD considered the survey universe when making its ruling.3 If you are thinking of using an existing taste test to substantiate a claim, be sure the population included in the survey is relevant to the claim.
Additional factors to consider when determining the right population to survey, include:
The relevant product category. Respondents included in the survey must participate in the relevant product category. For example, for a claim about a brand of yogurt, consider if the survey should include consumers of yogurt in general, of Greek-style yogurt or of a specific brand.
The relevant time frame. Survey participants should participate in the category in a relevant time frame but that time frame may depend on the category. Consumption or purchase frequency can vary across categories. Usage or purchase frequency data, if available, can help identify the appropriate time frame. For example, consider two different surveys, one about refrigerated fresh pasta and one about breakfast cereal. Suppose consumers typically eat refrigerated fresh pasta once every four weeks, whereas consumers of breakfast cereal typically eat cereal three to four times a week. The survey about fresh pasta may require respondents to have eaten a certain amount in the past two to three months. In contrast, the survey about breakfast cereal may require respondents to have eaten a certain amount in the past three to four weeks.
The demographics of the relevant population. Claims research for a national brand should be conducted in all census regions, in multiple locations in each region. The demographics of respondents should reflect the demographics of the population relevant to the claim. If available, usage or sales data by region can help identify the appropriate distribution of respondents across regions, age and gender. Depending on the product tested, the demographic profile of the survey respondents could look very different from the overall U.S. population. For example, consumption of some alcohol products may skew male, some candy products may skew younger and consumption of some condiments may have a regional skew.
The relevant product form. Depending on the product tested, claims research may include respondents who consume different forms of the product. For example, carbonated beverages are often sold in both bottles and cans. Ideally, respondents should test the product in the form they typically consume. Sales data can help identify how many interviews should be conducted for each form.
Other important factors. Depending on the product, consider qualifying respondents on additional factors. For example, you may exclude respondents who have food allergies or sensitivities, a cold or allergies or other conditions that might impact taste. If you are testing alcohol, exclude anyone taking medicine or with conditions which limit drinking of alcohol. Anyone with alcohol already in their system before the test should also be excluded.
2. In claim substantiation research, questionnaire design is very important.
There are several factors to consider when designing a claim substantiation questionnaire, such as:
Questions to include. Keep it simple. Include only questions necessary to substantiate the claim. Questions can have halo effects and bias responses to other questions. For example, if you ask respondents to rate a product on multiple attributes such as appearance, flavor, texture and color, their positive reaction to one attribute can result in more positive ratings on other, even unrelated attributes.4 In addition, questions not directly relevant to the claim could generate responses that undermine the strength of the claim in the event of a challenge.
Question order. To minimize order bias, start with questions about overall evaluations first, such as overall preference. Ask questions about specific attributes such as taste or texture afterwards.5 Consider rotating the order of attribute questions to reduce order bias.
Leading questions. Questions should not be worded to suggest that there is a correct or desired outcome. Response options should include both positive and negative wording and the order should be reversed. Half of the respondents should see response options in one order and the other half, the opposite order. In addition, responses need to include an “I don’t know” option, where applicable.
3. In claim substantiation, test protocols are important and must be consistent across all test locations.
Sensory claims refer to claims involving products experienced through the senses, such as food and beverages. When designing test protocols and analyzing results for sensory claims, there are several things to consider:
Product to procure. The products procured for the taste test should come from similar points in the distribution chain and should be as similar in freshness as possible. During procurement, the products should be handled and stored in a manner consistent with typical supply chain protocol for the products.
Product preparation. Products should be prepared and served according to the instructions on their packaging. Preparation instructions may differ between products in the test, adding executional complexity. Depending on the category, products may need to be served consistent with the consumer’s usual consumption method. For example, some spirits are consumed without ice, with ice or with a mixer.
Food and beverages must be served and tasted at the right temperature. This requires clear instructions about refrigeration and/or cooking, how to maintain and verify the temperature when testing and how quickly the product should be served. If the beverage is carbonated, it must be poured in a specific and consistent manner. In the case of alcohol, there are guidelines for how many ounces can be served to each respondent. In addition, cups or plates used should not impact the taste profile of the product and should be the same in all facilities.
Palate cleansing. During the test, respondents should cleanse their palates before tasting the products. Instructions should be clear about when and what respondents should use to cleanse. For example, instructions should specify whether to cleanse with flat, distilled or mineral water, as well as the type of cracker, if any, to use. Respondents in all facilities should be provided the same products to cleanse their palates. The time required between serves should also be specified as this may differ depending on the products being tasted.
Double-blind procedures. Interviewing facilities should execute the research so that the interviewers and other research staff do not know the brands of products being tested or served. This may require separate field instructions for staff with different roles in the research process and extra care regarding the delivery, labeling, preparation, serving and disposal of the products.
Consistent protocol across locations. For a national claim, the test will include multiple locations and it is important to insure testing procedures are the same across all locations. Instructions to interviewers and supervisors must be clear and detailed enough so protocols, including which products are purchased and how they are prepared, can be replicated in each research facility.
4. Consult resources available regarding claims research.
Legal counsel. You likely already work with in-house counsel to review advertising and packaging copy. Your legal team can be a good source of advice about claim substantiation. Sometimes outside counsel is brought in to oversee the claims-research process because of their experience navigating a potential challenge with regulatory bodies like the federal courts or NAD.
ASTM guides. ASTM International6 is a large worldwide organization that develops voluntary standards across many disciplines. It has published several useful guides for claim substantiation, including guides about sensory claim substantiation,7 serving protocol for the sensory evaluation of foods and beverages8 and sensory evaluation of alcoholic beverages.9 These guides provide detailed guidelines regarding how to design and implement claims research and also specify the survey results required to substantiate a claim.
Testifying research experts. There are survey researchers who are experts in claim substantiation. They are familiar with protocols for claims research and have testifying experience before the courts or NAD. They can design and execute your survey or can act as consultants to your usual research supplier. They will ensure the proper standards are followed. In the case of a challenge, they may be able to testify.
Decisions regarding past challenges. Decisions from past challenges can help clarify the expectations of regulatory bodies and how they may rule in cases. Information about challenges handled by the NAD can be found on its Web site.10 When challenges are handled through the courts, information about past cases can be obtained through Lexis Nexis.11
Specific standards
Taste test research to substantiate potential advertising claims should be conducted, as much as possible, according to specific standards that may differ from traditional research. Keep in mind the tips above when designing your claims research or when evaluating whether existing research can substantiate the claim you want to make.
Footnotes
2 Barber, William G. “The universe.” Trademark and Deceptive Advertising Surveys Law, Science, and Design. Ed. Shari Seidman Diamond and Jerre B. Swann. American Bar Association, 2012, pp. 27–28.
3 http://www.asrcreviews.org/nad-finds-mizkan-can-support-consumer-preference-claim-consumers-prefer-the-taste-of-ragu-homestyle-traditional-over-prego-traditional/ and http://www.asrcreviews.org/nad-recommends-mom-brands-discontinue-taste-test-claims-challenged-by-post-advertiser-to-appeal/
4 Lawless, Harry T., and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Second ed., Springer New York Dordrecht Heidelberg London, 2010, p. 366.
5 Lawless, Harry T., and Hildegarde Heymann. Sensory Evaluation of Food Principles and Practices. Second ed., Springer New York Dordrecht Heidelberg London, 2010, p. 363.
7 ASTM E1958-16a, Standard Guide for Sensory Claim Substantiation, ASTM International, West Conshohocken, Pa., 2016, www.astm.org/Standards/E1958.htm.
8 ASTM E1871-17, Standard Guide for Serving Protocol for Sensory Evaluation of Foods and Beverages, ASTM International, West Conshohocken, Pa., 2017, www.astm.org/Standards/E1871.htm.
9 ASTM E1879-17, Standard Guide for Sensory Evaluation of Beverages Containing Alcohol, ASTM International, West Conshohocken, Pa., 2017, www.astm.org/Standards/E1879.htm.