Defensible by design

Editor's note: Gail Conn has 25 years of experience in market research, the last 20 as CEO of Phi Power Communications, an online quantitative research firm. She has an undergraduate degree from Hamilton College and did graduate work at University College in London. She also has a strong tech background, having worked at IBM, Broadcast.com and Yahoo.

Would your marketing research or customer insights project survive cross-examination in a court of law? That’s not just a rhetorical question. It’s the reality in the world of conducting market research for the legal industry.

In the majority of corporate market research projects, weak wording, wrong questions (yes, they exist), flawed recruitment, biased question flow or “bad” verbatims (you all know what I’m talking about) usually go unnoticed. In litigation research, any one of those mistakes, done only once in a study, can destroy millions of dollars in legal strategy under hostile cross-examination. And imagine if you will that those single mistakes can lose cases worth hundreds of millions of dollars to your client. Imagine this was your research on trial. Nervous now? You better be.

Welcome to the high-stakes world of litigation, or legal, research. A world where there is NO “safe place” for ANY research weakness to hide!

Although similar to corporate quantitative research done for companies and organizations, litigation research differs in that every aspect is highly scrutinized. It must stand up to a direct attack by a strong opposition group. Line of questioning, sampling control and stimuli manipulation must all be documented and defensible. This is a requirement so that the court can easily understand that real people have been hurt or confused by the transgression (in the case of the plaintiff) or not (in the case of the defendant).

Want a real-life example of how this actually plays out? Read on!

The setup

A photographer did a photo shoot for a major consumer packaged goods company. She struck a licensing deal with the CPG firm for usage of a picture on the final product packaging for a women’s skincare product. Obviously, the CPG firm, being huge and all, had conducted a bucketload of conjoint analysis testing across all the components that went into the packaging design, color, wording, etc., before launching the final selection into the marketplace. This is basic, standard practice at all large CPG firms nowadays and this case was no different. The point is, the CPG firm knew that the picture it selected was what mostly drove purchase interest prior to launch.

Then reality struck. The product launched and was a hit and remained one of the company’s most successful products in its women’s skincare product line for years.

So successful in fact, that when the license for usage of the picture (clearly stated in the original contract) came up for renewal, the photographer was happily thinking the picture would be renewed for an extended period of time. 

It was not. The CPG firm reneged on renewing the license for the picture. But instead of introducing a new one, as is typically done in these situations, the company kept the package, with the now unlicensed picture, on the shelves and continued to reap the financial rewards, with its successful packaging, from a now large and loyal buying audience.

You see the problem, right? So did the photographer. She wanted what she was owed for the continued use of the intellectual property that she developed for the CPG firm according to the agreements. 

However, being a very large and very successful CPG firm, it dug its heels in and said, in effect, “Not going to pay you anymore. So sue us.” 

What happened? The photographer filed and sued. And filed and sued. And filed and sued again. Just a fact of our legal system – nothing is easy! The case had gone on for many years, with no progress being made, by the time we were brought in to help the photographer’s company against the giant CPG firm. The photography company had exhausted all regular and normal attempts to force the CPG firm to reconcile and continued to be stonewalled by the much larger, well-resourced company.

Bringing in the experts

As often happens in these types of cases, the photography company decided to take one last swing. And this time, it decided to forgo some of the money to do it right and brought in a large law firm on commission. That firm in turn brought in Larry Chiagouris, a legal expert witness with over 50 successful cases under his belt. He was also a professor of marketing for PACE University at this time.

Chiagouris brought in our firm, PhiPower, as we have done legal research for over 25 years and across hundreds of cases. Together with John Burshek, PhiPower’s partner and chief research officer, Chiagouris began strategizing to create a process that: had no flaws in any of its research steps; would prove a financial value for including the picture on the package; and, finally, could stand up to all challenges thrown at it by the opposing team.

The CPG firm didn’t back down and in fact dug in even harder, hiring a slew of its own research experts. Obviously, it needed to “prove” that inclusion of the picture on the package added little, if any, financial value to the revenues the product achieved in the marketplace. It conducted a massive conjoint study among product buyers, testing each of the current packaging components, picture included, against propensity to purchase the product.

No problem, right? Wrong. You should have spotted several major biases with the above approach immediately. But the reason the firm went with the approach it did was because it gave it the answer it wanted, which was: low levels of importance and valuation of the impact of the picture on the package.

In short, its “research” told a story that the picture on the package was merely incidental. Nothing to see here, nothing going on, keep moving.

Conduct a scan

The first thing Chiagouris and Burshek did was to conduct a scan of all published, post-launch product research that attempted to determine the financial value for each of a package’s components. You know what we learned? There’s practically zero work published on determining the value of a package’s components after the product has been on shelves for several years. 

So we came up with our own self-explicated decision choice research model. This is not a new methodology. It just had never been applied to this scenario. For those not familiar with the technique, it’s a preference measurement method where consumers directly rate the desirability of individual package components. And then they attribute importance ratings to each desired component. The system removes the restrictions of conjoint by not forcing a consumer to choose between various package component bundles.

Now, it’s one thing to come up with a promising concept for approaching a difficult and complex legal challenge. It’s another to apply it in such a way that, when all is laid bare to judge and jury, a) a non-geek, normal person can understand what is happening while b) it is conducted in such a rigorous, statistically meaningful way that the results withstand the onslaught of opposing experts whose whole purpose in life is to destroy your work, in any way they can.

We achieved both of these nearly mutually exclusive objectives because of strong work done by our creative department. Led by Peter Frederick, who heads up the department, the survey itself became a weapon for clarity and decisiveness.

Frederick worked to blend exposure to each of the visual components of the package, from the brand and the picture to the small print (e.g., number of ounces, etc.), individually and as the total package.

Why was this so important? As we’ve said, everything is vigorously attacked by opposing experts in this field of research. 

We had to establish that, yes, the respondent did know exactly how the package would look without the picture, without the brand, etc. That’s how the components of the package were creatively evaluated. The respondents could interactively see and feel the visual impact of each component, immediately, because they could see the package with and without each component. 

These requirements allowed us to meet the opponent’s challenge head-on with a walkthrough of the visual process, as seen by each respondent. 

The survey tool itself became a powerful factor in the results presentation. Whereas the opposing experts used a methodology (conjoint) that was very difficult to understand for a purpose for which it was not created, we quickly and visually demonstrated how each respondent saw and perceived each of the questions surrounding desirability, attribute importance and utility calculation for each of the package’s components, individually and collectively.

Both Chiagouris and Burshek were deposed and grilled by opposing counsel and their experts; Burshek for one full day – over eight hours of challenges to methodology, question framing, sample selection, analytical technique, etc. – and Chiagouris for multiple days. 

Brought everyone to the table

What we found in this study was very clear and ultimately brought everyone to the negotiating table able to see the situation the same way. Yes, a picture on a package can have a very significant impact on consumer’s continued selection and loyalty to a product. This is due to the implications regarding the quality of the product inside inferred from the picture. In this case, the picture was only slightly less important than the brand in its impact. 

This finding resonated all the way through to a very strong dollar valuation for the picture being on the package. A calculation against revenues gained by the product once past the point of reneging on the licensing agreement determined a financial number that was quite sizable.

What happened because of this? The CPG firm settled, not wanting to risk taking the case to open court. In short, as our attorney said when he called us, “We WON!”

Hand it over to AI

These days, the temptation in the research industry is to hand over much, if not all, of a project’s thought and execution processes to AI. And yes, this has already been tried in litigation research and has failed spectacularly under the harsh levels of criticism and scrutiny like those outlined above. AI simply cannot deliver to the rigor, clarity, expertise or quality in any of its “thought” processes, question formations, sample acquisitions, results analyses or presentations.

What about your own research processes? Would every decision you made on a project for how, who, what and why stand up to being picked apart for an entire day – or more?

Here is another example: Tiffany vs. Costco. Widely publicized when it occurred, our research also withstood the harsh criticisms of the opponents and played a very important role in the outcomes. 

Around 2013, Tiffany became aware that Costco was stating on the fact tag for an assortment of diamond engagement rings being sold through its stores that the rings were “Tiffany style.” Tiffany initially pressured Costco to stop what it felt was a misleading practice. But Costco refused to do so and in 2017 Tiffany took Costco down the legal route. 

The expert witness retained by Tiffany was Jacob Jacoby, for whom PhiPower did virtually all the research. Jacoby designed a survey instrument and approach to nail the case. 

We found respondents who were interested in, or had previously been interested in, purchasing a diamond engagement ring and had shopped at Costco. Using photos, each respondent was virtually guided through the experience of entering the store and going up to the jewelry counter. 

Once there, they were shown a diamond engagement ring and its accompanying fact tag. Here is where the experimental research design kicked in. Half the respondents saw the fact tag with the Tiffany style and half saw it without the Tiffany style. 

All respondents were asked one simple question: Who do you think made this ring? As you might expect, there was a very strong percentage of the Tiffany-style cohort who answered Tiffany. This simple little question was key in determining a verdict against Costco to the tune of $21 million dollars, including damages. Yes, one experimental research design using one specific question and the real-world result: $21 million. (The initial verdict was overturned in 2020 by an appeals court but, even with that, Costco ultimately settled out of court for an undisclosed amount.)

If you ever wonder about the potential power of research in terms of real dollars, the field of litigation research draws a direct line from the substantiation of one side in a legal dispute to a real-world dollar outcome. In the case of this story, saying a brand name has been misappropriated is one thing but showing quantitatively exactly how much, as this case proves, is worth multiple millions.

Not the same

We share these cases with you because litigation research is a fascinating area that sounds like it should be the same as corporate research and customer insights. We hope you’ve seen that it is not. 

Our other message is that it's always worth increasing the quality and defensibility of your own studies. Take time to think more deeply about the what-ifs and remember that if a project’s methods or outcomes are challenged in the corporate setting, the questioning you face will be nothing like what litigation research must withstand. And if you've built your study to survive that kind of scrutiny, you'll be ready for whatever the boardroom throws at you.