Editor's note: Finn Raben is director general, ESOMAR.
10 years ago, the advent of “always on” big consumer data was causing tremors in the insights industry. There was much hand-wringing, as industry provocateurs stood on soapboxes to proclaim the end of survey research; behavioral data was going to make the long-standing research and analytics industry obsolete.
If anything, this goes to show how difficult it is to predict wholesale changes in a sector. A decade on and the skills and expertise of market researchers are just as much in demand as they have always been. That’s not to say technology and always-on data hasn’t changed the industry, it’s just it’s been far less destructive than the doom-mongers originally forecasted.
But of course, these changes aren’t without their challenges.
As more data becomes readily available from a multitude of new sources, many of these new data streams are supplementing the more traditional research approaches, to provide far richer intelligence. Rather than the “death of the survey,” this new data has led to a change (and often an expansion) in the makeup of the client-side insight team, as well as the skill sets that are required. As customer and consumer behavioral data becomes increasingly prevalent, client-side teams are bringing analysis in-house rather than using outside agencies. When you combine this with the automated and DIY platforms that make it easier to conduct quick, tactical quant and qual research, a significant proportion of what would have been supplied by external generalist agencies a few years ago is now managed in-house.
This move away from outsourcing and back to insourcing started some years ago. As you can see from the global data in the companion chart from Cambiar Consulting, revenue for generalist agencies was already dropping in 2015, while revenue for specialist agencies was showing a significant increase over that 10-year period leading up to 2015. This is part of the change that has been gaining pace in the insights industry over the past 10 years and is impacting the relationships between brand-owners (clients) and their research providers.
Of course, the move in-house is not the only change. The commoditization of data is also affecting the relationship between client and research provider. As technology makes it easier to access data through automation and always-on devices, the collection of data becomes an ever-cheaper function, no matter what the size of the business.
On the positive side, this opens up the potential value of data to all and increases understanding of how to leverage this new information asset. However, the growing abundance of cheap data, along with tighter research budgets, has also led to a race to the bottom and a growing disregard for key quality aspects such as rigor and provenance. This has meant that many research firms who may have been consultative before are now (mistakenly) trapped in the role of a “commodity” supplier.
And thus we get to that now commonly debated issue, which was first raised back when the industry wondered where big data would take us, and which hasn’t changed much in the last decade: Why should a client work with a “consultant” rather than a “supplier”?
Consultant versus supplier
Data is just data; to turn data into insight is a much more complex process. For client-side researchers who are considering external agencies as partners in the search for insight, it may seem like an easy decision to make. You pick a consultant to work with when you need expertise, sector or methodological knowledge that you don’t have in house. You pick a supplier when you just need the data or you just need a tool.
However, I would argue that this approach is overly simplistic and detrimental to the company’s search for insight; it is also very dismissive of the added value that an external partner can bring. When an in-house legal team approaches an external solicitor or barrister do they not invest in explaining what they wish of them? When a salesperson makes contact with a new retailer do they not invest in explaining how they will both mutually benefit from the relationship?
Whether you’re dealing with a sample provider, a fieldwork company, a data analytics bureau or a full-service research house, the right relationship will always be consultative and mutually advancing. From the client perspective, open communication lines and a willingness to listen, sharing information on business priorities, strategies, cost pressures and market challenges are behaviors that build good relationships. While on the agency side, sharing insight about methodology, processes, technology, study design, samples, question-writing and more helps to build a relationship and ensure that it is not simply transactional; in such a relationship, both parties benefit through deeper understanding, getting better, more valuable data and richer insights.
For client-side researchers it’s a process that involves trust and time. Opening up to your agency and providing them an understanding of the bigger picture and the business questions you are asking to solve maximizes the chance of you getting what you need. But it’s easy to say that, and of course part of whether you’re working with a consultant or supplier depends on the agency you’re working with. Many research agencies talk about being strategic advisors and then behave like suppliers. They jump when clients say jump. Often this leads to poorly designed research outputs because the project or questionnaire itself has not been designed with the end game in mind. Scope-creep often leads to unrelated questions being put into surveys, moderators are told “just find out what people think about this too,” costs start to escalate and before you know it the content has little cohesion, making it even harder to find (and then tell!) a story.
Agencies often agree to changes from clients to keep their business and maintain good relationships – and in turn corporate researchers agree to changes and last-minute requests in order to please their stakeholders. Without a partner to review, discuss and challenge hypotheses and proposals both sides stumble. But the most important element in fostering a consultative and valuable partnership is through communication and openness, not just taking advice when given but asking for it and sometimes even challenging it. There is just as much value in the corporate researcher using these consultant behaviors to manage relationships with their stakeholders. You can develop the role of the researcher internally by listening and engaging with the broader strategic picture, using open and clear channels of communication and, of course, using a little pushback where appropriate. Your results will be better, engagement will improve and the true value of your expertise can shine.
When corporate researchers truly partner with their agencies or suppliers, magic happens. One example that comes to mind is that of Zappi and Coca-Cola Japan, a case study of which was presented at ESOMAR Congress last year. If you’re not aware, ZappiStore provides a series of automated data collection and analysis tools – and describes itself as a software company and certainly not a research agency. But rather than treating the firm as a supplier, Coca-Cola Japan brought it on as a true partner, embedded it within the Coca-Cola internal insights team and treated it as part of the company. This allowed Zappi to develop a tool for a new market, something that would have taken the start-up company years to do on its own. For Coca-Cola, this collaborative partnership led to innovation that revolutionized the way Coca-Cola Japan pre-tested its ads.
Although this was a unique partnership, it was held together by transferable elements that we can all learn from. Hisae Endo, group manager of knowledge, strategy and insights at Coke, said, “Transparency was a key element in the success of the project … Having Zappi work on the same level as the rest of the Coke team during the project was incredibly important for both teams. Honesty between both partners is essential in managing expectations on both sides and ensuring the best possible outcome.”
So, when corporate researchers truly partner with their agencies the relationship can really work. But the onus cannot be one-sided. An agency needs to be bold and confident in pushing back and corporate researchers need to understand that, in most cases, you have approached your agency or data provider for expertise, and when they push back it’s the expertise talking. As Anneke Quinn-de Jong, senior market intelligence manager brand creation at Philips, said, “We are continuously looking for quicker and more cost-efficient consumer insight solutions. Sometimes, we want a supplier to deliver just that. However, for more strategic projects, the third and very crucial factor is the added value we get from a true partner who understands our business challenges and thinks with us along the way.”
Magic will happen
Technology has driven many changes within the insights industry and for the corporate researcher the landscape is radically different. The need for expertise, cost and speed are now key drivers in the procurement process. But I believe that if corporate researchers can add “consultative” when considering both their internal behaviors and their external research partners, then magic will happen more often, both sides will get more value from their research and, ultimately, we will create a stronger industry for the benefit of us all.