Editor's note: Edward Appleton is a client-side European consumer insights manager. He can be reached at firstname.lastname@example.org. This article appeared in the November 11, 2013, edition of Quirk's e-newsletter.
Big data is perhaps the buzzword of the past year-and-a-half. And while the concept creates excitement, it's cloaked in uncertainty. According to a recent Ventana Research study (www.ventanaresearch.com/bgd), organizational immaturity with respect to people, process and analytical access technology are all hindering corporate big data initiatives from deriving maximum value.
In market research, there is a palpable sense of unease. Will analytics tools overtake the research industry in terms of impact and internal importance, including budget allocation? Will the qualitative "why" question become irrelevant in the wake of solid behavioral data? Can we ignore the discussion around big data as hype? Or should we try to adapt to big data, even if we don't fully understand it?
These are questions that I'm sure plague many of us. To help understand them, I spoke with Tony Cosentino, vice president of Ventana Research. Author of the book Into the River, which examines, among other topics, big data’s impact on market research, Cosentino is a longtime researcher and big-data observer. (Click here to read our 2012 conversation on the same topic.)
Tony, there's lots of talk about big data being massively overhyped, where overpromising and disappointment go hand-in-hand. What is your feeling?
Cosentino: The problem is that big data is being offered as a cure-all to every ill. You need to start by looking at the business problem you are trying to solve rather than getting lost in data. These days I prefer to talk more about new sources of information and technological advances than big data. New information sources that are becoming useful include various machine and sensor data, location data and different forms of social and textual data. New technologies are helping us to instrument, store, access and analyze these new data sources and turn them into something useful.
I am sure many market researchers love to read about big data being overhyped because it helps us feel less threatened. From an insider perspective, what's your take?
Market researchers have a lot to bring to the table because we make sense out of data and that is a big need in today's business. The threat comes from thinking that others know more than we do but the fact is that other departments are often just as confused and feeling just as embattled.
Market researchers cannot be afraid to become true analysts. By that I mean we need to understand all available sources of information and not just limit ourselves to attitudinally-oriented data. While attitudinal data has its place, behavioral and demographic data is becoming relatively more important. There are a lot of tools coming to market that are putting power into the hands of the analyst but the tools need to align more with the personas in the organization.
I recently wrote an analyst perspective on the analytic personas driving today's organizations. For market researchers, I would classify us somewhere between the knowledge worker persona and analyst persona. As such, we would do well to learn visual discovery tools (e.g., Tableau), commodity modeling tools (e.g., BIRT Analytics, TIBCO Spotfire, KXEN, etc.) and go on to learn something about structured query language and the R language. The first two provide opportunities for less-technically-oriented people and the latter two provide languages that will endure into the future for data integration and advanced analytics.
Can you give us some examples of where big data has truly made an impact and how?
It's really all over the map, depending on what you define as big data. Fraud, security, network and system administration, predictive maintenance, patient monitoring, any sort of grid-monitoring system - the list goes on and on.
From a market research perspective, the biggest ones are in areas such as a 360-degree view of the customer and bring different sources of information into the analysis of the customer. This is happening on a number of fronts.
In telecommunications for instance, call-detail records, which represent enormous amounts of data, are being sorted and combined with customer records to determine things such as individual quality of service. This can be married with many other sources of data to help determine customer value and propensity to churn. Machine-learning algorithms can then be used to determine the best action for a company to take to prevent churn at a cost commensurate with the value of the customer.
Which industries do you see big data impacting heavily in future, which less so?
New information and new technology is impacting every industry and every function but in different ways and at different speeds. For instance, health care and banking are driven much more by risk and regulatory compliance, whereas retail is driven more by performance and manufacturing more by cost reduction. All of these make sense given the nature of these businesses and the macroforces in today's economy.
One of my favorite categories is retail because it is facing such discontinuity. E-commerce, namely Amazon, has forced traditional retailers to completely rethink their strategy. JCPenney and Macy's provide a sharp contrast in how two retailers approached this challenge.
A few years ago, the two companies eyed a similar competitive space but since that time, Macy's has implemented systems based on big data analytics (i.e., in-memory technology and new information sources that allow for more accurate real-time price calculations) and is now sourcing locally for online transactions and can optimize pricing of its 70+ million SKUs both online and in-store in just one hour. The Macy's approach has, in Sun Tzu-like fashion, made the "showroom floor" disadvantage into an online and offline customer experience advantage.
JCPenney, on the other hand, used gut-feel management decisions based on classic merchandising strategies and ended up alienating its customers, generating lawsuits and issuing a well-publicized apology to its customers.
Do you know of examples where big data and market research are working well together?
I think we are seeing it more and more. I work with companies who are using feedback from stakeholders to drive better analytics and decisions but different from the traditional way. The feedback and collection of the data is getting baked directly into the process through collaboration software and other embedded data-gathering techniques.
For instance, we're beginning to see quantitative and qualitative feedback coming in from the sales force to drive better forecasting and product development. I've always felt that the sales force was an underappreciated source of information and we are finally leveraging this in a crowdsourced manner and incorporating it directly into our processes.
Social media is still a bit of a fool's errand in my mind and, as a point solution, the value is questionable at best. Where it will get interesting is when our natural language processing (NLP) systems get better or when Facebook introduces a categorical approach to likes and dislikes rather than the dichotomous approach it uses now. There will also need to be a better rating system for influencers and deciphering the integrity of the individual postings. Right now, various players are doing interesting work around NLP, such as driving text analysis into quantitative models. Such analytic technology can be applied to a variety of data from collaborative applications - to call-center recordings, chat rooms and the like. Speaking of such, Clarabridge just received another $80 million in funding, which speaks to advancements and market potential around these types of analytics.
Where does big data sit in the organizations you advise? What do the organizational structures look like? How do they interact with MR departments?
Good questions and I'm not sure there is a simple answer to any of them. I'd start by saying that much depends on the organization and the culture of the organization. For instance, a government organization will likely act much differently than a business that feels an existential threat.
What we are seeing in our research, however, is an empowerment of business users and business analysts almost across the board. This is being driven by a number of factors, such as industry competition, but also the ability for business users to rent from the cloud and not incur significant capital expenses. Traditionally, IT has made choices for new tools and provisioned a company standard but that is not always true today. The office of finance and corporate planning are very powerful parts of the business and we are seeing them start to move into less analytically-savvy parts of the organization, such as human resources, which is going through a renaissance of its own.
Since the 360-degree view of the customer is at the heart of many big data initiatives, the marketing organization is in a good position but only if the team is analytically savvy. Otherwise, operations, finance or IT will drive the new analytic paradigm for the organization.
Note that there is an interesting dynamic occurring here. The office of finance and the IT department are natural allies since they are often numbers- and tools-oriented. The marketing department has traditionally had a different orientation but the strength of marketing is in driving topline revenue by truly understanding and influencing the customer.
This is where market research comes into play. But to be honest, I do not see market research as a separate part of the marketing function going forward. Rather, the market research function will be embedded into the marketing insights and analytics function.
One often reads that big data can answer the what but not the why. What's your take? Is the why still important? Can qualitative researchers actually benefit from the wealth of why questions big data presents?
I've heard that but I'm not sure I understand what it means. If I were to guess, we're talking about needing to understand motivation and emotions at an individual level and then at a societal level. These are still important today, especially in marketing, but looking forward they may lose some significance.
For example, if I have enough data to continuously do stimulus and response testing and map that back to the profile of an individual, then I'm going to know what product offer or marketing message to serve up regardless of motivation. Big data and machine learning make this sort of thing more and more possible, though we are still a ways off, especially in offline environments.
The challenge with motivation and emotion is that they change - and change is a difficult thing to model. It's probably more important when we look at broader societal dynamics or when we don't have historical data (e.g., in an innovation-driven market). In both these instances, I would think that market research is imperative.
For the qualitative market researcher, the idea of a "wealth of why questions" is an interesting one and probably has merit since big data raises more questions than it answers.
I'd say that at a minimum, a qualitative researcher is in a great position to be not only a moderator but an action workshop facilitator, an innovation workshop facilitator, an ethnographer and an educator. Interestingly, this educator skill set starts to become a marketing skill set since education is becoming a much more prevalent form of marketing in a digitally-driven culture. All of these areas, especially in virtualized forums, should see increased organizational need going forward.
Having discussed these issues with Tony, a few points became clear:
- Big data isn't a particularly helpful term. We would do better to refer to specific new information sources and technologies.
- Everyone is struggling with the advances in data management, not just research.
- It's a quickly-developing space, with new names and providers that are currently probably only known to a smaller IT and analytics community.
- Equally, some or maybe just a handful of these companies may be game-changers - tomorrow's Google.
- Social media doesn't speak to the business analyst - at least not this one.
- Qualitative will thrive, according to Tony, as a discipline concerned with context, change, moderation and ethnography.
Do I feel more comfortable with the concept of big data as a researcher? Yes and no. It seems that technology advances in data analysis are myriad, rapid and encompass a far broader sphere than the market research world. And there are, indeed, examples where this has led to a greatly enhanced business and operations efficiency.
Have to fight harder
As to the impact on MR: Given the level of visibility big data enjoys, board-level attention will probably be directed to figuring out what value new analytics tools can deliver and what the ROI looks like. The insights department may not be so much in the spotlight and we may have to fight harder for our share of the budget. It may also impact the job market, with roles for analysts crowding out more general insight functions.
Certain aspects of current MR practice may have to change. I agree with Cosentino 100 percent that attitudinal data will probably give way in importance, at least in digital spaces, to behavioral measures - probably more rapidly than we think. And yes, we do need to broaden our skill set, become better at data synthesis techniques, get acquainted with at least some of the new software tools and, most importantly, become better at business problem-solving.
On a positive note, I don't expect that in the future machine intelligence and algorithms will replace human analytics experts. In fact, it's quite the opposite.
We repeatedly read about the need to start with the business problem that needs addressing, not simply with data that is often just noise. Similarly, we are often reminded about how the best business approaches are those that merge a logical, analytics approach with an intuitive, empathetic, lateral one.
Onus is on us
For us market researchers, it suggests we will very likely have a changed future (no surprises) - one where our role could be broader (as data synthesizers), where we are still tasked to understand the complexities and contradictions of human behavior in all its fascinating irrationality. The onus is on us to rise to the challenge of a rapidly changing environment, to get acquainted with some of the tools Cosentino mentions and get used to the fact that this process of adaption is likely ongoing.
Curious, as ever, as to others' views.