Editor’s note: Greg Mishkin is a vice president of research and consulting at Market Strategies International, Livonia, Mich. This is an edited version of a post that originally appeared here under the title “Answers to the four most popular big data questions.”
2013 was the year of big data, although not always in a good way. We saw some of the largest privacy breaches in history affect major brands like Target, Facebook and Adobe, as well as government related snafus (Edward Snowden, the NSA and the Federal Reserve Bank) impact hundreds of millions of people. The public now understands that we are leaving a data trace with every cell phone call we make, Web site we browse, debit card we swipe and security camera we pass. No matter where you stand on whether our data are being used responsibly, one thing is absolutely clear…
Big data is everywhere.
In addition to what we leave behind, we willingly offer our data in exchange for valuable benefits:
- We pay up to $100 to give the government detailed personal information in exchange for faster access through TSA PreCheck lines at the nation’s airports.
- We install connected thermostats that know when we are home, away and asleep, enabling companies to learn about our daily routines.
- We wear devices like Google Glass and Fitbit to track and share our activity with friends.
- And don’t even get me started on Facebook, Twitter, LinkedIn, Pinterest, Flickr, etc.
With this trend comes an insatiable demand for big data analytics. Marketing research used to focus on asking people how satisfied they are with a service or whether they prefer Product A or Product B. But businesses want to know more about their customers than what they are willing or able to reveal in a survey. Today, businesses want to predict the future and they are turning to big data to feed a new breed of predictive analytics. Here are a few of the questions I hear most frequently from clients:
Q: What is big data in simple terms?
A: In simple terms, big data is data so large and complex that it cannot be effectively analyzed using previously established systems, processes and resources. One of the easiest ways to understand it is by the three Vs: volume, variety and velocity.
Volume means that big data is big. Big data analyses do not typically look at hundreds or thousands of rows of data, but rather billions and trillions of rows, so forget desktop spreadsheets!
Variety means that big data analyses often bring together lots of different types of data from different sources in ways not done before. Sometimes the data are structured with clearly defined rules and logic, and other times they are unstructured (like streams of comments from Twitter or Facebook). As a result, the data are usually messy and require a fair amount of cleanup.
Velocity means that big data comes in fast and changes quickly. I have worked with large data warehouses that receive more than 25 billion rows of new information each day! This type of data analysis requires processes that can incorporate newly-generated data quickly and efficiently.
Big data can come from customer data, social media data or the “Internet of things,” which includes data resulting from Internet-connected products (e.g., Web browsers, cell phones, GPS navigation and connected cameras, cars and thermostats). Market Strategies is currently working with data as diverse as call detail records from mobile phones, smart energy meters, financial services transactions and gambling patterns, just to name a few. The point is that every aspect of your customer’s life leaves behind a data trail and, when analyzed correctly, these trails can lead to immense knowledge.
To effectively analyze big data you need an infrastructure that has been specifically designed to handle it (e.g., SAS, Hadoop or Teradata) along with specially trained data scientists. Many people are surprised to learn that while big data looks at more data than most people can wrap their head around, it is almost always riddled with holes. As Nate Silver tells us in his book, The Signal and the Noise, “Data-driven predictions can succeed – and they can fail. It is when we deny our role in the process that the odds of failure rise.” To increase the chances of success people with industry and data domain expertise need to fill these holes – not machines.
Q: If my company is just starting to consider using big data in our marketing research, what would be most useful to include?
A: There is a trend to replace the term “big data” with “small data.” Most organizations are sitting on an enormous amount of untapped data and should look to their own networks for transactional data that can help build a more holistic view of their customers. The data are often siloed across disparate teams and have not been combined because the task was deemed too difficult, expensive or nebulous. But impressive ROI can be found when data are combined into a single warehouse and integrated with existing and new data points from traditional marketing research sources.
At our firm we take clients through a two-step process to identify and document the goals of big data integration and to review existing qualitative and quantitative research, as well as internal databases. The first step involves one-on-one interviews or a roundtable discussion with all key stakeholders to create a shared vision of success. The second step involves a series of information-sharing meetings to dig into all of the existing internal data and other research to determine what has worked, what has not and what prior work can be repurposed.
Often there is so much excitement to get started that clients want to skip these initial steps or complete them in a cursory manner. However, doing so can prove fatal to the overall project. Without taking the time to build a thorough analytical plan there will not be a solid road map, and the likelihood of success is greatly diminished.
Q: What about privacy? Are we even allowed to use the data we collect from our customers?
A: Based on the public debacles that unfolded last year, privacy is a hot topic. As each situation is unique, there can be no hard and fast rules; however, there are best practices to consider:
- Different industries have specific legal regulations for the collection and use of customer data. This is especially true in health care (HIPAA), financial services (RFPA) and telecommunications (CPNI). Most organizations in these industries have internal or external counsel they can turn to for industry-specific advice. Big data analytics can be accomplished safely and effectively within these environments as long as methodologies are created with the regulations in mind.
- It is always important to remember the court of public opinion. Before using your customers’ personally identifiable information, ask yourself if using it is in the best interest of the customer. When you are conducting research to improve the customer experience, it is clear that this use benefits the customer. I like to use The Wall Street Journal test. Simply put, ask yourself what would happen if the details of the proposed action became a front-page headline: Would the negative fallout outweigh the positive benefits?
Big data can be scary and intimidating to the public and it is critical to consider the impact to your customers and stakeholders. By applying The Wall Street Journal test, companies can tweak their big data projects to minimize risks. It is important for companies to have an experienced partner who appreciates the inherent hazards associated with big data analytics and is able to keep them safe while gaining the most value from the data.
It is equally important to ensure that your efforts comply with your company’s established privacy policies. If they do not, change the methodology to comply with the policy or update the policy if it is outdated. The bottom line? When big data research is conducted appropriately and ethically, there will be no need to hide it from customers or regulators.
Q. Will big data replace surveys?
A: No. Here’s why:
Big data analytics does a really good job of telling us what our customers are doing, who they are doing it with, where they do it and when they do it. However, it does a pretty awful job of telling us why our customers do what they do. Understanding the why behind the actions is critical to marketers since they are tasked with finding ways to change customer behavior. A marketer is focused on how to get people to buy more of their stuff and less of their competitors’ stuff. Without fully understanding the why behind their actions, marketers are left to guess which strategies and tactics will actually motivate customers.
There is no doubt in my mind that there is a place for both big data analytics and traditional marketing research. By skillfully integrating the two, researchers can understand what their customers are doing, why they are doing it and, most importantly, how to change their behavior. Read our complimentary white paper on this topic to learn more.
Important takeaways
- Big data is everywhere and here to stay.
- Big data will continue to fundamentally change the way companies look at their customers and their businesses.
- Big data analytics can be intimidating to the public so we must be mindful of how customers and stakeholders might perceive this work.
- While big data analytics is real, it is not the panacea that many “experts” portray it to be.
- Integrating big data analytics with traditional marketing research allows marketers to understand what their customers are doing, why they are doing it and most importantly what can be done to influence or change their behaviors.