Artificial intelligence and machine learning
Editor’s note: Gaurav Kapoor is president of messaging optimization company Newristics, Scottsdale, Arizona.
Artificial intelligence (AI) and machine learning (ML) are already revolutionizing many industries like automotive and health care. There are self-driving cars on the road and machine learning algorithms can accurately predict tumor regrowth in lung cancer patients, outperforming clinicians by a wide margin.
How advanced is the market research industry in its use of AI and ML? Is it at the cutting-edge of AI technologies or still in the early stages of adoption? Almost all major marketing research companies claim to be using AI and ML in their offerings, but insights professionals working for large companies often lack transparency on how AI and ML is being used by research vendors. To many insights leads, AI and ML still feels like a black box and the risk/reward tradeoff of incorporating artificial intelligence into research projects is not something they can clearly evaluate.
The use of AI and ML in market research can be organized into six levels of increasing complexity and advanced use of algorithms. AI and ML are poised to disrupt the insights profession, making way for new companies that can forever change the way insights are captured from customers and converted into stories that can build brands.
Level 1: Improving research fieldwork
1. Panel management.
Panel companies have always had to manage engagement and churn among panel members. Machine learning algorithms can be very effective in predicting who is about to churn in a panel and when based on historical panel member engagement data. Conversely, ML and AI can also be used to mitigate churn and trigger engagement strategies and programs to retain panel members.
2. Real-time fraud detection/ID verification in surveys.
Whether companies are using consumers from their own first party databases or sourcing respondents from panel providers, there is a need to verify respondent IDs, prevent survey fraud and eliminate professional respondents or cheaters. ML and/or AI is ideally suited for real-time verification of survey respondents and detection of fraud signals in live surveys. Algorithms can be trained on past survey data to predict future instances of survey fraud and take real-time action to improve quality of surveys.
Level 2: Accelerating research projects
3. Qualitative research insight mining.
ML and AI are making qualitative research faster, cheaper and easier by automating the most manually tedious task of qual projects – mining insights out of the research. Speech-to-text algorithms have improved in accuracy and can transcribe qualitative research like focus groups, triads/dyads and one-on-one interviews within seconds of completion of the research. Advanced text analytics algorithms can also analyze the transcripts and summarize each interview for the researcher’s review and analysis.
4. Quant open-end survey response coding/analysis.
While quantitative research is not text heavy, open-ended response coding in quant surveys has also been a manually tedious task that is being replaced by natural language processing (NLP) algorithms. Open-end questions can be a rich source of organic insights, but due to the extra cost and time implications of including lots of open-end questions in a survey, marketers and researchers had deprioritized open-end questions in research. NLP can now analyze the text in open-ends and code them based on themes/topics, which can accelerate time to reporting.
5. Agile survey design and development.
DIY research platforms have been gaining popularity for years now and many of them offer AI-enabled survey design and development functionality. Natural language generation (NLG) algorithms can generate human-like text and can be trained to write survey questions by learning from past examples. NLG models can also make real-time suggestions, propose edits/enhancements and even create alternative versions of survey questions developed by humans.
Level 3: Improving actionability of research data
6. Finding hidden patterns in research data.
Unsupervised machine learning algorithms can look for patterns in data sets without a priori hypotheses. While historically such algorithms could only find patterns in large data sets with real-world data, increasingly ML algorithms can be trained on small data sets as well and can look for hidden patterns in market research respondent-level data.
7. Projecting survey data to real-world databases.
Marketers and researchers in many industries must project research data from hundreds of respondents to real-world databases with potentially millions of customers. ML algorithms can be used to build predictive models that can score customer databases using data from primary market research studies. While the predictive accuracy of such models can vary from one study to another, they do offer a way to make customer databases more actionable by augmenting them with research data when real-world data is not available.
8. Conversational AI for reporting insights from research.
Conversational AI can be used to change the way brand teams access results of any market research study. Instead of pouring through 100-page decks, running crosstabs online, or looking at data tables in Excel, brand teams can now “talk” to AI systems and get answers to their key business questions in human-like responses derived from the research data.
Level 4: Leveraging past/existing research for new insights
9. Search algorithms for indexing past research to improve discoverability.
Large brands can conduct thousands of research studies over time and often struggle to find the information they need quickly because of the large amounts of distributed research content they own. Advanced search algorithms can be used to search and index documents internally within corporations and improve the discoverability of not just past market research reports, but even specific customer insights embedded in these reports. While Google-type search algorithms are designed to go broad across billions of webpages, enterprise search algorithms are designed to go deep within documents and index at a more granular level so that marketers and researchers can simply search for insights on their servers and find documents that feature such insights.
10. Extracting meta insights from past studies and presenting narratives.
Some of the market research conducted by brand teams is standardized in structure and is repeated often (e.g., ATUs, brand health, message recall). Advanced language models can be used to analyze data across waves of studies and create text-based narratives that are easy to understand. Text-to-speech algorithms can then be used to convert text summary narratives into speech. Conversational AI allows you to ask questions like, “Why was my brand equity score fell below Competitor X in this tracker” and get an explanation instantly.
11. Insight aggregation and dashboarding across studies.
With the advent of DIY market research platforms, brand teams are now used to accessing data and insights from research studies through dashboards. AI is now also being used to aggregate insights across many studies and to make them accessible through business intelligence dashboards.
Level 5: Creating new types of market research
12. Social listening research.
NLP has enabled an entirely new way to gather organic customer insights from social media conversations and posts. Analysis of unstructured text from social media posts is being used for many use cases like brand sentiment, trends monitoring, drivers/barriers identification, customer journey and more.
13. Voice/video-based research.
AI is transforming data collection in market research beyond surveys to voice and video-based respondent feedback. AI can now be used to analyze large audio/video files in seconds and extract information on behaviors, emotions, brands, people/personalities and much more. Video-based research is especially redefining the field of shopper insights as AI-enabled analysis of shopper videos at scale is now possible.
14. Chatbot-based research.
Chatbots are shaping customer experience for many brands and are also opening a new avenue to get feedback from customers. Chatbots already being used to engage customers on websites can also be used to collect customer feedback through microsurvey questions. Chatbots can even execute adaptive survey flows and customize questions based on customer personas.
Level 6: Replacing primary marketing research
15. Predictively scoring ideas without research.
A large proportion of all marketing research conducted is to test ideas – concepts, messages, claims, packaging, etc. Deep learning algorithms can learn from past idea testing research and can be trained to predictively score ideas in the future without any customer feedback at all.
16. Predictively analyze effectiveness of brand messaging without research.
Brand teams spend a significant percentage of their research budget on message effectiveness related work through message recall studies, ROI analytics, competitive messaging audits, etc. AI can be trained to score messages on effectiveness and messaging from all brands and all channels in a product category can be analyzed at scale to create a messaging effectiveness scorecard without conducting any primary market research.