The future of consumer insights: Balancing AI and human insights
Editor’s note: Jeremy Korst is president of GBK Collective, leading the company’s strategic marketing and insights practice.
While there’s been chatter about AI’s potential for years now, the recent launch of OpenAI’s ChatGPT has generated significant interest and is sparking countless predictions on how AI may impact our lives and reshape industries. While its “large language model” and “generative AI” technologies seem seductively intelligent and almost sentient on some occasions, it also raises concerns about accuracy, privacy, bias and ethical implications – as well as potential implications for the world of work.
Wharton Professor Christian Terwiesch recently put ChatGPT to the test, literally – asking it to complete the final exam for his MBA Operational Management course – and ultimately gave it a passing grade. While noting certain limitations, he found that the AI did an excellent job addressing MBA-level operations management and process analysis questions, including those that are based on case studies. He reports that “ChatGPT has shown a remarkable ability to automate some of the skills of highly compensated knowledge workers in general, and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers and consultants.”
Terwiesch is joined by others who are trying to get a handle on how many and which jobs may be most impacted by this innovative technology. A recent report by Goldman Sachs suggests that nearly 300 million jobs across the U.S. and Europe could be exposed to AI automation. As such, many are trying to understand how many and which jobs may be most impacted by this innovative technology.
How will generative AI impact the workforce?
A recent working paper by a team from OpenAI, OpenResearch and the University of Pennsylvania finds that “around 80% of the U.S. workforce could have at least 10% of their work tasks affected by the introduction of GPTs, while around 19% of workers may see at least 50% of their tasks impacted.” The researchers agree with Terwiesch’s findings that some high-income jobs and those that require a college degree may be most at risk.
One profession that could experience significant impacts is consumer insights and survey research. But what will be the balance between job enhancement versus disruption? A recent Qualtrics survey found that 80% of market researchers believe AI will have a positive impact to their roles, and 93% see it as an opportunity. Only 7% of researchers – both young and old – saw AI as a potential threat. However, the abilities to generate crisp, logical questions that effectively solicit insightful consumer responses as well as evaluate large and seemingly disparate sets of resultant qualitative and quantitative data are both essential aspects of many research roles and are seemingly well suited for generative AI models like ChatGPT.
Further, the researchers’ perspectives reflected in the Qualtrics survey are seemingly at odds with the OpenAI, OpenResearch and Penn study that identifies survey research as one of the occupations with highest potential exposure. This is in line with findings from another recent academic paper by researchers from Princeton, Penn and NYU that finds market research analysts in the top 10% of those roles potentially exposed to generative AI. [Note: I asked ChatGPT to suggest some additional recent academic papers on this topic, but unfortunately, while it provided multiple references, I couldn’t find several of the papers via Google Scholar or SSRN. So, college term paper writers and researchers, beware!]
3 job-enhancing features related to generative AI and marketing research
Discussions with researchers on my team at GBK Collective as well as research professionals and our clients have raised several potential benefits and job enhancing features related to ChatGPT and similar AI tools:
1. More efficiently developing survey questionnaires and discussion guides. ChatGPT seems well suited to help draft or refine a survey or discussion guide. In fact, I have been working on a survey related to electric vehicle purchase considerations. So, I asked ChatGPT to draft a set of survey questions for me. While it took some iteration, and for some reason the tool couldn’t get past 17 questions even though it promised 24, ChatGPT came up with a well-constructed list of multiple-choice questions, including several that were not yet contemplated in my own survey draft. It seems likely that AI models will find a home helping researchers, marketers and product managers create and refine survey drafts.
2. Identifying insights or context not readily apparent. Machine learning and analytics have advanced the research profession by enabling analysts to more easily glean insight from melding multiple sources of internal customer data and third-party data along with results from primary research. ChatGPT takes this an important step further by adding more context to those insights. This also includes the ability to “read” and synthesize open-ended questions (like a word cloud on steroids) as well as coloring primary insights with supplemental sources of corroborating or contradicting materials from the public domain. All of this can help brands develop a more complete picture of their target audience.
3. Testing or simulating study results. Given that AI is trained on very large and diverse data sets, it could be well suited to test surveys or even simulate potential results. In fact, Cambridge researchers recently found that AI had the ability to predict certain political polling results and that “proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups.” As AI models progressively improve, it’s not beyond imagination to think that one day marketers and product managers could just “ask” an AI chatbot for insights and input that historically would have taken a separate primary research study. This has the potential to make consumer insights more accessible and efficient for companies of all sizes, fueling more customer-centric and outside-in mind-sets.
In the Qualtrics survey, researchers identified several areas that could be impacted or augmented by AI:
3 limitations and concerns related to generative AI and marketing research
Even beyond ChatGPT’s well-documented proclivity to provide confidently inaccurate information on occasion, as well as potential biases, there are additional limitations and concerns related to the use of this tool by marketers, researchers and insights professionals:
1. Inability to predict breakthrough trends or innovations. Given that AI models are trained on vast troves of existing data, one potential shortcoming may be in the limitations to comprehend or predict completely new trends or innovations. That is, if the underlying data do not yet include signals related to these future happenings, the model may not be as facile in reasoning about it. So, researchers and marketing executives should be cautious when trying to use AI to understand or predict truly breakthrough trends or discontinuous innovations.
2. Lack of emotional understanding. While ChatGPT excels at reasoning over logical and factual data, it’s not yet adept at understanding human emotions, latent needs or (seemingly) irrational behaviors or outcomes. This means traditional research will likely still be needed when trying to understand important emotional or psychographic considerations. Companies must be aware of the limitations of AI in this regard and ensure that they use it in conjunction with other research methods and managerial judgement when developing strategies.
3. Customer privacy and proprietary data. A key strength of generative AI models may be one of their biggest weaknesses: the ability to meld and “learn” from all the data they ingest. This raises complicated issues around respondent privacy and potentially exposing proprietary company and customer data. Researchers must ensure that they have appropriate safeguards in place to protect respondent privacy and that they comply with data protection and privacy laws. Further, whenever considering the use of shared AI tools to analyze sensitive data, users would be wise to also consider potential competitive risks – who wants to train ChatGPT to become “smarter” for one of their competitors? Some companies are already taking action to limit its use, including several large Wall Street banks. And a recent Gartner poll showed that nearly half of surveyed HR leaders were working on formal employee guidance for the use of ChatGPT. Perhaps future development and commercialization efforts will include separate private “instances” of the AI chatbot that protects company-specific proprietary data and interactions within a corporate firewall. Perhaps OpenAI should conduct an “old fashioned” survey to understand how many and which companies would be willing to pay for these features?
The future of AI and consumer insights
The question on whether AI will be good enough to complete meaningful cognitive tasks or assist with traditional research has been answered. Now the question is how will it – and we – evolve and adapt?
While AI has incredible potential as a tool for consumer insights and research, it’s important to strike the right balance between AI and other research methods. In other words, brands need to stay aware of the limitations and risks of solely relying on AI, feeding it more and more data. While AI is emerging as an incredible tool to augment human intelligence, it doesn’t replace the need for critical thinking by teams, asking the right questions of data and constantly seeking out better data and insights to understand human behavior. Ultimately, survey research professionals need to ensure they understand and are adept at using this quickly moving technology.