Generating more questions than answers
Editor's note: Bryan Orme is CEO of Sawtooth Software, with over 30 years of experience in survey research. He is the recipient of the 2017 Charles Coolidge Parlin Award, awarded by the American Marketing Association for leadership in the marketing research industry. Find Orme on LinkedIn: linkedin.com/in/bryan-k-orme/.
AI is disrupting the marketing research industry and making many time-consuming workflows much more efficient. I don’t use the term disrupting lightly. I think its impact on our industry is bigger than the internet, online panels and big data were. This disruption isn’t without controversy: some AI applications are near-universally seen as slam-dunk wins, whereas others are fraught with controversy.
This article presents findings from Sawtooth’s annual customer survey regarding AI use and its impact on research practice followed by a comparison to the variety of AI-related opportunities referenced in Quirk’s magazine articles, editorials and columns.
What motivates and qualifies me to write on this topic? Sawtooth has provided DIY tools for general survey writing since the 1980s (we’re not just the “conjoint guys”). I’ve been at Sawtooth since the mid-1990s. We’ve kept busy over the decades in the transition from DOS to Windows to web and now with AI-enabled web surveys and analysis.
Despite the attention given to more advanced or speculative applications like synthetic data in trade magazines like Quirk’s, for Sawtooth users, AI’s greatest impact is found in more practical, day-to-day uses – particularly coding assistance and open-end analysis.
Sawtooth feedback survey
Sawtooth conducts an annual survey of its customers and this year we included questions to understand which AI capabilities and activities are having the greatest impact on their research practices. Sawtooth customers consist mainly of researchers within consultancies, client-side researchers within brands or corporations as well as healthcare and academic/government researchers.
Drawing on our own experience, as well as input from ChatGPT, we identified 12 AI-related insights research activities. We asked Sawtooth customers to consider each activity using a grid question, indicating whether their organization uses it frequently, sometimes, not at all or if they are unsure. The question and list of activities are shown below.
How much does your organization use AI for:
- Composing questionnaires
- Reviewing questionnaires for bias, clarity and design errors
- Analyzing open-end responses
- Translating questionnaires into different languages
- Suggesting interpretations of the data
- Creating reports
- Smart open-end probing (“You said XXX, tell me more about...”)
- Synthetic data for augmenting or taking the place of real respondents
- AI-generated data, for testing statistical models
- AI-led/moderated surveys
- Helping to write code (such as in R, CSS, HTML, JavaScript, Python, etc.)
- Detecting low-quality respondents or bots
We carried forward (piped) AI activities respondents reported their organizations did frequently or sometimes into a single-select question that asked which one had the most positive impact on their research practice. In case we neglected to include an important, high-impact AI activity for a given respondent, we asked an open-end where they could describe the AI-powered activity that we failed to mention.
The results are shown in Table 1 (sorted by %Use).
The striking thing is that just two activities dominated the others in terms of having the most positive impact on research practice: helping to write code and analyzing open-end responses. The frequent mention of coding is likely due to the more technically-oriented workflows of survey programmers and quantitative analysts (who dominated the sample) and may not generalize to the broader insights profession.
I find it interesting that we’ve seen so much attention at Sawtooth research conferences, in LinkedIn posts/articles and in journals/industry magazines dedicated to investigating (and hotly debating) the value of AI-generated synthetic data. Yet, a modest 22% of Sawtooth customers say they use AI-generated synthetic data and only 1% reported it having the most positive influence. This may reflect the early-stage nature of synthetic data – widely discussed, but not yet broadly adopted in practice. Or, it could indicate that synthetic data is falling short in terms of delivering real value, and I admit I am one of the doubters when it comes to the value of AI-generated synthetic data. 
Review of Quirk’s articles 2023-2025
After seeing the 2026 Sawtooth feedback, it made me wonder how closely the AI activities reported by Sawtooth users align with the topics discussed in Quirk’s print magazine – specifically across its non-sponsored articles, columns and editorials.
To explore this, I examined issues beginning with the first print edition after the November 2022 release of ChatGPT, covering January/February 2023 through January/February 2026. I manually (not relying on AI) reviewed and categorized AI-related topics mentioned across 37 articles, columns, and editorials. My tabulation is presented in Table 2.
The frequency that topics appear in Quirk’s articles does not necessarily reflect how widely they are used in practice. It may reflect more the interest level that authors and editors have in discussing them.
“Using AI to analyze open-ended responses” (text, voice and video) was the top-mentioned activity, aligning closely with Sawtooth customers’ reported most frequent usage. In contrast, synthetic data ranked fourth, being mentioned in 30% of Quirk’s articles, but ranked lower at 12th place among Sawtooth users in terms of actual use.
Sawtooth users identified “using AI to assist with coding” as the most impactful activity on their work, yet it ranked seventh in terms of frequency of mention in these Quirk’s articles. This likely reflects the fact that coding is a practical, day-to-day task for many survey programmers and quantitative researchers – highly valuable in practice, but less compelling as a focal topic for a trade magazine catering to a more general audience.
A limitation of our Sawtooth customer survey is that we did not explicitly include “AI-assisted quantitative analysis” as a category in the grid question and respondents didn’t mention it either in the open-end follow-up. Large language models can still be inconsistent in this area and these quant researchers typically rely on established tools such as Sawtooth, tabulation software, R, Excel and Python. By contrast, “AI-driven quantitative analysis” was tied for the most frequently discussed potential use of AI in Quirk’s articles.
Opportunities and pitfalls
AI’s biggest impact is not in generating insights, it’s in accelerating the work behind the scenes. Tasks that used to be time-consuming, such as coding open-ends and drafting documents, are readily completed by large language models. The challenge is making sure to double-check AI-generated output, much as you would review the work of a bright but inexperienced research assistant.
I use AI almost every day to help with writing documents, e-mails and marketing content. I also rely on it for research tasks such as refining code, conducting literature searches and for discussing statistics/methodology as if I were collaborating with a research colleague. As an executive, I use it to stress-test decisions and challenge my thinking.
For good reason, many of us have concerns about AI’s impact on the insights profession.
Will AI take my job? Experts suggest that those who learn to leverage AI will become more valuable than those who do not. However, if AI reduces the amount of work flowing into an organization or department, even the best employees who aggressively adopt AI may face pressure.
Will research budgets shrink? Some in the C-suite might see AI as a convenient reason to cut research budgets. This could lead to fewer primary research studies, reduced investment in strategic research and smaller budgets for training. “Can’t we just ask AI for the answer?” they may say. The pendulum may swing back in the future, but for now these pressures are real. AI is grounded in patterns from past data and struggles to forecast for truly novel situations or behaviors that have no close precedent. Much of the work we do in new product development, messaging, emergent attitudes and pricing involves asking real respondents about products, new ideas and prices for which the answer is hard to find in past data. Using AI for these endeavors can be like trying to drive a car while looking out the rear window.
Are we training up the next generation of researchers? We and other event organizers have noticed a drop in attendance for in-person training since the COVID-19 pandemic. While 2022 showed some signs of recovery, the recent emergence of AI may be contributing to this decline. At the same time, we’re seeing an increase in virtual training opportunities, but these fall short of replacing the rich experience of attending an event in person with like-minded colleagues and mentors. This leads us to the next point…
Are we shortchanging skill development? If AI handles many of the foundational tasks that junior researchers once performed, the next generation will miss out on critical learning experiences. Some skills cannot be shortcut, and without having performed them, researchers are deprived of these essential developmental experiences.
Are we trading off quality for AI convenience? I am cautious about relying on AI for research design, questionnaire writing and drawing conclusions from data. Sure, AI can provide valuable feedback, suggest creative directions and stimulate human thinking. But in the hands of a rushed or inattentive researcher, it can also produce shallow, unoriginal, misguided and even erroneous results. It's no surprise that three dictionaries/publications named either "slop" or "AI slop" the word of the year for 2025 (Merriam Webster, Macquarie and The Economist).
What are the implications for researchers and organizations? Consider the following:
- Focus initial AI efforts on practical workflow improvements (starting with coding, smart open-end probing and open-end analysis) where impact is already being proven.
- Be cautious about overinvesting in speculative areas like synthetic data until we see clearer value.
- Continue investing in primary research, especially for novel or forward-looking questions where past data falls short.
- Ensure junior researchers still develop foundational skills, even as AI automates parts of their workflow.
- Maintain strong quality control, particularly when using AI for research design and interpretation.