7 ways insight teams are using AI moderation (that you’ll actually want to try)
Editor's note: This article is an automated speech-to-text transcription, edited lightly for clarity. To view the full session recording click here.
How are research teams using AI moderation in their organizations?
Erica Dinger and Betsy Shaak of Voxpopme walked through how to start using AI moderation and the benefits insights teams have seen after adopting it. Using real-world studies, they shared the seven ways insights teams are already using it and insights they’ve gained.
Session Transcript
Joseph Rydholm
Hi everybody, and welcome to our presentation “7 ways insight teams are using AI moderation (that you’ll actually want to try).” I'm Quirks Editor Joe Rydholm, and before we get started, just a quick housekeeping note that if you'd like to interact with other attendees during this session, you can use the chat tab to submit your comments and questions. Our session today is presented by Voxpopme. Enjoy.
Betsy Shaak
Hello everybody. Thank you so much for joining today. We're going to get started in just a minute here with some intros, but as you are joining, please drop where you're joining from in the chat. It's one of my favorite things about doing these webinars. So again, thank you for joining. Hi, my name is Betsy. I am the VP of product at Voxpopme. So excited to be chatting with you today about this methodology and I've brought our head of research. Erica, do you want to introduce yourself?
Erica Dinger
I do. Betsy, I love these conversations we have with each other and with all of the wonderful participants. I'm Erica Dinger. I am the head of research for Voxpopme, and we're excited to talk about AI moderation today.
Betsy Shaak
Yes, yes, absolutely. It's a good shout. While we're talking through this, please drop questions or comments in the chat. This is so much more fun when we're having an open conversation, so we're jumping right into it. Today we're going to talk about seven ways insights teams are using AI moderation that you actually want to try. It is absolutely everywhere right now. Most researchers at this point know what it is. Maybe you've even tried it.
Erica Dinger
Right. But the real question is where does it fit in? And today we're going to explore seven of the most popular and practical ways that insights teams are putting AI moderation to work. And this is based on hundreds of real-world studies that Betsy and I have run over the past few months.
Betsy Shaak
Yes, and total disclaimer here that yes, we do run AI moderation studies and we've definitely run a few, but we also focus in a lot of different areas at Voxpopme0. We are not just an AI moderation provider. So, we’ve seen where it’s worked and we’ve also seen where it hasn’t worked. And we are going to share those a little bit as we’re going through this.
Any way that you do qualitative research, we have a tool that can help you to do that all the way from, I want to speak to respondents at scale like you can with AI moderation all the way to, I'm doing some face-to-face and I need to capture those insights later down the line.
Erica Dinger
Yeah, that's exactly right, Betsy. Aside from AI moderation, there are a ton of ways to get real-time insights from respondents. You can do videos, you can see if your repository already has the answer, maybe you've asked this question or somebody else on your team has asked this question and you have access to it. You can speak to a group of respondents or you can even do live in-person interviews just like we used to do with M&M’s and Diet Coke in the back room. But today we're actually going to focus on AI moderation.
Betsy Shaak
So, this is where the real unlock kind of comes in. The technology is exciting, but the value really comes in knowing when and where it actually fits and when and where it doesn't fit. We're going to show a couple examples. Leading brands are already weaving this into their workflows and where it delivers the biggest impact. And that's where we wanted to focus today on impact.
Erica Dinger
Yeah, we've analyzed more than 100 AI moderator projects that have been run by insights teams over the past six months that we've had the opportunity to work with, and nearly half of those studies are falling into A&Us, right, attitudes and usage. But we're also seeing an incredibly strong thread of adoption across concept testing, across attitudinal studies, usage and product experiences and shopper insights, which are all really exciting ways, I think, to use AI moderation. And I think especially maybe when, it's so crazy how quickly things have come along, but maybe even just a few months ago, it felt like maybe it was only good for one or two of these types of studies and now we are seeing it adopted really across a wide spectrum of studies.
Betsy Shaak
Yeah, I think it's also because U&A is what the easiest ones to test it with, right? Your first one, let's just do it U&A to see what the experience is like. AI moderation is not a single stage tool. Sometimes you get these methodologies of tools that this like, this is only really good in early exploration or this is only good in this stage. There are definitely use cases for the methodology throughout the entire stages of research from the exploratory qual, that's the U&A cultural insights all the way to post-launch feedback. We’re seeing it, it scales, those human style interviews to fill the gaps between quant and the live moderation.
Erica Dinger
And what you're looking at here are really the top use cases that we showed in the previous slide just overlaid across the mid, early and post feedback. And as you can see, AI moderator is being used heavily across all of these stages now and particularly mid and post.
We also saw that three use cases, A&Us, perception and understanding and UX and usability really span across these multiple stages. So those are places where AI moderation is kind of being fit into each of these and building, you can definitely build on what you learned in each stage in the next stage. So that is all really interesting to see develop as this moves along.
Betsy Shaak
And UX can mean so many different things that I'm not really surprised that it's such an amalgamous term at this point.
Erica Dinger
It means many things to many people, and that's OK.
Betsy Shaak
Yes, absolutely.
Erica Dinger
So, we're going to give you a walkthrough of the top seven use cases that AI moderator is being used for. And we're going to start with A&Us. So, this is the No. 1 use case. It makes up almost half of all the projects we've seen so far. And these studies really rely on structured repeatable questions. Basically, exactly what Betsy said for making it the perfect first step in AI moderation.
You can run dozens of interviews in parallel. You can uncover trends and attitudes and you can probe automatically for richer context, which really, even though you've got those repeatable questions, it makes every interview unique. And then, concept testing is where we've seen really the fastest growth. And with stimulus testing now built into the AI moderator, you can upload visuals, you can get real-time reactions. It's really perfect for early creative exploration, pack testing or ad iteration or as you can see here, this wonderful beverage that Voxpopme is going to launch.
Betsy Shaak
And I think it's really important to mention this, that the iteration point is so important here. Gone are the days that you need to pull up to research with like, oh, we have four months of concept testing that we've already done. Now we're going to spend all this money on this one big final test. I look at it as the bowling alley bumper, right? So, when you're a kid or you're me, if you have to go bowling, you pull up your bumpers and your bowling ball's going to go all the way down. This is the bumper rails. Okay, we're going to go in this direction. Oh, we want to see if, let's pull this thread, and you're bumping down the concept test.
Erica Dinger
Yeah. Because it is so much faster and less expensive, you can actually iterate a lot more than you would sort of doing it traditionally, where you have really expensive creative, you have really expensive groups that you're putting together, it's a lot easier to do that bumper exploration, Betsy. We're going to start using that. I love that.
Betsy Shaak
Love a good bowling option. All right, so on the attitudinal and value side of the house, AI moderation works for studies where you’re exploring belief, behavior, values, participants, we're seeing, feel really comfortable talking to an AI. Obviously, there is no judgment in a situation where you're talking to an AI agent, but if you're in a situation where you really need emotional depth or the emotional way that the moderator shows up is very important, don't use AI moderation for that use case. Use a human moderator.
I think that there is definitely a time and place for the attitudinal and value studies with the agents because they're giving the participants the option to be more honest and have more reflective responses. One thing I'm really happy about is how this technology responds to, “Um, I don't know, give me a minute.” Because in the early days of this two years ago, it was not there yet. And now it's getting to the point where it's starting to feel more and more human. I don't know if anybody here has actually ever been on the phone with an AI agent for a gym or something, but it's getting to the point where you can't really differentiate the voice from a real human. So, it takes those moments of reflection very, very well now.
Alright, product experience and IHUTs. I think some of the stories here are my favorite stories about people using this methodology. Now, if you're in a situation where you want somebody to really have back and forth with a researcher about, I don't know, you want them to taste like chocolate and you want the researcher to be eating it with them and having a conversation. Don't use AI moderation for that. But there are a lot of really cool product experience and IHUT use cases.
Some of my favorite, Erica, you and I have talked about this before, the beer drinking study where they had people drink a beer at night and talk with the AI moderator. It was partly about their day and then partly about the beer that they were drinking. The AI just really handles capturing the experience very seamlessly. And the following-up with the smart probes, when people mentioned those details about the sensory experience or the usability, I think it handles it really well.
Erica Dinger
It does. And there really is something about having people do it in their own homes and in a place where it's the normal time. And that's why AI moderation is such a great fit for this. You don't have to wait for your moderator, your live human moderator to be ready at the same time as your person is, especially if you're dealing with time zones and all these other things. It's just when they would naturally eat a piece of chocolate, drink a beer, wash their face, do activities of daily living or do these activities in their own home, in their own time and just have that experience when they want to have it
Betsy Shaak
Or when they're not doing those things, but they have time to talk about it. Another one of my favorite stories about this is if you're trying to talk to new moms about diapering, you cannot schedule a time for that. Good luck. But maybe you can talk to new moms about diapering while their child is napping and they get to do it whenever they want. I love that.
Erica Dinger
You're telling me that the naps aren't regularly scheduled like ironclad?
Betsy Shaak
I wish. I wish. All right, let's keep going.
Erica Dinger
Let's talk about shopper research and path to purchase studies. On these, AI is bringing you a structure and a consistency. So, whether it's exploring journeys on Amazon, when you want to scroll with somebody, maybe you want to go to Target with somebody or you want to be in store with somebody as they search for something in their deli section. AI moderators can really uncover what's driving your choice. Do that deep dive, do that follow-up questions about, oh, why or why does it feel that way? What shoppers are noticing first, and obviously, super important, what barriers are holding them back? And sometimes it's really hard for people to really talk about those barriers, to really understand what those are. But in a conversation, it's a lot easier to get to that.
Betsy Shaak
And a disclaimer on this one, I do not recommend having an AI moderator tell people to go to a store and shop at a loud Walmart with music blaring in the back. It will struggle with this because there's just so much. If the audio is bad, the audio is going to be bad. But having them talk through that experience or maybe take it in the car after they've shopped, there can be some situations where this gets a little bit funky. Just full transparency. If it can't figure out who's talking because there's so much noise in the background, it's not going to be able to respond well, just like you or me.
Erica Dinger
Yeah, that's a really great point. And a lot of times with this, what we might do is show the shelf and ask people, what are you noticing first? And that's where the stimulus really comes in handy.
Betsy Shaak
That's true, that's true.
Erica Dinger
So, we're also seeing exploratory qualitative research. And I think this is so wonderfully important and a great use of AI. It's really bridging that gap between quantitative and qualitative research. And after a large-scale survey, there's a lot of questions. Even if you've done a large-scale survey, and you can really dig into the why respondents answered the way that they did, it's quick, it's scalable and it's going to add that human context to the numbers that you're seeing. So, instead of “Other, please specify,” you can actually dig into the why to the, what else, to the what did we miss? What did we not put in our scale, or what did we not put in our list?
And then, finally, customer experience last on our list, but not in our hearts. Many teams are now layering AI moderation into their feedback programs. And I think this is just, Betsy, you've heard me talk about this so much, when you're talking about CSAT, when you're talking about NPS, we could do a whole seminar on NPS, there's a huge difference between that, “Why did you rate it this way?” That's fine. There’s a, it's great, that's fine. And there's a, eh it’s fine and there's, that's fine.
Those are three very different sevens. And if you don't know if that seven is about to become a supporter or promoter or that seven is about to disconnect completely, you can be in real trouble. This is a way to figure out where it's going before it goes there, get you that first kind of indicator of where your score is headed and it can be crucial to insights teams.
Betsy Shaak
It's also something that can also be achieved by a video survey. But if you want that additional probing on that, yes, use the AI moderation for it. And I think something that's really, really important with the NPS or CSAt numbers is, when I'm talking to researchers about it, they don't love that for the reason that you described, but it is a quantitative measure that they can report to their boss, to their board to whatever stakeholders that you have.
Adding the color of the qual, of like, yes, the NPS score went up 2%. Like, here's some of the stories as to why we're hearing that or here's some of the responses. Adding that color of what's actually being said about it is the difference between, frankly, somebody remembering that the NPS score went up by 5% or something. And it's also the difference between connecting with those people who are actually taking that survey and who are actually giving those results. So, I'm saying that for all of these, right, this is qual, this is why qual is so powerful. But I think especially in those areas where you are really, really pushed to go into that quant space, adding some of the qualitative elements to this really helps in delivering that message.
Erica Dinger
Yeah.
Betsy Shaak
OK, so, super-duper high level, how does AI moderation compare with other methodologies?
When we did some of this research, this was for research that we did for Quirk’s New York. We did a large-scale study about people talking to AI agents. And when it comes to setup effort, AI moderation is obviously a very low setup effort. I would almost even put that down to a one now because I can literally talk to the Voxpopme platform and be like, “Hey, I'm trying to figure out this with blah, blah, blah, blah,” and everything's already set up for me at this point.
On emotional expression, you do get less emotional expression with an asynchronous video or an AI moderator than you do with human, but oh my gosh, is it better than the quant. The emotion is very, very different. And then, oh, one thing I forgot to mention, the setup effort on the human side, it's just so heavy and just so frustrating that it makes the difference between the emotional expression seem like not that much. When I did this study, I had two no-shows, and I was like, OK, well, it is what it is. Thankfully, I wasn't a researcher that had to report this on the following Monday.
Erica Dinger
Yeah. I also want to say about the emotional expression, it's actually really good. I'm always blown away. I look at literally hundreds of studies a year that we have done with our clients or for our own work, and I'm constantly just humbled by how open and honest and vulnerable respondents are, whether that is async video or AI moderator.
Betsy Shaak
Yep. Yeah, yeah, yeah. It's fair. That goes down to the topics about sensitive topics. It goes down to that relationship that we have with the truth, right? You're more likely to tell the truth if a human's not in the room. It's just kind of how it is. So, the trick is really knowing when to lean into that methodology and when to keep it personal. And that's where that researcher's background and understanding and their skills really come into play.
Erica Dinger
Yeah. What are our yes key takeaways today? The big takeaway is, we say this in everything I think that Betsy and I do together, AI moderation isn't replacing qualitative research and it's not replacing your brain as a researcher. It's expanding your toolkit. It's great for structured studies, especially ones that need scale, that need speed and they need that depth in there as well. And when you layer it with human moderation and get that empathy and nuance for those topics where you need it, or for those areas where you want to go deeper, it's great. You're getting richer insights than ever. You're balancing speed, time, cost, all of it and you're still getting insights that are really going to be impactful in your organization.
Betsy Shaak
And then, yeah, just to wrap up, everybody joining today qualifies for a free AI moderator pilot. I think it's a really good time of year to run these as well as we're going to strategy planning going into January. Some of the topics that I've seen ran for these: What do our consumers think about this strategic initiative that we're looking into for Q1? What has the competitor experience been this year? I've seen a lot of stuff felt like Christmas, right now is the time to do the Christmas shopper bits.
So, it's a great way to test one of the use cases that we were talking about today. And also in a really good time of year two to run some additional free research. So, you can scan that QR code to get started. We'll book you with a pilot with our team.
Erica Dinger
I was going to say the same as you are, right. Thank you for joining us today and talking about this.
Betsy Shaak
Yeah, yeah, absolutely. We will also drop our LinkedIn URLs in the chat. We are happy to talk about this outside of this forum as well and connect with us there too. All right. Thank you so much. Have a good day.
Erica Dinger
Yeah, thank you.