Stop Bots, Not People: Survey Design in the Age of Online Fraud
Editor's note: This article is an automated speech-to-text transcription, edited lightly for clarity.
Fraudsters have lots of programs and tools they use to gain access to and complete surveys. All they care about is the ability to gain the incentive at the end.
Toluna has learned what the most common tricks are and found tactics to combat them. These tactics lead to better data quality and when done right do not affect real respondents.
On September 25, 2025, during Quirk’s Virtual Sessions – Data Quality series, Marie Hense, global head of quality and Yosra Ahmed, junior associate director of Toluna gave a presentation on what tricks fraudsters use and tactics to combat them.
Session transcript
Joe Rydholm
Hi everybody and welcome to our presentation, “Stop Bots, Not People, Survey Design in the Age of Online Fraud.”
I'm Quirk’s Editor, Joe Rydholm. Thanks for joining us today!
Just a quick reminder that you can use the chat tab if you'd like to interact with other attendees during today's discussion. And you can use the Q&A tab to submit questions to the presenters, and we'll get to as many as we have time for at the end during the Q&A portion.
Our session today is presented by Toluna. Marie, take it away.
Marie Hense
Thanks so much Joe, and welcome to our session everybody.
My name is Marie, and I'm the global head of quality at Toluna. I'm joined today by Yosra Ahmed, who is our junior associate director, one of the research experts from our in-house research team.
Together what we'd love to do over the next 30-minutes is to talk all things fraud detection when it comes to in-survey quality screening, but also how we balance that with the experience that our real respondents have as they're going through a survey.
So, Yosra will be here to provide us with some real-life examples and insights from her day-to-day execution of research. And hopefully, at the end of the session, we'll have provided you with some interesting insights, as well as some hands-on recommendations that you can take away and implement in your own survey design going forward.
So, with that being said, what I would love to start with today is having a look at how fraudsters actually operate, because understanding how they operate is directly related to how we design our surveys and how we understand what we need do to tackle fraudulent activity in our survey design.
First of all, this is how most of them operate. There'll be other ways of working, I suppose, but what is important to fraudsters in a nutshell is to defraud as many surveys as possible in as little time as possible, so being efficient and effective in their operations.
What they will do is they will try to create as many survey accounts as possible on all available platforms. They will then try to identify appealing surveys, and we'll talk about in a moment what an appealing survey looks like for a fraudster. They will then try to establish a safe path through this survey.
So, figuring out how to qualify for the survey easily and then figuring out how to circumvent all of the existing quality checks.
Once they've figured that out and they've made it to the end of a survey, they will automate this path. They will use automation. You'll hear of bots quite a lot, but bots are essentially just that. At the beginning there was a human that found a way through a survey and then automated this survey path with the help of algorithms or tools to then take a survey over and over again.
And that's exactly what they do in the last stage.
They then target this survey that they've identified and they use the automation, the bot that they've programmed, to take this survey as many times as they can from their different accounts. That then means they can really, with one bot, try to really maximize their gain in this survey.
Now a question that I get asked quite a lot is, ‘is this actually worth it?’
The answer, unfortunately, is yes, it is worth it because what we're looking at is international fraud, global fraud, even.
We're not looking at fraudsters from the U.K. defrauding surveys in the U.K., or fraudsters in the U.S defrauding surveys in the U.S. That is less worth it. But what is worth it is, for example, a fraudster in India defrauding surveys in Europe or North America.
The reason that's the case, now this is some simple math here, is the national minimum wage in India is about 178 growth piece per day. Now 178 rupees is about £1.60. So, to gain £1.60 in survey incentives really does not take that many surveys. So, if they are quite successful and effective in their operation, then they can make quite a lot of money.
The key things here are exchange rates and the international aspect of fraud.
For example, we see quite a lot of fraud in the U.S. originating from Venezuela because in Venezuela the U.S. dollar is very strong. So, unfortunately, the reality is fraud is worth it, and it will not go away.
That is key to remembering when it comes to protecting our surveys and protecting our research because we can never let our guard down, neither when it comes to respondent screening nor survey design.
Now these fraudsters are highly organized.
They communicate with each other and they share their approaches. You can go on social media and you can find videos and instructions on how to defraud different types of platforms and different types of surveys. They share their knowledge, they share their tools, they share their insights on what kind of surveys are easier to defraud than others and what kind of technology to use to do that.
What is extremely important is to have a fraud screening approach that is multilayered, that happens at lots of different stages in the research process and that uses a lot of different data points.
For example, screening a respondent at the registration stage before they enter a survey, when they're in the survey, which we'll talk about more today, but also after the survey is done and using different data points, such as device information, behavioral information and identity information, because over the years the fraudsters learned how to manipulate a lot of these data points, such as the device settings on their mobile phone or on their laptop or desktop to make themselves look authentic and genuine.
So, the more data points we collect and the more varied those data points are, the more effective our fraud screening approach. Having these different layers at different points in time and from different angles is absolutely key to an effective fraud screening approach.
Now let's talk about what fraudsters love.
We already said they communicate with each other, they share information about which surveys to look for, which ones are the easiest to target and, of course, the one thing they're after are the incentives.
They love surveys with high incentives, and that means if you've got a survey that has a high incentive, for example, looking for a really niche audience or you've got a long LOI, long length of interview, that means your survey will possibly be targeted more than other surveys. So, that's one thing to watch out for.
But there are a few other things that fraudsters love.
They love obvious screeners. The easier it is for them to make it into the survey, the less risk they run of being taken out of the survey or terminated in the survey the easier it is for them to be effective and successful in what they're trying to do, which is trying to get to the end and get their incentive.
Obvious screeners make that really easy for them.
For example, if the first question of a survey is, ‘Have you visited Heathrow airport in the last week? Yes, no.’
Well, the fraudsters have an easy time guessing who we're looking for. We're probably looking for people who have visited Heathrow Airport in the last week. And even if not, if they say ‘yes’ and they get screened out, they know that ‘no’ is the right answer. They'll just use one of their other accounts to try to make it in.
So, having multilayered screening questions that are not obvious are absolutely key, and the first defense to fraud detection.
Now of course, fraudsters don’t like quality checks because with quality checks they get removed from the surveys. A survey without quality checks is great for them because they don't have to worry about being removed at any point once they've made it through the screener.
Let me jump to the bottom item here. Even better are surveys with automated quality checks.
Why? Because if fraudsters get terminated in real-time, that makes the survey much less appealing.
If a fraudster knows that quality checks are only applied post field, maybe two days, three days, maybe even a week after field fieldwork closes, they know that they can make it through the survey. They can get their incentive, and by the time they are detected for being low quality, they've probably already cashed in and they're already achieved their goals. So, surveys that don't have real-time automated quality checks are quite appealing to them as well.
Then let me go to the yellow box here. Surveys without open-ends are quite appealing to them as well.
Why is that the case?
Well, to look authentic and genuine when it comes to open-ends, there can't be repetition because of course we check for repetition across open-ends. So, they have to put more effort into their bot and into their automation so that every time their bot runs through the survey, it provides a different open-end.
More effort means more time means less efficiency. The fraudsters don't necessarily like that, but they'll still go for it. Having an open-end doesn't completely scare them off, but it means more work on their end. And of course, more work means less efficiency.
So, having an open-end in a survey is one of those many points that we're talking about today that can help with making your survey less appealing to fraudsters.
Yosra Ahmed
Okay, so it's widely known there are some traditional data checks, which we do as researchers when it comes to open-ends, speeders, attention checks and straight lining.
For open-ends, we check gibberish answers, foreign languages and things like that. For speeding, we check complete central durations. For attention checks, we add red herring questions such as, ‘What year are we in?’ For state for straight lining, we check unreasonable repeated selection of the same answers in a grid.
But fraudsters know about these checks and they're usually easy to spot, such as standalone attention checks.
In that case, we need to be smarter about how we identify inconsistent behavior, while also developing these classical checks to become better.
For example, in open-ended questions, we're not just looking at what is written in the text box, but how it got into the text box using metadata. Such as whether copy and paste was used to fill out this open-ended question.
It's important to mention here that these checks still have a place and they are an important hygiene factor. However, we can only expect to catch lazy fraudsters or real respondents who were inattentive.
Also, classic red herring questions or attention checks can be quite disruptive to the survey experience for a real respondent because they traditionally tend to be unrelated to the survey topic and break the respondent's train of thought.
Let's have a look at some case studies and some markets with specific challenges.
It's important to keep in mind that not all markets are the same, but rather some markets are more challenging than others, making it harder to identify fraudsters and requiring an extra layer of checks, which we will go through in a few minutes.
Let's give some examples here on markets with high fraud rates or high fraud cases such as North America, Europe and Australia. They have strong currencies and lots of survey supply, which makes them a great opportunity for fraudsters.
This is also prominent in China, India and Southern Asia because a lot of fraud originates from there. So, many devices and identity checks are less effective, making it easy for fraudsters to get in.
In addition, India and China are markets with growing research demand and increasing fraud opportunities.
Markets like Australia are also an easy catch for fraudsters as they wake up in the Southern Asia time zone, so they can easily start their day by trying some surveys while they're waiting for Europe and North America to wake up. They know that they would be caught more easily if they were completing surveys at off hours. So, they're killing the time until those markets are safe.
One of the very challenging markets that we conduct research in is the Chinese market where a lot of fraudsters exist. What makes this market even more challenging is that they have their own ways to pass through the standard quality checks, which we have just mentioned.
In China specifically, this is because lots of fraud originates from there but also because of strict local privacy laws which make it more difficult to conduct extensive device and identity checks like in other markets.
Therefore, focusing on behavior in survey quality checks is extremely important in these kinds of markets.
The question here is, ‘How can we identify these fraudsters and what are the signs in data that can signal such a risk?’
So, as researchers, we have noticed things in our research that can help us in that.
First are the monthly fluctuations.
As we all know, generally monthly fluctuations are normal, but it becomes suspicious when it hits some KPIs such as awareness or consideration, which normally don't vary significantly from one to another.
One of the most important KPIs to check is aided brand awareness because fraudsters are usually lazy, making them pick a few brands and brand awareness questions to make it quick. So, checking this KPI can also be of great help.
In addition to that, the data sense check is also important and crucial, because data collected can largely contradict market data or external data sources, which is also a sign of fraud or a data quality issue.
All of this just comes on top of the standard quality checks that we have just discussed a few minutes ago, such as poor quality, open-ends, illogical responses, et cetera.
This brings us to an important question, which is, ‘What additional layer of quality checks can be included to ensure the data quality?’
Traditional quality measures, which we have previously discussed, including red herring questions, are not enough obviously. Here we are listing a few additional layers of checks on top of the traditional ones which still should be utilized.
Number one is adding fake brands to the list.
As researchers, we have noticed that fake brands have proven to be an effective trap for fraudsters since they do not spend enough time to check the brand list in detail or fake options in general.
Number two is checking non-quota variables. In fact, non-quota variables can be also checked for abnormalities.
An example for this could be the purchase frequency, ethnicity attitudes, urban versus rural, or it is in China. Although income might also be considered as a quota variable in some markets, it can still be checked because fraudsters sometimes pretend to be from hard-to-reach audience because they know they're hard to get and they have higher likelihood of qualifying.
Another quality measure that can be followed is checking the data while fieldwork is running.
As Marie mentioned, instead of doing the checks at the end of fieldwork and after fieldwork is closed, doing these checks while fieldwork is running can save more time and help in speeding the process of identifying these fraudsters and taking the corrective actions.
For example, you can swap your quality checks in the surveys, such as the attention checks to break the automated bots, because usually these bots are pre-designed, so this helps in disturbing the process of the bots.
Another suggested technique to manage the data quality is having some variables that make the checking process easier and more efficient.
Since fraudsters are usually very lazy, and this is something not necessarily unfortunate for us as researchers, as we discussed they tend to choose few brands at most of the questions, especially the brand awareness question. So, having an automated or a hidden variable for these kinds of KPIs can help flag those respondents automatically and identify them as fraudsters if a pattern exists. As well as monitoring these brand KPIs in general.
Marie Hense
So, summarizing what Yosra talked about, the best quality checks are those that are tailored to the survey.
Examples that she mentioned were inconsistencies between answers, so illogical answering behavior or very unlikely combination of answers. An example here being, someone who says they're 18 years old and they have five children. I mean that's a classic one, but again, not impossible but unlikely.
This is about collecting as many data points about consistency or inconsistency as possible, or somebody saying they use a brand but they're not aware of it. It's possible that in the moment of spontaneous or prompted brand awareness, they forgot about the brand and then they remembered that they actually use it. But again, it gives additional data points on whether or not someone is being consistent in their answering behavior.
The key thing to remember is that fraudsters don't think about their answers and they are not telling the truth, they are lying. So, staying consistent, especially across an entire survey can be quite difficult for them.
So, consistency and logic checks are a very effective way of identifying any outliers.
Now, Yosra also mentioned overstatement and understatement.
Understatement being if someone selects far fewer options in a question than expected. So, as she said, sometimes fraudsters are lazy, so they will only select one or two answers to quickly make it through the question to the next. But we've also seen the opposite where they select a lot of options because they think that increases their likelihood of staying in the survey. So, both sides of the coin need to be monitored. And hidden variables, like Yosra said, are a really good option to do that effectively.
And fake options, whether that's fake brands or fake sports or fake countries or whatever it may be, but again, fraudsters don't think about the content of the survey that they're answering. They're not telling the truth, they're not recalling actual events or actual purchase behavior or actual awareness or actual opinions. They are just trying to manipulate the survey and make it through as quickly as possible. So, having these trap questions in there is really important.
But then on the other hand, as Yosra mentioned, metadata is very important as well. So, that's anything that doesn't come from the survey input itself, it's the surrounding data.
So, for example, response timing, if there are a lot of responses coming in at off hours, so between local hours of let's say 1:00 a.m. and 5:00 a.m., that could be an indication that something is going wrong with the survey.
The same goes with looking at device information such as the distribution of operating systems or browsers used in the survey.
As an example, a few months ago we observed a sudden influx of surveys being completed from Linux devices. Now we know that Linux has a very small market share in most countries across the world. So, when there is an increase and a much larger proportion of completes coming from Linux, that is an indication that something isn't quite right.
In fact, we identified that there was a trend, if you want, a few months ago among fraudsters to use Linux. Monitoring this then allowed us to identify, as quickly as possible, any kind of inconsistencies. So, this metadata is extremely valuable when monitoring and evaluating the quality of a sample as well.
And like we said, the respondent level quality checks are extremely important, but aggregate level quality checks are equally important. So, validating data with secondary data sources such as sales data, cross-checking data points with each other to see whether they make sense.
So again, these logic checks, but to see in a brand funnel. For example, if we see awareness of a brand is much lower than awareness for an ad on an aggregated level in the same project, that's an inconsistency that needs to be looked into because that could indicate that something's not quite right.
And the same goes for tracking studies, ensuring that we check wave on wave movements and look at consistency there.
Now we may say, ‘okay, well why don't we just include as many quality checks as possible in our survey design because fraud clearly is a challenge that we are facing as an industry.’
We've learned that we want to identify inconsistencies, we want to trap fraudsters, we want to include open-ends because they make it more laborious for fraudsters to program their bots and so on.
So, how about we include as many quality checks, as many attention check questions, as many open-ends as we can?
Well, we could do that, but we need to think about our real respondents as well. And this is where the real balance comes in because if we only think about fraud detection when it comes to quality of data, then we forget about the needs of our real respondents.
What are the needs of our real respondents?
Well, let's put it this way, participants take a survey because they've got, let's say five to 10, sometimes more minutes to spare. What else could they be doing in this time?
They could be going on TikTok. They could be going on Instagram. They could be writing a message on WhatsApp. They could be watching a video on YouTube.
All of these applications that we as researchers are competing against are sleek, addictive, easy to use and have a high entertainment value. We are not competing one survey against the other. We are competing against those other ways of spending their time. So, we need to ensure that our surveys feel as rewarding as it would be to spend the next five to 10 minutes scrolling through TikTok, for example.
Now of course this is different and it's rewarding in a different way, but no incentive that we could provide to respondents would be enough to make them force their way through an unenjoyable survey experience. What we see is that respondents drop out and they may not come back.
What that means is with every survey that we design, we need to not just think about fraud screening and creating extra hoops for fraudsters to jump through. We need to remember that those are hoops that real respondents have to jump through as well.
So, integrating our quality checks as much as possible is absolutely key to retaining our respondent's attention, motivation and to compete with all of these other ways that respondents could be spending their time.
Some examples of things to think about when it comes to balancing fraud screening or quality screening with the experience of respondents.
Metadata is a really good example because that is data that's being collected in the background. The respondents don't need to do anything for that data to be collected, time of day, device types, things like that. They don't need to answer any extra questions. They don't need to jump through extra hoops.
So, thinking about the right metadata to monitor for your survey means a good respondent experience, but still plenty of data quality points for you.
Integrating quality checks into the survey design as much as possible is another point.
Standalone attention check questions. If they must be, then they can be included, but a much smoother experience and option for respondents is to include them in the questions themselves and to tailor them to the topic of the survey.
They also make them stand out less, which means that they're not as obvious for fraudsters. So, there is a win-win there in terms of survey taking experience and identifying fraudulent respondents.
Another, the third point is combining existing questions. So, using the questionnaire to your advantage, identifying illogical behavior and inconsistent answering and using those as flags.
And then also of course, pattern recognition, things like straight lining and repeated answer patterns. This again leverages the existing questions from the questionnaire and looks at behavior on those questions rather than creating extra hoops for respondents to jump through.
Yosra Ahmed
Okay, so now after we have learned more about fraudsters, their tactics and how to spot them, let's briefly sum it up.
First of all, as we have discussed, survey fraud is worth the effort and fraudsters will continue, especially in the AI era because AI helps drive automation more efficiently.
Despite the traditional checks being no longer sufficient on their own nowadays, they're still important as a first layer of checks. It's also important to keep them fresh and develop them over time, such as not only looking at content of open-ends, but also how it got there.
In addition, not all quality checks can match every survey, so it's important to use the ones that match your survey for better respondent experience while ensuring the data quality at the same time.
Also, as Marie mentioned, metadata can be of great help when available. By metadata, as also Marie explained, we mean survey related data that's not survey data per se, such as behavioral respondents. This can include copy and paste, typing, speed, time of day, device information and other kinds of information which we can use as also a quality check.
Moreover, quality checks on the respondent level are important but are not sufficient on their own. This is because some data quality issues appear on the aggregate level, and they do not appear on the respondent level.
Last but not least, it's important to understand that not all markets are the same, making it crucial to understand your market and its relevant challenges so that quality checks can be tailored according to your needs and to your market as well.
So, with that, we come to the end of the presentation. We want to thank you so much and we would like to open the door for any questions that anyone might have.