Data quality: What mobile app sessions reveal about respondent sources
Editor’s note: Henry Legard is chief executive officer, at Verisoul, Austin, Texas-based fraud prevention firm.
Data quality. It’s the No. 1 topic of conversation at every market research conference and industry networking event.
While many marketing researchers are committed to delivering quality insights to their clients, which can only come from legitimate respondents in the right geolocation, there’s still a shocking lack of transparency in the industry around which sources are the best and what normal really is.
So, what can be done to ensure quality across our industry?
Monitoring survey source quality in a standardized way helps ensure the right sources get more business. That will then encourage buyers – including aggregators, market research firms and end brands/consulting firms – to ensure they’re procuring high-quality supply. If they don't, it becomes clear that they’re contributing to the problem and putting the best businesses at a competitive disadvantage.
In response to this, Verisoul released a market research report* with 50 million sessions across 3,700+ mobile respondent sources (apps) having been analyzed. Billions of data points were taken across surveys, games and personal finance, browsers and social media/streaming.
We received a number of thoughtful questions since the report was released, and this article shares an overview of the most meaningful, actionable observations with the wider industry.
Ninety one percent of traffic is good traffic. However, 9% is risky, with 3% being extremely high risk.
You may be thinking – this seems impossibly low! And more broadly, you’d be correct. However, note that this report was for mobile apps specifically, which tend to have better data quality than web traffic. Web fraud rates are around 26%, whereas mobile apps hover in the 9-10% range. In addition, the report did not include content checks (thus, more cleaning is likely necessary beyond the 9%).
Around 80% of all respondents came from the Top 20 sources (in a report of 3,700+).
While this degree of concentration was surprising, we had hypothesized the market would follow the 80/20 Pareto principle.
What we didn’t expect, however, was a slight negative correlation between app review scores and quality. We had assumed that better rated apps would attract better users, but it turns out easy payouts are what most reviewers are looking for – and reviewing them favorably.
Location spoofing is the biggest fraud factor.
Fraudsters most commonly leverage proxy IPs to spoof locations, both to mask local IPs (especially when spinning up dozens of accounts) and to access geographies with higher-paying surveys. For example, we found that 34% of all location spoofers originate in Nigeria.
It’s also important to note that scaled, automated spoofing isn’t as common as human-led device farms. Taking surveys manually can be a lucrative career in some geographies.
Some sources are more prone to fraud than others!
It’s important to add the context of the user base and the incentives at play for the various source types. A news app, for example, may offer the carrot of “another article free” vs. a clear payout, and thus sees the lowest fraud rates of any source. In the same way, browsers and search also don’t facilitate clear payout incentives for users.
The type of source also impacts the level of fraud prevention measures already in place. Often, survey apps are built with fraud prevention in mind. On the other hand, browsers often differentiate themselves by enabling privacy settings fraudsters love.
The big question: What can I do to prevent fraud?
There are several ways in which you ensure that you’re minimizing your exposure to fraud-ridden sources:
- Look for survey-specific apps: Survey-specific apps outperform games, browsers and social apps, in accordance with incentive and payout structures.
- Larger sources have slightly higher quality: While some high-volume fraudulent sources exist, most of the big players maintain <15% fraud rates.
- Age isn’t really a proven factor: We found no statistically significant improvement for older or newer sources.
- Don’t use reviews as a primary indicator of quality. In fact, the inverse may be true. There was no correlation between high reviews and higher quality data. Turns out fraudsters also like to leave reviews – and appreciate apps that are easy targets!
*Access the mobile report (registration required) or view the “Fraud or Not” list (registration required).