Cyberfraud is at an all-time high, impacting billion-dollar global brands, social media giants, government institutions and small businesses alike. Recently, Forbes cited research which predicts that cybercrime will have cost $6 trillion globally by 2021. Like a computer virus, fraudulent activity is evolving and new strains are surfacing every day. Unfortunately, given the incentive-driven environment of the survey world, our industry is not impervious to these threats and sample companies in particular face intense pressure to eradicate poor quality from the equation. There is no silver bullet or quick-hit therapy to save us from the various hazards that threaten sample quality and data integrity – what’s a researcher to do?
To combat these bad actors, researchers should focus on layering in various proactive mechanisms to counter online malicious behaviors, overzealous professional respondents and natural duplication resulting from multi-vendor reliance. Leveraging both technological and methodological strategies are vital elements to a winning formula; this philosophy rings true for both consumer and B2B research. While the onus of data integrity cannot solely rest on the shoulders of sample providers, as a researcher there are critical questions that you should ask of your sample partners. Doing so will position you and your study for success and mitigate the various quality risks lurking at every turn. To ensure sample quality, here are five questions to ask your sample provider:
1. How is your audience vetted and recruited and what specific sources are used for recruitment?
Control is the name of the game! In a sea of sample aggregators and brokers, the plain truth of the matter is that companies which manage their own proprietary asset have more control and quality oversight on the participant experience as well as visibility to where respondents are coming from and how they are performing. All sources are not created equal and they can have a massive impact on your data quality. Understand the origin of each source and what steps your sample provider is taking to monitor performance.
2. What information is collected among your audience?
The more a sample provider knows of a respondent and their history, the greater predictive power the firm has in forecasting future behaviors. As it relates to consumer sample, ask to see a list of the various demographics (as well as psychographics) and general profiling questions collected against the sample asset. Ask to see the B2B-oriented questions used to profile panelists (firmographics). Is this screening done at registration, throughout a panelist’s lifetime or via a pre-screener (on-the-fly, prior to survey entry)? Thorough vetting should be conducted at registration and throughout a panelist’s tenure. Waiting to screen respondents until the last minute before being delivered to your survey is a recipe for disaster and will result in false positives. Ask to see the specific questions used to ensure these are not leading in nature and if there are mechanisms in place to terminate over-endorsers. B2B-specific red-herring questions are a great way to test authenticity and true business expertise. Lastly, ask your sample provider if the firm is Microsoft Certified. Microsoft certification is the gold standard for B2B sampling and the level of rigor required is second-to-none. Working with a certified vendor will help mitigate risk and position you for success.
3. How is your audience incentivized?
Gain an understanding of how participants are motivated. What is the average incentive offered? How are incentives provided and at what frequency? Are participants rewarded when they don’t qualify? Essentially, these questions can help you understand if there are conditioning behaviors being generated to help maximize margin on the sample sale. To offset learning effects, panelists should be rewarded properly and consistently and terminating experiences should be managed suitably to avoid dishonesty and attrition.
4. What is the frequency of contact for your audience? How is your sample frame balanced?
Gain an understanding of how often panelists are participating. Does this frequency seem reasonable or overenthusiastic in nature? As a business professional yourself, call into question the likelihood of someone participating more than one or two times a week. A high survey cadence can produce respondent fatigue as well as engender overzealous behaviors (e.g., the notorious “top-two boxers”). Similar thinking should be applied for consumer sample as well. If a respondent is participating in multiple surveys per day, the proclivity for one to provide thoughtful and meaningful responses decreases significantly. Understand how your partner generates and balances the sample frame used for your study. Aside from demographics – or in the case of B2B, firmographics – what other variables are used to control for sample noise and volatility? A “quality-first” sample firm will use attitudinal and behavioral measures to create stability and replicability. Leveraging outside data sources such as the General Social Survey or the Pew Research Center can be effective strategies for blending the sample and controlling for the variability that is often present in sample sourcing.
5. What mechanisms are in place to mitigate poor quality?
Your provider should have several layers in place to prevent a wide spectrum of negative behaviors such as blatant cyberfraud (i.e., scripted bots, click farms and hackers) as well as other undesirable behaviors (i.e., dishonesty, lack of expertise and over-endorsement). Just this past year, I’ve spent countless hours demonstrating InnovateMR’s newest quality tool, Text Analyzer™. Our team is thrilled to bring this cutting-edge quality product to the market research industry. In a world of quantitative analyses, it can be difficult and time-consuming to assess the quality of open-end responses. The Text Analyzer™ uses advanced data science and algorithmic techniques to evaluate respondent answers in real time. The tool checks for gibberish, profanity, duplication, copy/pasting and bot-activity as well as context. Available in several languages, this technology evaluates the contextual relevance of the respondent answer as it relates to the specific question presented. InnovateMR’s clients are saving hours of time manually reviewing open-end responses and instead providing an invaluable qualitative assessment for their various stakeholders and constituents. This tool is just one of dozens our team has developed to increase quality. Implore your sample provider to demonstrate these quality layers. The more detail, the better! If you come away from the conversation feeling unimpressed or lacking specifics, it is time to move on.
Research done right
Sourcing high-quality sample can be an arduous road with many pitfalls and threats along the way. If you are finding exceptionally cheap and cheerful sources of sample, it is probably too good to be true. Do not be afraid to ask the tough questions. Get specific, be relentless and leave your naiveté at the door. Recruiting real B2B professionals is an investment and as such, sourcing this audience for your survey requires an investment on your end too. As I have said before, buying high-quality participants is not expensive, it is simply priceless!
Are you interested in learning more about InnovateMR’s recruitment and panel management expertise? Check out our resource library at bit.ly/36NBZ6b. Or better yet, e-mail us today to schedule a discussion with InnovateMR’s chief research officer and resident data quality expert, Lisa Wilding-Brown at email@example.com.