Stop validating synthetic data after the fact. Make predictions first.
Editor’s note: Jeffrey Mead is the founder of Lewsearch, a synthetic panel research platform. He holds an MBA. Find Mead on LinkedIn.
Sometime in the next two weeks, the National Retail Federation and Prosper Insights & Analytics will release their annual back-to-school survey. The survey they've run since 2003. The one every retail reporter cites in July. Last year it landed July 15: $858.07 average spend per K-12 household, fielded from 7,581 real consumers.
Today, before their number exists, I put mine on the record: $856.83.
That number didn't come from people. It came from 1,500 simulated parents – AI respondents generated from census demographic distributions and polled through a fine-tuned language model. It took nine minutes to field. And when NRF's real number comes out, I'm going to publish the gap between mine and theirs, whatever it is, on the same page as the prediction.
I run a synthetic panel company, so you should be skeptical of everything I say next. That's fine. Skepticism is the point of this exercise.
The problem with how our industry talks about synthetic data
Every vendor in this space (and there are a lot of us now) claims accuracy. Almost none of us can be checked. The standard move is to validate against benchmarks retroactively and privately: run the model against last year's Pew data, report the wins, quietly file the losses. The buyer has no way to distinguish a calibrated instrument from a confident guess, because every claim arrives after the answer was already known.
Researchers have earned their suspicion. G. Elliott Morris and Verasight have documented synthetic subgroup errors of 10 to 30 points in political contexts. Anyone who has actually worked with these models knows they have real failure modes: they collapse toward consensus, they miss low-salience populations and they are worst exactly where the sample is most expensive – small subgroups and novel topics.
The fix isn't better marketing. It's the oldest idea in science: you make a prediction first and then get graded afterward.
A prediction registered before the ground truth exists can't be cherry-picked. If my parent panel says 92% of back-to-school families will shop mid-July deal events, and NRF's humans say 85% again like the last two years, I'm about seven points off and everyone can see it. If my spend number lands within a few dollars, that means something too, precisely because I couldn't have reverse-engineered it.
So, we started publishing a recurring public data report built around that mechanic. Three predictions are now frozen against NRF's 2026 release: mean K-12 spend ($856.83), the share of parents who started shopping by early July (32% – and I'll flag now that NRF's differently worded instrument found 67% last year, so this one may miss badly) and deal-event participation (92.1%). The scorecard gets appended, unedited, when their data drops.
Notice that middle one. I'm telling you in advance which prediction I'm least confident in, and why: Our question offered an "I buy things as needs come up" answer that NRF's doesn't, and instrument differences move numbers as much as sampling does. Every researcher reading this knows that. It's exactly the kind of caveat that never survives contact with a marketing department, which is why I think publishing it is the credibility play, not the weakness.
An hour after publishing, the transparency paid for itself in a way I didn't plan. Staring at that 32%, I realized the study had a genuine design flaw: The simulated respondents were never told what day it is. A human survey-taker knows it's early July; my personas answered the question, "When did you start shopping?" with no idea what "now" means. So, we fielded the entire study again the same morning (still before NRF's release) with exactly one sentence added to each persona: "Today is Tuesday, July 7, 2026." The early shopping number jumped from 32% to 85%. AI-shopping adoption in "the past three months" jumped from 20% to 45%. And every attitudinal question – trust rankings, ad reactions, spend amounts – moved from zero to three points. One sentence of temporal grounding: a 53-point swing on time-referenced questions, nothing on attitudes. Both runs are published side by side, both frozen, both getting scored. What we did not do is inject news or topic context into the prompt – that would launder the very human data we're being graded against into the model's answer, and the scorecard would become circular.
What the simulated panel found
The study itself asked something the annual trackers don't fully cover yet: whether Americans are ready for AI to do their shopping. We polled 5,000 simulated U.S. adults alongside the parent sample, and one pattern stood out sharply enough to survive any reasonable error bar.
Americans punish AI on the message and reward it on the service. Told that an ad was made mostly by AI, 92% of our simulated adults say they'd trust it less: near-unanimous, across every age and income cut. But 63% would let an AI assistant complete a purchase for them within a spending limit, and 56% say a retailer offering an AI shopping assistant would make them more likely to shop there. Same public, opposite reactions, depending on whether AI is making claims at them or doing work for them.
If that holds up against human data (and that's a testable "if"), it has a practical implication for anyone spending money this season: build AI into utility, not into creative claims.
Adoption, meanwhile, is narrower than the discourse suggests: 20% of our simulated adults have used AI for any shopping task in the past three months, concentrated almost entirely among the young and the affluent. The gap between comfort with delegation (63%) and actual use (20%) is the kind of leading indicator worth tracking quarterly.
Are the numbers right? Partially, probably. That's what calibrated instruments do – they're usefully wrong by a known amount. Our pipeline's published pooled error is 7.47 percentage points across 460 cross-validated benchmark questions, and 10.68 on a fully held-out set we published even though it's worse. These consumer questions are not benchmarked, which is precisely why the NRF comparison exists.
What I'd ask of the industry
I don't think synthetic panels replace human research, and I'd distrust anyone who says otherwise. The honest use case is narrower: directional reads in hours instead of weeks, instrument pre-testing before you spend real fieldwork money and coverage of decisions that were previously made with no data at all because the timeline made research impossible.
But if a synthetic sample is going to earn a place in the toolkit, vendors have to stop asking for trust and start generating evidence. My ask is simple: Make preregistration the norm. Every synthetic vendor should be publishing predictions against upcoming human benchmarks: NRF in July, the big trackers each quarter, elections when they come – before the ground truth exists – and posting the misses next to the hits.
We'll be doing it monthly. The first scorecard arrives when NRF publishes. If the numbers are ugly, they'll be ugly in public.
The full study is public at report.lewsearch.com, and the validation methodology can be found at lewsearch.com/methodology.