The trap of data analysis
Editor’s note: Tim Bock is founder and CEO of Displayr and founder of Q, a Sydney-based analysis market research software firm, with over 25 years of experience in data science and market research. He holds a doctorate in marketing from the University of New South Wales and publishes weekly insights on the state of AI and research in his LinkedIn newsletter “Understanding People at Scale.” Find Bock on LinkedIn.
Every now and then I get into a small panic about AI and the future of data analysis.
I give a data file to Claude or ChatGPT, ask it to analyze the data and write a report and the output is often impressive. The structure is good. The writing is good. The charts are getting better. The tables can be great. The interpretation is often plausible.
As somebody who creates data analysis software for a living, this causes the odd moment of terror. But then, happily for me, I find mistakes.
I can spot the mistakes because I use datasets I know well. I can spot the mistakes because I have spent 37 years studying, practicing and teaching data analysis. I can spot the mistakes because I use AI a lot and know where it tends to go wrong. And I can spot the mistakes because ChatGPT often does the analysis in Python, which I can read.
The problem is not that AI cannot do useful data analysis. It plainly can. The problem is that AI data analysis is most useful when the human using it is already expert enough to notice when it has gone wrong.
That creates a nasty trap.
A novice can ask AI to analyze a dataset and receive something that looks like expert work. It may have tables, charts, statistical language, neat conclusions and confident recommendations. But if the analysis is wrong, how would the novice know?
This is where people often reach for the Dunning-Kruger effect: the tendency for people with low ability in an area to overestimate their competence, partly because they lack the skill needed to recognize their own mistakes.
AI can produce confident-looking answers without having a reliable mechanism for knowing whether those answers are correct. And now humans armed with AI can do this too.
Making AI useful for marketing researchers
A more charitable version of the same point is that AI creates a lot of false positives. It finds patterns that may not be real. It explains things that may not be true. It gives answers that sound right before they have earned the right to sound right.
How serious is this problem?
I think it is more serious than it first appears. I believe there are two loops that make AI useful.
The first is loop engineering: get AI to do the work, check the work, revise the work and repeat until the output improves. This is how AI has become so good at writing code. It writes code. It runs the code. The code fails. It reads the error message. It fixes the code. It tries again. The loop works because the feedback is often immediate and objective.
The second loop is the human in the loop. This is how I write most of my articles. I create an initial draft, ask AI to improve it, read the output, push back, edit, repeat and keep going until I’m happy.
In both cases, there needs to be an evaluator. Some mechanism needs to decide whether the work is good.
With code, the evaluator can be automated. There is a mouthful of jargon that describes how software automatically evaluates software: CICD, linting, regression tests (no, not diagnostics for driver analysis – same words, no common meaning) and compilers. The software industry has spent more than 60 years building ways to tell programmers when something is wrong.
AI can use that feedback. Data analysis is different.
If you calculate the bottom two-box instead of the top two-box, the software does not necessarily throw an error. It just gives you the wrong answer.
If you accidentally filter the wrong group, use the wrong weight, ignore a missing value code, reverse a scale, double-count respondents, confuse correlation with causation or interpret a random fluctuation as a meaningful pattern, the output may still look perfectly professional. There may be no error message, crash or obvious sign that anything has gone wrong. That makes loop engineering much harder.
The AI can check whether the code ran. It can check whether a chart was produced. It can check whether the report contains headings and conclusions. But it cannot reliably check whether the analysis is substantively correct. And the human-in-the-loop version has the same weakness.
If the analysis says a brand’s market share is 21%, but the truth is 12%, how would you know? Sometimes you can spot a result that looks implausible because it clashes with what you already know. But the whole point of market research is to study things you do not already know. That means your ability to evaluate the answer is limited precisely where evaluation matters most.
Getting the numbers right
This is why I am not yet convinced that general-purpose AI models can safely replace serious data analysis workflows. They can assist, accelerate, suggest, write code, summarize and help experts get more done.
But unsupervised data analysis is a different thing. The bad news for many AI enthusiasts, and the good news for people like me, is that reliable data analysis remains harder to automate than it looks. Maybe I’m just focusing on small flaws while missing the larger transformation.
But I don’t think so. In data analysis, getting the numbers wrong is not a small flaw. It is the whole game.