Editor’s note: John Dick is CEO of CivicScience, a Pittsburgh-based research firm. This is an edited version of a post that originally appeared here under the title, “What the hell just happened?”

When I put my nine-year-old daughter on the school bus this morning, I had some pretty deep thoughts about the world I was sending her off to. I also had a random morbid thought – a mental image actually – of the bus running over a bunch of political pollsters who were being thrown under it.

As a company started on the premise that traditional polling was headed down this path, you’d think we might be feeling some kind of affirmation today. I took a lot of flak for keeping our business out of the lucrative political polling game this cycle. I certainly was tempted. Thankfully Ross McGowan, our head data scientist, kept my head on straight. He knew something was off. He knew that our solution wasn’t ready for what looked to be such a complex election.

Can we measure how people feel about fast food restaurants, smart phones or the NFL? Yes. But none of those topics evoke the kind of visceral, emotional underpinnings – at the scale and complexity – of a national election. I always say, “Regardless of the research method you use, if something comes back 51 to 49, don’t bet your business on it.” In politics, we have no choice but to bet the farm on 50.1 to 49.9.

Give the pollsters a break 

But, no, I’m not finding any joy in the ill-fortunes of the pollsters. For one, they weren’t as tragically bad as the prevailing narrative would have you believe. At the time of this writing, it’s looking like Hillary Clinton will end up winning the popular vote by about 1 percent, after the California tally is complete. Pre-election polling averages had Clinton winning by about three points, well within any single poll’s margin of error.

The LA Times conducted the one national poll consistently showing Trump ahead and was seen bragging about it. The problem with that? Their final numbers showed Trump winning the popular vote by three points, which means they were more wrong than the others – just wrong in the right general direction.

The pollsters performed much less impressively at the state level. Polling averages consistently had Clinton ahead by roughly four points in Pennsylvania and Wisconsin and two points in Florida and North Carolina. She lost them all. Still, at least mathematically, it wasn’t abject failure. Most polls were off by three to five points, still a reasonable margin, depending on the sample size. Being 95-97 percent correct gets you an A in most schools.

The problem isn’t that the pollsters erred. It’s that they all (well, almost all) erred in the same direction. They all missed something. There was clearly a widespread underestimation of support for Donald Trump, an overestimation of enthusiasm for Hillary Clinton and a general miscalculation over who would vote and who would stay home. I saw signs of this but just couldn’t completely put my finger on it or quantify it at a level that made me confident.

We will spend the next few weeks digging into our numbers. My preliminary look at things tells me that certain people, namely a lot of blue-collar Democrats and white women were either dishonest with pollsters or avoided them. I wrote over a year ago about the emerging effects of social shaming and retribution on how people share their opinions. It’s a very real and tricky thing.

It also looks like groups of non-frequent voters turned out while many frequent voters did not. But those are over-simplifications.

We must fix polling

I know that panicking about the polling industry may seem a bit melodramatic, given all the other problems people are coming to terms with today. In fact, we may even have a right to be angry at these polling firms (and the media who propagate them) for misleading us with their confidence and scientific conviction – a lot of people were very irresponsible.

The truth is that we cannot have a healthy society without reliable public opinion polling. And, I’m not just saying this because we need to be better at predicting election outcomes. That’s the lowest and least valuable form of polling in my opinion.

Perhaps if we didn’t only ask people how they feel every four years, we could anticipate and address voter discontent before it engulfs us. If we listen to real people every day – not just the media and intellectual elites who dominate Twitter – maybe our leaders could be more informed and attentive. We need to put our brains together to make this happen.

I’m sad that our nation is so divided. I’m sad that so many people are angry and hurt. But I’m also sad that everyone is so surprised. The greatest country in the world should never be this dumbfounded.