Editor’s note: John Dick is CEO of CivicScience, a Pittsburgh-based research firm. 

A couple of years ago, our office used an imaginary swear jar to collect a fine when anyone uttered the term big data. At the time, most self-respecting statisticians and database engineers would scoff at the term, along with other buzzwords like data scientist.

Eventually we succumbed. Engineers changed their job titles to data scientist – which, by the way, instantly increased their salary requirements. No roles or skills changed in any meaningful way. The hard-core Carnegie Mellon computer science grads around the industry had been working with large data sets and tools like Hadoop for a long time. (Note: Saying big data will still elicit an eye-roll or two around the office.)  

There is no denying that the pace and scale of data being generated today is unprecedented. It’s an exciting and promising development, for the most part, with the potential to fundamentally alter everything from health care and energy to politics … and even marketing research.

Many in the marketing research industry view the proliferation of data and analytics as a scary frontier (at best) or a threat to their job security (at worst). Who needs people to administer 20-person focus groups and 800-person survey samples when consumer information is flooding in by the terabyte while you sleep? Looking for a great answer to that question? Important things like forward-looking intent and in-the-moment perception are best ascertained by asking people things. While consumer data sets are inherently backward-looking, the best predictive analytics will never be as reliable as asking someone, “Are you planning to switch your wireless carrier this month?”

I’m not here to defend the sustaining virtues of analog marketing research techniques. I’ll be happy to do that another day. The truth is that large-scale data and analytics are here to ...