A predictable result
Editor's note: Brooke Reavey is an assistant professor of marketing at Dominican University, River Forest, Ill. Sheri L. Lambert is the academic director of the MS-Market Research and Insights Program and an assistant professor of marketing at the Fox School of Business at Temple University, Philadelphia.
After yet another shaky performance in the 2020 elections, pollsters are being roundly derided for “getting it wrong” and the death knell is being sounded for the industry. But, as Nobel laureate Niels Bohr once said, “It is difficult to make predictions, especially about the future,” and this is even more true when those predictions are about human behavior.
Voters are, after all, human. Connecting attitudes – when people say they like or dislike something – to behavior – actually purchasing the product – is a problem that marketing researchers have spent decades contemplating, researching and adjusting statistical models to improve accuracy and predictability. In the business world, the market is more understanding when analysts (pollsters) over- or undershoot forecasts based on consumer attitudes. At worst, sales remain stagnant. At best, sales increase. Either way, botching a sales forecast does not generally trigger widespread criticism quite like what occurs with political polls. However, pollsters who use the same statistical techniques as marketing researchers (i.e., surveys based on attitudes) appear to be held to a different set of standards.
Pollsters need to look at a multitude of facets of what the voters are saying and how they are saying it. Pollsters need to update how they conduct and collect their research. To that end, what is being said on social media platforms cannot be discounted.
Prior to election night last November, the overriding expectation was for a “blue wave” that would see Joe Biden sweep past Donald Trump with a comprehensive victory. However, the results were much closer, with Trump’s support overperforming expectations. An anticipated Democratic clean sweep of the main legislative and executive branches of government did not materialize. Overall, the polls were on point in evaluating Biden’s support but they underestimated Trump’s voters.
This isn’t the first time we have seen this. Past polling results show several high-profile polling misses, including Trump’s 2016 victory following earlier predictions of a Clinton landslide.
Every election cycle, pollsters are tasked with predicting the direction of the election based on survey research. And survey research, as all market researchers know, is only as good as its sample.
It is true that political polls are more difficult than a typical consumer market research study, because the linkage between attitude and behavior is more distant than in most consumer goods. Specifically, the attitude link is part of it but then there is the all-important model of who is likely to vote. This exists as well in consumer research but what we are likely seeing is a degree to which the disconnect between attitude and likelihood of action is determinative of the answer, which is why the weighting is so important. In consumer research, this secondary linkage is well developed. Also, other than big-ticket items like cars, for CPG, the time between attitude measurement and purchase is frequently quite short.
Garbage in, garbage out
As the old saying goes: Garbage in, garbage out. Market researchers commonly refer to this phrase by explaining that if the sample of people being interviewed is mediocre, so too will be the results. Pollsters know this adage well and take strides to prevent their polls from ending up as rubbish. The methodology of the poll is typically cited by the poll and poll aggregators like Nate Silver’s FiveThirtyEight take this into consideration when crunching their numbers and offering their forecasts. Additionally, after 2016’s presidential polling upset, pollsters adjusted their methodology for the 2020 presidential cycle. According to Geoff Skelley and Nathaniel Rakich of FiveThirtyEight, pollsters adjusted their methodologies for 2020 in the following ways:
Weighting results differently based on education. Many pollsters included more from the “high school or less” group and less from the “college or more” group. Ipsos and Pew Research Center even went so far as to weight the data within racial groups, a technique that is not typically done.
Paying attention to where the respondents live: rural, suburban and city. As was discovered in the 2016 polls, not enough time and effort was taken to ensure that rural voters were included in polls.
Adjusting methods in which they reach respondents. Polls have always deferred to either internet panels or via telephone by using random-digit dialing. But both methods are fraught with problems regarding self-selection and non-response bias. Moreover, pollsters speculate that many respondents may have hidden their support for Trump from the interviewers due to social desirability bias. In 2020, pollsters expanded their internet panels to include more people and they also adjusted their phone interview methods switching between random-digit dialing (typically used for landlines) and using a person to manually dial the phone number (a process required for calling cell phones).
As survey researchers know all too well, the incidence rate of the study – the prevalence of the topic being studied in society – drives up the cost of the study. The lower the incidence, the more expensive the study and vice versa. Pollsters run into this problem as well.
The difference between polling for the presidential election versus a House seat is like night and day. Regarding complexity, a Senate race often falls right in the middle. According to Pew Research, as of election day, there were 245.5 million Americans older than 18 and 157.6 million registered voters. The incidence rate of finding eligible voters for the presidential election is much higher than finding eligible voters for a House of Representatives election because there are 435 congressional districts with House elections versus one country for a presidential election.
When the sample is further stratified by race, gender, education level, residence and likelihood to vote (based on previous voter behavior), the sample size needed for a statistically valid research study requires many thousands more respondents than are typically used in a political poll. The more respondents there need to be, the more costly the survey is, regardless of the method, to reach the respondents. Therein lies the problem. Media outlets and political campaigns have limited resources to spend on polling. Historically, pollsters have chosen to keep the sample size “reasonable” by conducting more, smaller polls over time, each with a moderate error margin as a way to monitor voter sentiment. If this trade-off is poorly executed, there are consequences: an erosion of public trust in the polls, lower donations that might help a candidate win a tight race and incorrect or misleading opinions regarding public policy questions that frequently accompany the poll (i.e., sentiment on a mask mandate, the importance of the COVID relief bill, etc.).
Key methodology suggestions
So, what can be done differently? Marketing scholars have worked alongside firms for decades to help improve the accuracy of models that convert purchase intent (attitude) to a purchase (behavior) and in this case from intending to vote to actually vote. Some of the key methodology suggestions regarding the future of polling are the following:
Machine learning can help with sampling. Using exogenous data points at the local level (i.e., voter turnout, previous response to survey) can help marketing researchers home in on the areas that require more responses.
Demographic “buckets” are misleading because they ignore intersectional factors. Demographic categories are often treated as monoliths and journalists and campaigns frequently speak about groups of voters as such (the “Black vote,” “the rural vote”). Digital marketing researchers learned long ago that segmentation works best at the individual level (micro-segmentation). Pollsters lag in this understanding. There is quite a bit of variance within each demographic bucket (i.e., race, education level, etc.). It’s myopic to assume that all white people will vote the same way (which seems obvious) but then why do pollsters lump all Latinos into a single demographic bucket, as if Mexicans, Puerto Ricans and Central Americans were monolithic? Sampling methodologists often forget the importance of understanding how social aspects intersect. Knowing what we know now, what candidate would a highly educated female Hispanic who is a practicing Catholic have voted for? That was a trick question. Would it depend on whether we were told that the person in question was of Mexican heritage, living in a suburb of Dallas vs. of Puerto Rican heritage living in Brooklyn? By viewing the tabs that are often broken out by individual demographic categories, the marketing researcher does not get the whole picture of the voter. In “demo” buckets, we don’t consider all of these factors. The most obvious example of this in 2020 was that the polls missed the tens of thousands of Cubans who appeared to vote for Trump mainly because they were voting against Biden.
More time and money are needed to increase the sample size, thus increasing accuracy. News agencies and campaigns need to spend more money on polling and less money on expert commentators and/or ads. Whoa – we said it. There is a reason why we teach confidence intervals and error margins. It’s time to start paying attention to them if we want more accurate polls.
Machine learning can assist in election predictions. It’s early days but marketing research has several AI startups that combine attitude and behavior data to create a comprehensive picture of consumers. By combining polls (attitude) with voting history (behavior) as well as including data on local consumer touchpoints (local employment levels, number of polling locations per thousand voters, online consumer sentiment using geolocation from social media, etc.), AI can help predict the directionality of the vote (i.e., Democrat vs. Republican). This does border on creepy but these inputs are helpful in determining the reliability of the data.
Election forecasting is actually a two-model process and it isn’t completely clear at which point in the process the model accuracy needs improvement. First, polls need an accurate understanding of voter sentiment from a representative sample of voters. Second, there needs to be a model of turnout, which is who is likely to vote. For the polling process, we can assume two trade-offs. First, the more expensive the poll, the more accurate it will be (larger sample, more representative). Second, the more recent the poll, the more accurate it will be. Aggregators like FiveThirtyEight recognize this. One question that should be asked is whether the emphasis on recency is worth it. Perhaps it is better to conduct fewer, more expensive polls and to assume that sentiment is less labile than commonly thought.
Integrating sources
So, how should the polling industry react to the recent U.S. presidential results? The industry should better predict the outcome of the election by integrating other sources of data including social media, past behaviors, election issues, voting turnout, state results, money raised, advertising budget, media trends as well as economic indicators. Or, at the very least, start taking advice from the marketing research personnel who work with brands daily to translate attitude to behavior.