Embrace Variables

A poll is a snapshot in time, but it’s not the full picture

September 4, 2019

What is a political poll? Essentially, it’s a collective answer to a relevant question, weighted by the likelihood it’s true.

As the chill of fall nears and the 2020 campaign season starts to heat up, the topic of polling reemerges from its increasingly brief hibernation within political science lecture halls and FiveThirtyEight features. As part of our series Everyday Explained, wherein ordinary issues receive expert insight in an approachable format, we asked two CU Denver professors to provide a little guidance on the subject of political polling and everything the practice entails.

The start of political polling

In its infancy, American political polling consisted of questionnaires mailed via postcard to readers of popular weekly magazines. In 1936 Literary Digest sent more than ten million surveys to its subscribers and to registered automobile owners. Based on more than two million responses, the magazine predicted the presidency would pass to Republican Alf Landon. If that name sounds vaguely less than presidential, it’s because Literary Digest failed to account for a selection bias inherent in its method of polling: namely, a population sample limited to individuals with sufficient income to afford magazines and/or vehicles at the height of the Great Depression.

A statistician by the name of George Gallup, however, did account for such biases and correctly predicted the results of Franklin Roosevelt’s reelection from a sample size of only fifty thousand respondents. How? Because Gallup took efforts to ensure his polling selections, while fewer, were more randomly distributed and more representative of demographic variability, which therefore yielded a more reliable result.

The importance of being random

The precision of a polling forecast depends on the pollster’s effectiveness at eliminating biases (or identifying and adjusting accordingly for them) when selecting a sample to serve as representatives of a broader population. A sample taken at random is likelier to represent the electorate at large better than a sample constrained by bias. However, the very act of selection might seem at odds with the need for responses sourced at random, but Michael Berry, PhD., associate professor of political science at CU Denver, explains what randomization means in principle:

“Statistically, it means that every element of the population [i.e., each person] has an equal likelihood of being included in the sample.”

In practice, this requires constant calibration of the techniques by which a poll solicits responses from participants, particularly with regard for changing modes of communication. In the age of cell phones, a poll conducted solely via landline would exclude the increasing percentage of the population that uses only their mobile devices. Any such sample would not account for variations in socio-political viewpoints between landline owners and cell phone users. And the same economic circumstances that influence phone plan decisions influence the decisions voters make at the ballot box.

Professional pollsters adjust for these variables, but even the most precise and unbiased forecasts can still miss the mark.

So why do polls still get things wrong?

“A thing with a small probability can happen, and does sometimes,” says Paul Teske, PhD, Dean of the School of Public Affairs. Most opinion polls aim for a confidence level of 95%, a level of near-certainty that nevertheless accepts the possibility the same polling process, if replicated, could produce an alternative result in a few, rare instances.

With any sample, there is a degree of risk that it is a bad sample, one that fails to capture the wider electorate’s true degree of support for a given candidate or cause. Though the reasons for such discrepancies vary from an inclination to share more socially desirable responses (shame) to a reluctance to answer the phone (disinterest), those factors can have a distorting effect.

“Anytime we’re trying to make inferences about a broader population from a subset of the population or a sample, there are opportunities for bias or error,” says Berry, elaborating on the multitude of reasons the outcome of a poll can be wrong. After all, in election forecasting, the extent to which a random sample aligns with the actual electorate is a comparison one can make only after the electorate has shown up to vote. “Polls as predictions of election outcomes … That’s a difficult bridge to build.”

Yet that’s exactly why the electorate and their leaders look to polls, to help them navigate and make sense of the future’s uncertainties.

Forecasting the future

From 1936 to 2008, the Gallup Poll correctly predicted all but one winner of the presidential election. And, despite the flaws of its technique referenced above, Literary Digest accurately predicted the five elections between 1916 and 1932. It seems a political poll is reliable for as long as the social variables its methods account and control for hold constant. A foreseeable constant, then, is the need for ongoing calibration.

Predicting the future has always been a risky business. As a science, polling promises only near-exactness, but it is critical to undertake such analysis whenever a population is faced with difficult questions its people must answer together. At addressing these challenges, statisticians and pollsters are far more skillful than the augurs the public may regard them to be, even if the scientific process can appear equally mystifying.

As we approach a future where overlapping campaign seasons blur into an epoch, it would be reasonable to conclude polls will remain a necessary fact of American political discourse.

(With 95% confidence, anyway.)