Op-Ed: Is Biden a shoo-in? Can Sanders come back? Here’s why predicting political outcomes is so hard

It’s been another tough political season for the pundits. Take Super Tuesday. On the Monday before the vote, a typical headline read something like this one from The Hill: “Sanders Poised for Big Super Tuesday.”

On Wednesday, those headlines were replaced by ones like this from CNBC: “Super Tuesday results: Joe Biden shocks the world.”

Predictions gone wrong are nothing new for the punditocracy, of course.

I’ve just finished writing a book about the ’60s, and came across forecasts like these in my research: “Jack Kennedy won’t get anywhere” (syndicated columnist Drew Pearson in 1959), and “It will be a Johnson-Humphrey ticket again in ’68” (US News & World Report, 1967).

And it wouldn’t be fair if I didn’t mention a headline that topped a Nation article I wrote in June of 2016. It read, “Relax, Trump can’t win.”

Journalists have always been better at reporting what has happened than at predicting what will happen. In recent years, we hoped that Big Data would help us get things right, by enabling predictions based on scientific evidence uncontaminated by passionate convictions and political hopes. Nate Silver, who set out to do this on the website he founded, FiveThirtyEight.com, succeeded gloriously in 2012, getting 49 out of 50 states right. But then of course he (and the other Big Data people) failed miserably four years later.

Data geeks, like the ones at FiveThirtyEight.com, the New York Times’ Upshot and the Washington Post’s Simulator, start by gathering historical data from dozens of polls, using statistical techniques to weight the polls based on their past performance. Adding in other data, they then perform computer simulations of different possible outcomes of elections, often thousands of times, to come up with probability estimates of the most likely winners.

Big Data people complain that the public and the pundits don’t understand such “probabilistic estimates.” Silver points out that in his final analysis of 2016, on the morning of election day, he gave Trump a 29% chance of becoming president.

Baseball fans understand statistics better: They know that a batter who gets a hit 29% of the times he’s at bat (a .290 batting average) is more likely than not to make an out — but he’s also going to get on base nearly a third of the time. That’s not such a bad average in baseball.

But in politics, somehow, when we hear that someone has a better than 70% chance of winning, we think that means victory is a certainty.

Another problem with political predictions is that, no matter how powerful the computers and how smart the analysts, the probabilities they come up with are only as good as the underlying polling on which they are based. And polls have gotten more complicated in recent years.

For one thing, Americans are less willing to participate in phone surveys than ever before. The Pew Research Center reported recently that only 6% of the people they call in their surveys today are willing to participate — part of a steady downward trend from 1997, when the participation rate was 36%.

Many pollsters, including Pew, are moving away from telephones to online surveys.

Silver doesn’t see a problem with the falling response rate for telephone polls. “The polls are all right,” he said.

They’ve been as accurate recently as they were in the past several decades, he notes. Yes, most of the 2016 polls predicted a win for Clinton. But the polls were even further off in 1996 (Clinton-Dole), and 1992 (Clinton-Bush), and much further off in 1980 (Carter-Reagan). Much internet-based polling, Silver says, has been even less accurate.

And even if Big Data were 100% accurate, different analysts would interpret it differently. Here’s an example: Bernie Sanders and many progressive strategists cite data in arguing that beating Trump will require mobilizing the millions of young people and people of color who haven’t voted in the past. The Biden campaign has concluded, after analyzing the same kind of data, that winning depends on appealing to moderates who may have voted for Trump in 2016 and are now ready for change.

So, which of them is right? The 2018 midterm can be interpreted as buttressing either strategy.

Turnout in 2018 was the highest in a midterm since 1914 and up 13 points over the previous midterm, in 2014. And of course the result was that the Democrats recaptured the House with a 40-seat gain and flipped seven Republican-held governorships. So that would appear to support the Bernie strategy of trying to turn out new voters.

But Ruy Teixeira of the Center for American Progress draws on data from Catalist to argue that 89% of the Democrats’ improved performance in 2018 came not from new voters but rather from vote-switchers — 2016 Trump voters who had changed their minds. His conclusion: Persuading Trump voters to switch is a better strategy in 2020 than trying to turn out first-time voters.

Steve Phillips, also at the Center for American Progress, argues that “no further wooing of swing voters” is needed. He has focused on how the United States has changed demographically since Trump’s election in 2016, including looking at how it has grown more diverse.

Those changes, he says, mean that, “if 2020 were an exact replay of 2016 with all racial groups voting in the exact same way in terms of turnout and partisan preference,” Democrats would “win the popular vote by three points, take back Michigan, Pennsylvania, and Wisconsin, and carry the Electoral College 279-259.” In his view, “no further wooing of swing voters needed.”

So we end up where we started.

When it comes to politics, there really is no dispassionate science. The pundits are like all the rest of us: They don’t know what’s going to happen. What makes them different is that they act as if they do.

Jon Wiener is professor of history emeritus at UC Irvine and a contributing editor at The Nation. His forthcoming book, with Mike Davis, is “Set the Night on Fire: L.A. in the Sixties.”