Analysis: Can political polls still be trusted?

WASHINGTON -- Over the last couple of decades, the percentage of people willing to answer a telephone survey has plummeted -- both in the United States and abroad. The drop has led many to wonder whether polls can still reliably tell us much about public opinion.

A new study from the Pew Research Center, one of the country’s best-known polling operations, provides some reassurance on that score -- but also some questions. For anyone who spends time looking at polls in this election year, the results are important to know.

Polls -- at least the good ones -- continue to do a pretty accurate job of predicting election results, as the Pew study points out. In 2010, for example, the average poll done within nine days of the November election hit candidate results within 2.4 percentage points.

But the percentage of people who answer the phone and complete a survey -- what pollsters call the “response rate” -- has continued to drop. Fewer people will answer the phone, and, once they do, fewer will agree to do a survey. In 1997, Pew calculated the response rate for its surveys as 36%. Today, it’s 9%.

Part of the drop involves cellphones. People are less likely to respond to a poll on a cellphone than on a land line, but because one in three U.S. households can only be reached by cellphone, most large polling firms now include them in their samples.

The drop in response rates makes polls more expensive to run -- a pollster has to dial a lot more numbers to get a 1,000 completed interviews. Moreover, if nine in 10 households won’t answer a poll, can survey results still be considered valid? The question depends on how much the small group that will answer differs from the large majority that won’t. On that point, Pew’s results offer some reassurance.

Pew did three things to try to figure out how similar poll respondents were to those who don’t respond. First, it did what it called a “high-effort” survey -- designed to get a higher response rate -- and compared the results to a standard poll. For the high-effort poll, Pew made more than 25 calls to each phone number. Instead of doing the polling over five days, it took 2 1/2 months. It sent letters to households that hadn’t responded, explaining the project. It used particularly experienced interviewers. And it offered rewards of up to $20 for completing the poll. All that effort brought the response rate for the high-effort survey up to 22% -- an improvement, although still less than the rate for a standard poll 15 years ago.

Next, Pew compared survey results with information from government databases on questions including how many people are registered to vote, their age distribution, marital status, number of children and how long they had lived at their current address. Finally, Pew compared its survey respondents with large, commercial databases.

On most measures, the study found, the people who responded to surveys were the same as everyone else. The percentage of Democrats, Republicans and independents did not change substantially; neither did the ideological distribution. Nor did the percentage of registered voters, homeowners versus renters, high-income versus low-income people, Social Security recipients or the share who are hunters.

The one big difference was that people who answer polls are also more likely to volunteer in civic organizations or talk to their neighbors. In other words, the kinds of people who agree to sit for a poll seem to be more civicly engaged than the rest of the population. More than half the people in the Pew survey, for example, said they had volunteered for an organization in the past year. A nearly identical census question indicates that 27% of people have done so. Similarly, 29% in the Pew survey said they had contacted a government official in the past year, as compared with 10% in the census’ Current Population Survey. Not surprisingly, people who answer surveys also seem to be more interested in politics than people who don’t.

[email protected]

Original source: Analysis: Can political polls still be trusted?