Transcript
Nathaniel Rakich: The death of polling has been greatly exaggerated. After the 2016 and 2020 elections, when the polls had a rough year, a lot of people wanted to leave them for dead. But in 2022, polls had one of their most accurate election cycles in recent history.
But not all polls are created equal. Some are OK to consume — and some are not so good for us. To tell which are the apples of the polling world and which are the $1 pizza slices, FiveThirtyEight has come up with ratings for every pollster that’s out there. But how do we calculate them, and how do they work? What’s the deal with FiveThirtyEight’s pollster ratings?
Remember back in high school when teachers gave you a letter grade like A, B-, C+, etc.? Of course, I was a straight-A student, but those other letters do exist. And we use all of them to describe pollsters. For example, four pollsters have A+ pollster ratings, meaning we think they are the absolute best in the business. Meanwhile, 11 pollsters get a big fat F — meaning we don’t trust them as far as we can throw them. In fact, we only give out F pollster ratings if we believe that a pollster is outright faking its data or engaging in other unethical behavior.
Pollster ratings are mostly based on one simple factor: how accurate the pollster has been in past elections. And when we say “accurate,” we mean something specific. We don’t care whether a pollster picked the correct winner of the election. Instead, what matters is how close the pollster got to the election’s final margin.
Let’s say we have an election with two polls: one showing the Democrat winning by 10 percentage points, and one showing the Republican winning by 1 point. Now let’s say the Democrat ends up winning the election by 1 point. Which poll was more accurate? Not the one that picked the Democrat to win by 10; it was 9 points off the final margin. Instead, the more reliable pollster is the one that missed the final result by only 2 points, even though it “called” the winner wrong.
FiveThirtyEight’s pollster ratings also take pollsters’ transparency into account — specifically, whether they share their data with two professional polling organizations. These pollsters tend to have better methodologies, so we give them a little bit of extra credit in the ratings.
Finally, there’s been a lot of talk recently about whether the polls are biased toward Democrats or Republicans. So that’s something we calculate too: whether a pollster has historically published numbers that were too good for one party or the other. For example, SurveyMonkey is a pollster that has historically overestimated Democrats by almost 5 points, while the Trafalgar Group’s surveys are, on average, more than 2 points too good for Republicans.
So no, polls aren’t perfect — and some are less perfect than others. But overall, they’re still the best tool we have for measuring public opinion and predicting elections. With FiveThirtyEight’s pollster ratings in hand, you’ll be better equipped than ever to consume polls responsibly.
