All of this writing and data analysis is a lot of work! So after more than five years of posting, I've finally launched a Patreon to help pay the bills.

5/28/19

Everyone misread the "Sanders voters don't pay attention to politics" poll

Over the past week, multiple pundits - including Nate Silver, Chuck Todd, and Harry Enten, among others - have seized on a new Quinnipiac poll to make some confused points about how much attention Sanders supporters are paying to the Democratic primaries. The kernel of truth to their argument is that Sanders is currently winning the largest share of voters who say that they are paying little-to-no attention to the primaries. But from there, they extrapolate all sorts of conclusions that defy both basic math and common sense.



Massive error margins

To appreciate why this is, let's walk through the numbers. First, Quinnipiac asked respondents how much attention they were paying to the primaries. Their answers look like this:



Then, they asked respondents in each group which presidential candidate they prefer. Here's what the "A little / none" category looks like:



These are the numbers that appear repeatedly in all of these reports: Sanders, for example, has 28% of the "little or no attention" group, its largest share. But remember: this group itself is only 23% of all respondents, which means that the Sanders voters are only 28% of that - in other words, we are talking about 28% x 23% = 6.4% of all voters. Here's how all three of the groups break down:

I have only marked Sanders' percentages here, but it should be easy enough to see the problem. This is a poll with a ± 5.6% margin of error, and all of the variation in Sanders' support is taking place within a range of 2.9%. If just seven Sanders supporters had said they're paying "a lot" of attention to the primaries instead of "a little", the numbers here would be completely reversed. The microtrend that all of these articles are based around is way too small to be statistically significant. 

Thus when Chuck Todd says that "the less you pay attention, the more likely you are to support Sanders," he is going well beyond what the data can actually tell us. And even if this were true, the point is trivial: by more likely, he just means 2.9% more likely.


Apples to oranges

It's not clear to me that any of these pundits understand that the percentages in Quinnipiac's poll represent fractions of each sub-group rather than fractions of the whole. For example, here's Harry Enten:

Nothing about this table makes sense! These numbers do not represent "Bernie Sanders' support by attention paid to campaign" - if that were true, each of the columns would add up to 100%. What they each actually represent is the percentage of each attention-group that supports Sanders. To find the percentages Enten is trying to show here, you would have to stack the blue blocks from the previous chart on top of each other, which would give you all of the Sanders supporters broken down by attention-group:

By looking at the Y axis, you can get a sense of what the Sanders constituency actually looks like: ~20% are paying a lot of attention to the primaries, ~40% are paying some attention, and ~40% are paying little to none. Enten's proportions are completely wrong because he isn't adjusting for the fact that each attention-group is a different size. This also gets him into trouble when he directly compares this years' numbers to 2015, which had a much different attention pattern (only a 26% minority were paying "a lot of attention" in May 2015).


Name recognition, again

Even if we ignored the margin-of-error issue and the apples-to-oranges comparison, there's one last problem that derails this entire analysis: name recognition. The blue chart above represents the sum total of Sanders voters in this entire survey. What if we compared this to bars representing other candidates?

If you're having trouble seeing some of the sub-groups on this graph, there's a reason: just about everyone is getting responses in the single digits. And that explains why Sanders voters account for so many "a little / no attention" responses: Sanders also has more "some" responses than every candidate except for Biden, and places 5th in "a lot" responses. He's pretty popular across the board! As the other candidates start to gain traction, we can only assume there will be dramatic shifts within each of these sub-groups. It would for example take just a dozen "A little / none" respondents moving from Sanders to Warren to put her in first place among "low-attention" constituencies.

Silver, to his credit, also notes the name recognition issue in his discussion of this poll - but curiously, he ignores the possibility that it could play out in Sanders' favor. Instead, he only speculates about whether Sanders' "support has been propped up by name recognition." Similarly, about a week ago, Silver brought up name recognition again - this time, to cast doubt on polling that showed Sanders outperforming nearly every other candidate against Trump.

It's difficult to see errors this serious and selective and not suspect that there's some motivated reasoning at work, but by god I'm trying.