Samuel Minter – Page 9 – Election Graphs

Win Chances from Poll Averages

This is the second in a series of blog posts for folks who are into the geeky mathematical details of how Election Graphs state polling averages have compared to the actual election results from 2008, 2012, and 2016. If this isn't you, feel free to skip this series. Or feel free to skim forward and just look at the graphs if you don't want or need my explanations.

Last time we looked at some basic histograms showing the distributions for how far off the actual election results were from the state polling averages, but pointed out at the end that we don't just care about how far off the average is, we care about whether it will actually make a difference to who wins.

For instance if we knew the poll average overestimated Democrats by 5%, this means something very different depending on the value of the poll average:

If the Democrat was ahead by 15% the overestimation wouldn't matter, because it wouldn't be enough to change the outcome. The Democrat would just win by a smaller margin.
If the poll average showed the Democrat ahead by 3% the overestimation would mean the Republican is actually ahead. This is the case where the overestimation would actually make a difference.
If the poll showed the Republicans ahead the overestimation of the Democrats wouldn't change the outcome, it would just mean that the Republicans would win by a bigger margin.

Two sided by individual results

So lets look at this same data we did in the last post, but in a different way to try to take this into account… Instead of bundling up the polling deltas into a histogram, we consider all 163 results, and try to figure out at each possible margin, how many poll results had a big enough error that they could have changed the result.

As an example, if you look at all 163 data points for cases where the Democrats beat the poll average by more than 15%, you only find 2 cases out of the 163. (For the record that would be DC in 2008 and 2016.) That lets you infer that if the Republicans have a 15% lead, you can estimate the Democratic chances of winning given previous polling errors is about 2/163 ≈ 1.23%, which of course means in those cases Republicans have a 98.77% chance of winning.

Here is the chart you get when you repeatedly do this calculation:

As you would expect, as the polls move toward Republican leads, the Democratic chance of winning diminishes, and the Republican chance of winning increases. The break even point is almost exactly at the 0% margin point. (It is actually at an approximately 0.05% Republican lead based on my numbers, but that is close enough to zero for these purposes.)

This is good, because it means even if there are some differences in the shape of the distribution on the two sides, at least the crossover point is basically centered.

Taking the exact same graph, but coloring it differently, we can look at not which party wins based on the polling average at a certain place, but instead at the chances that the polling average is RIGHT or WRONG in terms of picking the winner.

For instance, at the same "Republicans lead by 15% in the polling average" scenario used above, there is a 98.77% chance the polling average has picked the right winner, and a 1.23% change the polling average is picking the wrong side.

Here is that chart:

Looking at it this way, and again looking at 1σ being getting things right about 68.27% of the time, and 2σ being getting things right about 95.45% of the time we can construct the table below. With the asymmetry, it is different for Republicans and Democrats:

	68.27% (1σ) win chance	95.45% (2σ) win chance
Republicans	Margin > 1.76%	Margin > 8.26%
Democrats	Margin > 3.01%	Margin > 12.56%
Average	Margin > 2.38%	Margin > 10.41%

Basically, given the results of the last three election cycles, Democrats need a bit larger lead to have the same level of confidence that they are really ahead.

Now, doing something for Election Graphs based on this asymmetry is tempting as it has now showed up in two different ways of looking at this data, but…

A) It is certainly possible that this asymmetry is something that just happens to show up this way after these three elections, and it would be improper to generalize. 163 data points really is not very much compared to what you would really want, and it is very possible that this pattern in an illusion caused by limited data and/or there are changes over time that will swamp the patterns seen here.

B) A big part of the appeal of Election Graphs has always been having a really simple easy to understand model… including having nice symmetric bounds… "a close race is one where the margin is less than X%"… without that being different depending on which party is ahead… is a big part of that, even if it is a massive oversimplification.

So lets look at this in a one sided way again…

One sided by individual results

Once again we we calculate a "chance of being right" number, as we did earlier, but now just looking at the absolute error. We will assume we only know the magnitude of the current margin and the previous errors, not the directions, and that errors have an equal chance of going in either direction.

Running through an example:

At the 5% mark, there were 107 poll averages out of 163 that were off by less than 5%. These 107 would all give a correct winner no matter which direction the error was in.

The remaining 56 polls had an error of more than 5%, so could have resulted in getting the wrong winner. But since we are assuming a symmetric distribution of the errors now, we would only expect half of these errors to be in the direction that would change the winner. So another 28.

107+28 = 135. So we would expect at the 5% mark the poll averages would be right 135/163 ≈ 82.82% of the time, and wrong 17.18% of the time.

Repeat this over and over again, and you get the following graph:

So if you are looking to be 68.27% (1σ) confident that a state poll average will be indicating the right winner, the leader needs to be ahead by at least 2.49%. To be 95.45% (2σ) confident though, you need an 11.36% margin.

	68.27% (1σ)	95.45% (2σ)
Margin	2.49%	11.36%

These last two ways of looking at things have the 68.27% level considerably narrower than the 5% that is the current first boundary on Election Graphs. If we are thinking of this as the boundary for "really close state that could easily go either way" this seems counter intuitive.

If we kept the same "Weak/Strong/Solid" names for the categorizations, then in the final state of the 2016 poll averages Michigan would get reclassified as "Strong Clinton" instead of "Weak Clinton", while Ohio and Georgia would move from "Weak Trump" to "Strong Trump". Given that Michigan ended up actually won by Trump, this seems like it might not be a good move.

Or maybe it would just be even more important to be clear that "Strong" states are actually states that have a non-trivial chance of flipping. The "Solid" category was intended to be the states that really should not be expected to go the other way. "Strong" was originally intended to indicate a significant lead, but not that a state was completely out of play. "Weak" was supposed to be states that really were so close enough you shouldn't count on the lead being real.

If we changed the categories in a way that moved the first boundary inward, it would be important to make this very clear, perhaps by changing the names.

If, on the other hand, we moved the "close state boundary" outward to the 95.45% (2σ) level, it just seems way too wide. A state where one candidate is ahead by 11.36% just isn't a close state. Maybe it is true that there is a 4.55% chance that the poll average is picking the wrong winner. This seems to imply that. But even so, it seems misleading to say that big a lead is still a close race.

There is something else to think about as well. The analysis above assumes that the difference between the election result and the final poll average is not dependent on the value of the poll average itself.

But is that reasonable?

One might think (for instance) that maybe poll averages in the close states would be more accurate than polls in the states where nobody expects a real contest. The theory here would be that because close states get more frequent polling the average can catch last minute moves better than in less competitive states that are more sparsely polled. (At least this logic might make sense for a poll average like the Election Graphs average that uses the "last X polls", other ways of calculating a poll average may differ.)

Or maybe there is some other factor at play that makes it so we can say more about how far off a poll may be if we take into account what that poll average is in the first place.

And that's what we will look into in the next post…

You can find all the posts in this series here:

Polling Averages vs Reality

2018 is over. Multiple candidates have announced they are at least investigating running for President in 2020, and a few are even past that stage. But before Election Graphs starts posting new graphs and charts for 2020, one more look back at the past.

This will be the first in a series of blog posts for folks who are into the geeky mathematical details of how Election Graphs state polling averages have compared to the actual election results from 2008, 2012, and 2016. If this isn't you, feel free to skip this series. Or feel free to skim forward and just look at the graphs if you don't want or need my explanations.

For those of you who just want to know about 2020… Keep checking in… actual new graphs and charts and analysis for 2020 will be here before too much longer! How much longer? I'm not sure. But after this series of posts is done, getting up the basic framework of the 2020 site is my next priority!

Now, for the small group of you who may be left… any thoughts, advice, or checks on my math are welcome. While this is all interesting on its own, some of this is just me thinking aloud as I figure out what (if anything) I am going to do differently for 2020. Please email me at feedback@electiongraphs.com for longer discussions, or just leave comments here. Raw data, Excel spreadsheets, etc., are available on request to anybody who wants them, although fair warning, they aren't all cleaned up and annotated to be scrutable to anyone other than me without some explanation or effort.

Anyway, one of the key elements I called out in my 2016 Port Mortem was the need to "trust the uncertainty" and look at the range of possibilities, not just the prediction's center line. Part of this is just being vigilant in avoiding the temptation to reduce things to a single point estimate rather than a range of possibilities. Another part is repeating over and over again that a 14% chance of something happening isn't the same as 0%. Although pretty much everybody doing poll analysis did explain these things to some extent in 2016, in retrospect it is clear that there was still too much emphasis on that centerline by most people, including by me.

But another important element is defining what that uncertainty is. Ever since I started doing presidential election tracking in 2008, I have used "margin less than 5%" to define states I was going to categorize as close enough you should take seriously the possibility they could go either way. I also used 5% as the limit of "too close to call" for the tipping point metric. I had a second boundary at 10% on the state polls to mark off the outer boundaries that you could even imagine being in play if a candidate made a huge surge.

Those numbers were just arbitrary round numbers though. With three election cycles of data behind me now though, it is time to do some actual analysis of the real life differences between the final polling averages and the actual election results, to get a better idea of what kind of differences are reasonable to expect.

Over the next few blog posts I will look at this in several different ways, then decide if anything about Election Graphs should change for 2020.

One sided histogram

First, let's just look at a simple histogram showing how far off the poll averages were from the actual margins.

For each of the three election cycles, we have 50 states and DC. For 2012 and 2016, we also have the five congressional districts in Maine and Nebraska. (I didn't track those separately in 2008 unfortunately.) For each one of those results, I look at the unsigned delta between the final poll average margin and the margin in the actual election results, and show a histogram for each of the three election cycles, and a combined line using all 163 data points.

Somewhat improperly using the "Nσ" notion… with 1σ being about 68.27% of the time, and 2σ being about 95.45% of the time, we see that 68.27% (1σ) of the poll averages were within 5.49% of the actual results. But to get to 95.45% (2σ) you have to move out to 13.39%.

	68.27% (1σ)	95.45% (2σ)
Margin	5.49%	13.39%

[Note for sticklers #1: There is no way this kind of analysis is significant to 0.01%, so showing two digits after the decimal point is false precision and the value depends on my choices for how to interpolate… among a variety of other things. But I've standardized on two digits after the decimal for everything in this series of posts anyway because… well, just because. Feel free to round to the nearest 0.1% or even 1% if you prefer.]

[Note for sticklers #2: Each election cycle I made some modifications to the fiddly details of how I calculated the averages, including things like what I did if there were less than 5 polls, how I dealt with polls that included more than one version of the result (for instance registered vs likely voters or with and without third party candidates), and if I used the end or middle of the field dates as the date used to determine poll recency. These differences may technically make it improper to do calculations that combine data from these three cycles without recalculating everything based on the same rules. I contend that the differences in my methodology over the three cycles were minor enough that it wouldn't substantially change this analysis, but given the amount of work that would be involved, I have NOT spent the time to convert 2008 and 2012 to match my 2016 methodology in order to confirm this.]

It is very tempting in the context of Election Graphs to just move my boundaries between "Weak" and "Strong" states from 5% to 5.49%, and the boundary between "Strong" and "Solid" from 10% to 13.39%. Both of those numbers are kind of close to where the old boundaries are, just expanded a bit to show a bit more uncertainty than before, which seems intuitively right after the 2016 election cycle.

Two sided histogram

But wait, why just look at the magnitude of the errors? Isn't the direction of the errors important too? Are the polls systematically favoring one side or the other? Very possible. Time to do that histogram again, but taking into account which direction the polls were off:

The pattern isn't symmetrical, although it is certainly possible (perhaps even likely) that if I had data for a few more election cycles it would become more so. At the moment though, while the peak looks to be very slightly on the side of Democrats doing better than the poll average (in other words the polls showed Republicans doing better than they actually were), when you average out all the polls, the bias is actually that the Republicans beat the poll averages by 0.69%. (In other words, the poll averages showed Democrats doing slightly better than they actually did.)

The asymmetry is notable here. When the poll averages overestimate the Republicans, most of the time the error is 6% or less, but when the poll average overestimates the Democrats it is often by quite a bit more.

I didn't put it on the plot because it was already pretty busy, but you can also use this to get the ranges for the central 1σ and 2σ:

	Middle 68.27% (1σ)	Middle 95.45% (2σ)
Range	D+4.61% to R+7.45%	D+12.21% to R+13.32%
Avg Limit	6.03%	12.77%

Maybe these numbers could be used to define category boundaries? You would have to either explicitly have different category boundaries for the two parties or use the averages of the R and D boundaries to make it symmetric. The median is also not quite at the center… it is at an 0.01% Democratic lead. But that is probably small enough to just count as zero.

But looking at how many polls are a certain amount off favoring one party or another doesn't really hit exactly what we want.

See, for what we care about on a site like Election Graphs, if we have a poll average at a certain point, we don't actually care that much if the actual result is that the leading candidate wins by an even bigger margin. We only really care if the polls are wrong in the direction that leads the opposite candidate to win.

I'll look into that in the next post…

You can find all the posts in this series here:

Clearing things out for 2020

It isn't quite time to get everything launched properly for the 2020 version of Election Graphs… but 2020 is getting closer and closer. Here at Election Graphs HQ, I've been occupying myself with other things for a while, like launching the Wiki of the Day podcast a little over a year ago, and continuing with my weekly current events podcast Curmudgeon's Corner, but it is well past time to start getting things ready for 2020.

Behind the scenes I've done some additional analysis of the 2008 through 2016 election results that will inform what I do for 2016. I'll post a bit about that soon.

But first just to note the immediate changes to the site itself. The 2016 charts and graphs have been moved off the front page of Election Graphs in favor of a blog format where the analysis posts and such will live. In the last cycle, these had been posted on my personal blog, but now they will live natively on Election Graphs proper.

I haven't built out the new detail pages for 2020 yet. Those will be coming over the next few months. My target is to launch fully right after the 2018 general elections are over.

The archival material is still there of course, just tucked away a bit. For those looking for that:

Thanks again to everybody who followed our analysis in 2016. I'm looking forward to doing it all again for 2020.

Stay tuned.