We predict that one of three outcomes will happen

Nate Silver is arguably the world’s foremost psephologist. This sounds like someone who foretells a person’s future by feeling the bumps on the head, but it is not—that’s a phrenologist. Psephology comes from the Greek psephos meaning “pebble,” which the Greeks used as ballots. It is the branch of political science that deals with the study and scientific analysis of elections.

Today Ken and I want to make a prediction about the election. We predict Nate Silver will be 0.95-correct with probability 0.95.

Silver is a statistician, a sabermetrician, a psephologist, and much more. The term “sabermetrician” comes from the acronym SABR—the Society for American Baseball Research. He devised a system called PECOTA for forecasting the development of skills in aspiring young baseball players, which supplements traditional methods of “scouting” them. He ventured into political science with the blog FiveThirtyEight, now hosted by the New York Times. Its name stands for the 538 electoral votes at stake in tomorrow’s contest between President Barack Obama and challenger Mitt Romney.

Silver has created quite a stir among those who make election predictions, because:

1. He has gone out on a limb giving Obama a 92% likelihood of winning what many news sites and pundits are calling a “tossup” election, when most national polls show the candidates within 1% of each other. (This jumped up at 10:05pm ET tonight—Monday night—from 86%.)

2. He approaches the task very differently from other experts, relying on poll aggregation and other quantitative modeling.

3. His work identifies and exposes systematic bias in polling firms.

4. He himself is known to have supported Obama in 2008.

5. He puts a huge amount of effort into his work, and has a track record of being very good at it.

Still, many with traditional ways of scouting the political landscape think Silver is reading head bumps or has baseball bats in his belfry. That the tug-of-war over perception of each candidate’s chances can itself move the outcome intensifies the passion. The outcome will be seen as much as a referendum on Silver’s kind of methods. Since both of us have been involved in court cases using evidentiary statistics, we both feel some stake in this. But first we have some caveats.

## The Three Outcomes

Obama and Romney are about to find out who will be sitting in the Oval Office for the next four years. Or maybe not. We believe that there are three possible outcomes:

• Obama wins.

• Romney wins.

• None of the above.

The last seems quite possible, given how close many think the election is and given the number of lawyers that are available to fight on both sides. Already today there is word of a lawsuit over provisional ballots in Ohio. The recent and continuing disaster from the huge storm Sandy also gives pause for thought—indeed Andrew Appel (whom we know respectively as a Princeton colleague and college classmate) pointed out a legal disconnect in New Jersey.

What do you think? Here is a poll that lets you voice your opinion.

Ken and I have different opinions on the outcome, as we’ll explain, but both within the margin of agreeing with Silver.

## Dick: A Previous Election

I, Dick, remember the election of Ronald Reagan over the incumbent President Jimmy Carter. The electoral college vote was a landslide, Reagan got 489 votes and Carter only 49. Here is the map of the states:

I recall this so vividly because Bob Sedgewick and I planned a special election night. We had a paper that we needed to finish and the predictions before the election were that it was close, and many thought that we might not know who won until late that Tuesday evening. So Bob and I went off to a hotel, got two rooms, and planned to work all night on our paper while listening to the election news. We checked into the hotel, and went down to get dinner.

When we returned to the room around 8 o’clock we turned on the TV and started to work on the paper. Then at 8:15 pm EST, our time, NBC News surprised everyone with the projection that Reagan was the winner. This was the first time that exit polling was used to make such a projection.

NBC News projected Reagan as the winner at 5:15 PST, well before voting was finished in the West, based on exit polls. I still recall sitting there dumbfounded. There went our great plan to work all night, while the news slowly came in about the election. We soon gave up working, and retired for the night. The paper got done eventually, but not that night.

I don’t think tomorrow will be that quick, but I do think by 10pm it will be clear that Obama is winning even without knowing results in Colorado or Nevada, and the formal call will come by 11pm with the West Coast results as in 2008.

## Ken: Several Recent Elections

Since 1972 I’ve followed the dancing numbers without a thought of getting work done, except for 1984 when I was five hours ahead in Oxford and knew my near-namesake was a lock. In 2000 and 2004 they were very late nights indeed.

Mainly because 2004 came out redder than the exit polls had indicated, 2008 held suspense even though Obama ultimately won the popular vote by 52.9% to 45.7%, and every state went as Silver had projected except Indiana. This time no one projects a margin wider than Silver’s current 50.9% Obama to 48.2% Romney, and today’s final CNN poll has 49% each. Therein lies my first caveat—I expect a long night.

Ironically the expansion of early voting has the effect that more ballots will be counted later. According to a note that was at the bottom of George Mason University professor Michael McDonald’s “Early Voting Statistics” page last Thursday (since changed), over 75% of Ohio’s early votes were being sent by U.S. mail, and they were allowed to be postmark-cancelled as late as today, with counting by the same Nov. 17 date as absentee ballots. Voter-ID and other laws are creating more cases where those appearing on Election Day will have to cast provisional ballots, which are counted only upon presentation of ID or resolution of possible conflicts (such as previously having requested an absentee ballot).

Given that the early vote has been heavily Democratic, this could all narrow the margin as reported on Election Night itself. Early voting and provisional-ballot filtering also cause new systematic polling hazards, such as noted Saturday by McDonald here. That these factors have considerable magnitude was underscored by Pennsylvania State House GOP Leader Michael Turzai’s comment that the state’s new voter-ID law (which was stayed past the election by court order) would “allow Governor Romney to win the state.”

## The Silver Tossup Line

My second and main caveat, however, comes from Silver’s numbers themselves, specifically his own projected error bars—and precisely because we mostly agree with them. Our saying above that we are 95% confident that the result will hit Silver’s 95% confidence interval is not vacuous or snarky—it means we think he’s found the right center and variance. If there were 1000 independent elections, we’d be happy to bet the outcome would lie within his range about 950 times. The problem is:

There’s only one election.

Well there are 53 state elections plus DC for President, and many Senate and House races that Silver has also forecast, but they are not independent. For something that has become a controversy in itself, and is being given outsized importance between quantitative and qualitative views of life, there is only one trial. And this lends weight to the corollary:

There is a 50% chance that the result will be further from Silver’s projection than the bounds of his 50% confidence interval, with a 25% chance of overestimating Obama by that much or more.

Are these self-declared bounds wide enough to be noticed if this happens? Of course they are narrower than the margin of any single poll, typically 3–4 percentage points, meaning 6–8% wide. That’s the point of poll aggregation: by combining results he narrows the confidence window. Exactly this was needed to declare the Higgs Boson discovered last July—the results and error bars of two independent experiments were combined.

Mousing over the election graphic on Silver’s site shows error bars ${2\sigma = \pm 46}$ in electoral votes on Obama’s projected total of 315. Since 50% of the area is within 0.674 sigmas, this gives a median error of 16 electoral votes. On Halloween, Silver had Obama at ${300 \pm 58}$, implying a median error just under 20. In the popular vote, Silver currently shows the error as ${\pm 1.9\%}$, which implies an “expected miss” of 0.7%.

The 10:05pm just-posted numbers narrow his stated confidence quite a bit—what chutzpah!—but still show an expected miss by about the size of Ohio (18 electoral votes) and upwards of 1% on the popular vote. Such a deviation would still equate the election to a nail-biter.

## Lucky Eclipse?

Whether an Obama total around 290 or less would be viewed as “half miss” by Silver is arguable. When I conceived this post late last week the bars were wider. I’ll hold to my view that the systematic uncertainties are “objectively” greater than his model accounts for, analogous to what I’ve said here about the projected error bars in my own chess research needing expansion as shown by empirical testing. Of course my application allows empirical testing where Silver’s doesn’t: there is only one election.

Whatever the underlying reality—and we’ll find out soon—the stakes for popular perception of statistical science on the whole are certainly high, and higher with tonight’s movement. I am reminded of the effect of the 1919 observation of light bending during an eclipse of the Sun for testing general relativity. The closeness to his prediction made Albert Einstein famous in the public eye overnight. But as implied here, the inherent statistical uncertainty of the experiment (partly on hindsight) was so great that the closeness was fortuitous. The recorded bending could easily have been only half what was predicted, and perhaps indiscernible. Well for my own professional purposes I’d like to see Silver’s number hit bulls-eye, but I won’t bet on it.

## Open Problems

What do you think?

We urge all of our American readers to vote as early and as often as the law allows.

## Update 12 noon ET 11/6

FiveThirtyEight currently shows Obama at 313 +- 48 and 50.8%–48.3% in the popular vote. The +- 48 still rounds to a median deviation of +- 16 electoral votes. The “Election Eve Final” post by Professor Sam Wang of the Princeton Election Consortium opens with this statement:

If state polls perform as well as they did in 2004 and 2008, most aggregators should get within +/-15 electoral votes and 48/51 races correct.

He pegs Obama’s number at 332, which is 16–20 on the other side of Silver’s recent numbers. It should be noted, however, that the 29 electoral votes of Florida—seen now as a real tossup—make the difference highly “chunky”. Taking away Florida and Colorado (with North Carolina and Arizona also for Romney) brings that down to 294, which is about the same deviation downward from Silver’s 313. I’ll take a stand and say that as of midnight ET, Obama’s network-called EV total will be under 300. That doesn’t necessarily conflict with Dick’s prediction.

Wang will have this to “reduce uncertainty” and “squeeze more out of the data”. The New York Times observes that out of the 512 subsets from 9 battleground states, 431 give Obama 270 EV or more.

### Update 6pm ET 11/6

Sam Wang flipped Florida back to Romney, so that PEC, conservative prognosticator Scott Elliott, and UIUC CS Election Analytics (noted by a commenter here) all show Obama at 303 or 304 EV. At FiveThirtyEight, 10:10am ET is still the latest update. In Scott Aaronson’s election item, I commented that “538” is also the number of voters Al Gore needed in Florida. InTrade currently has Obama at 70%, which has basically been steady all day and yesterday. That was Silver’s odds on Oct. 22, when he was projecting 291 EV for Obama. Thus the difference between the projections in the range 303–332 (give-or-take Florida) and ones around 290 seems to be basically one between the poll-aggregators and InTrade.

Update, midnight ET: Given that (1) it was called for Obama shortly after 11pm, (2) only 282 (CNN) or 291 (others) electoral votes were called by midnight, while (3) VA and FL are also looking strong for Obama, it looks like all three of Dick, me, and Silver will be right. (Silver showed FL pale-blue at the last minute while I was finishing the original post.)

Update Nov. 7: Looks like Silver hit the bull’s-eye not just on every state but also on the popular vote. It will likely be somewhere in the narrow range between his 2.5% difference and Sam Wang’s 2.2%; I had “under 1.8%” and hinted “under 1.5%”, so I was wrong there.

[several format and word fixes, sourced photos; the reference to “53 state elections” is not a typo]

November 6, 2012 8:27 am

I don’t think you can predict elections from polls. Truman-Dewey is a classic example. I know polls are better today than back then, but people will say things differently than what they will vote.

Sports is the best example of how one can use statistics to predict outcomes. I’ve done this with the NCAA brackets.

• November 6, 2012 4:29 pm

Truman-Dewey is *not* an example of that. Newspapers stopped polling weeks before the election just as Truman was surging. If they had been polling until the very end they would at the very least held back the headlines.

• July 28, 2014 12:08 pm

You’re on top of the game. Thanks for shnrgai.

November 8, 2012 10:07 am

I am surprised at how accurate Nathan Silver’s predictions were. But this is just one election.

November 14, 2012 12:33 pm

He was also accurate during the last presidential election

November 6, 2012 10:21 am

Another good website, comparable to 538, is Princeton Election Consortium http://election.princeton.edu/ . Nate Silver and Sam Wang were both on NPR’s Science Friday a couple weeks ago. Interesting discussion: http://www.sciencefriday.com/segment/10/19/2012/making-sense-of-presidential-polls.html .

I am looking forward to the 2016 race! Any ideas about nominees?

• November 6, 2012 11:02 am

Indeed! As I e-mailed to Dick an hour ago, I am about to post an update including PEC’s forecast. A discussion of 2016 nominees (mostly on the GOP side) is in Bill Gasarch’s post here.

3. November 6, 2012 3:13 pm

Among plausible outcomes, this one with 7 connected components for Obama to just 1 for Romney (excepting Alaska and Hawaii) has the highest Betti numbers.

November 6, 2012 4:28 pm

Another good website with solid methodology (Bayesian Statistics + Operations Research)
I tracked them during 2008 elections and following them for 2012 as well.

electionanalytics.cs.illinois.edu

5. November 7, 2012 1:19 pm

Why did you only urge your American readers to vote as early and as often as the law allows?

• November 7, 2012 4:33 pm

Not sure whether your “only” means “only American” or “only as the law allows”. Anyway, it’s a cultural reference to this. Adding “as the law allows” made it serious, else it would have had a smiley :-). Our blog style shies away from smileys and exclamation points, except when quoting people.

• November 8, 2012 5:16 am

Thanks – I meant “as the law allows”. I am no lawyer but I think I voted as often as the law allows …

November 7, 2012 3:44 pm

Dick Morris predicted a Romney Landslide.
So did some other conservatives.
Jim Cramer predicted an Obama Landslide.
I doubt any of these people will be held accountable.

I am reminded of Moneyball (the book and movie- true story- about using
statistics intelligently in baseball). There is an old guard that still goes on
intuition and feelings, and a new guard that uses sophisticated mathematical
tools. How long does it take for the old guard to realize they are just plain wrong?

The conservative websites that tried to disuupte Nate Silver did not point to
other analysis or stats that they did. They pointed to the fact that he is
an Obama Supporter.

Will they learn from this? I doubt it. Will Cramer? I doubt it.

• November 7, 2012 4:50 pm

This aligns with my thoughts exactly while revising Dick’s original intro. An excellent article following this aspect up is here. And the Romney campaign’s get-out-the-vote and polling operation called Project ORCA was apparently biased enough to account both for their misleading internals and their delay conceding, according to this.

• November 8, 2012 11:31 pm

As Stephen Colbert said at the White House Correspondents’ Association Dinner (2006) — “Reality has a well-known liberal bias”

November 8, 2012 12:12 pm

Although Obama’s reelection looked like a foregone conclusion to the rest of the world, I wonder how many new readers you’ve gained by using the title “the election outcome” two days before D-Day. That was clever.

November 8, 2012 1:43 pm

I’m wonder what Nate predicted before Sandy showed up ?

• November 8, 2012 2:50 pm

Serge, actually it was at 11:59pm the night before Election Day—I rushed a bit to get a Nov 5 day-stamp. The title was thematic for the post: the outcome would be important to judge Silver and the quantitative methods overall, but (quoting Dick) “There’s only one election.” I wish I’d changed his subtitle to “Statistical methods on trial—only one trial.”

Sunkhirous and general: one can track Silver’s predictions for every day by mousing over the graphics at the upper right of his site. In the chance-of-winning projector, Obama does jump up from 72.9% on Monday Oct. 29 to 77.4% on Tue. Oct. 30, but I think that’s a day too early to factor in anything from the storm response—and the Sun. Oct. 28 value had already been 74.6%. Clicking on the “Now-cast” button, meaning the “if the election were held today” projection, shows the same slope all thru the last 4 weeks, since Oct. 12. Thus Sandy made near-zip difference IMHO; Romney’s “momentum” had already reversed over two weeks before.

The two events that show most clearly in his curves are the post-convention bounce which was extended by Romney’s mid-late September gaffes, and the Denver debate. There’s also a June 25-29 bump up which began before the Supreme Court ruling on Obamacare. I regard the ruling as the most important single “event” factor, not for causing a bump but in the overall trend, because it deprived the GOP of the “no accomplishments” cudgel.

9. November 8, 2012 11:35 pm

Yahoo News The Signal (David Rithschild Ph.D. applied economic Wharton Business; David Pennock Ph.D. computer science U. Michigan) also predicted 50-of-50 state outcomes (again, #51 Florida still pending) — back in February, long before the Republicans had even selected a candidate.

http://news.yahoo.com/blogs/signal/signal-forecast-goes-50-50-florida-outstanding-062332033.html