The Higgs Confidence Game

Can Nature play games against us?

Peter Higgs once recorded an autobiographical lecture with the immortal title “My Life as a Boson.” This refers to the Higgs boson, which was named for him even though we don’t know whether it exists. We may, however, learn what is formally called evidence of its existence later today (Tuesday) from officials at CERN. This depends on assessment of confidence intervals projected by two teams of experimenters, and how the data from CERN’s Large Hadron Collider (LHC) measures up to them.

Today I (Ken) wish to talk about statistics and social convention in “hard science” such as particle physics. Dick and I are curious whether the assumptions behind the confidence intervals can be violated on both sides: by humans owing to unexpected selection bias, and by Nature possibly acting like a cheating prover in an interactive protocol.

Higgs is a Professor Emeritus of the University of Edinburgh in Scotland. He was actually beaten into print in 1964 on the Higgs mechanism by Robert Brout and François Englert. Higgs had tried to publish, but his paper had been rejected as “of no obvious relevance to physics.” He was, however, the first to point out that the prevailing theory of the mechanism required the existence of a new elementary particle. Note that there are other theories of the Higgs mechanism that lack or conceal the Higgs boson, as well as Higgsless models including this paper.

The Higgs is massive compared to other elementary particles, but ironically may not be massive enough. At about 125 giga-electron-volts (GeV/ with in natural units) it would be a featherweight in boxing weight class stated in pounds. If the top quark weighs in over 175, which used to be heavyweight but is now cruiserweight, the vacuum with its dark energy could well be only meta-stable. This means it could quantum-tunnel to a lower energy configuration, which would knock out our cosmos at the speed of light. Forget the Mayan calendar for 2012 or the latest doomsday preacher—the real rapture may devolve upon Geneva during today’s press conference.

The G-d Particle

Higgs himself believes neither the particle nor the mechanism should carry his sole name, and was happy that he, Brout, Englert, and the three authors of another 1964 paper (Gerald Guranik, Carl Hagen, and Tom Kibble) were all awarded the 2010 J.J. Sakurai Prize for this work. He may have gotten his wish, as the popular name “The God Particle” has stuck to the boson. This is the title of a 1993 book by Nobel prize-winning physicist Leon Lederman and science writer Dick Teresi.

According to Higgs, Lederman had wanted to title the book The G*d*mm Particle to emphasize how elusive the boson was. His publisher declined to have a swear word in the title, but thought it fine to use just “God.” However, they could have settled on the Orthodox Jewish practice of writing “G-d” to avoid situations where the fully-written name might be erased or discarded. The title The G-d Particle could then be read with Lederman’s original meaning or not. Higgs is said to join many scientists regretting the “God Particle” name, more from concern over hype than irreverence.

The Higgs mechanism explains how certain elementary particles acquire mass, via symmetry-breaking. It is not clear whether this determines mass for all particles. By wave-particle duality, the boson is a ripple in the Higgs field. If mass is good, then the boson could be called the “good particle” or the godfather of mass. Finding it—in the form hinted by the present experiments—would complete the set of building blocks for the Standard Model (SM) of particle physics.

There is much less doubt that the Higgs field exists and is good, indeed providential.

Confidence in Discovery

What strikes us is that when and whether the SM Higgs boson is declared discovered depends on a social convention on assessed confidence intervals. Readings that are three standard deviations () away from limits required by the “null hypothesis” of no Higgs presence give 99.7% confidence, but may only be called “evidence.” It takes a event to claim discovery. We have blogged about the sigmas before in connection with the claim for faster-than-light neutrinos.

Rumor has it that the ATLAS team will claim a deviation, and the CMS team will claim about . These would aggregate to about if the results are independent. Whether independence holds between them is being argued, but we have a more basic question first: Where did the value(s) of “” come from?

When an experiment is repeatable, we can soon pin down the standard deviation and interpret accordingly. A first-time or one-off or rare event, however, heightens the inconvenient question discussed here by physicist and blogger Sean Carroll:

What are the error bars on your error bars?

I’ve confronted this issue in my chess research as also described in the post mentioned above. My probabilistic model of move choice projects confidence intervals based on representing legal moves as multinomial Bernoulli trials, after fitting playing-skill parameters to training data. Besides the Bernoulli error there is modeling error from imperfections in the computer chess analysis used as data and from assumptions such as all move decisions being independent that are not-quite-true.

When applied to players accused of cheating by consulting computer programs during games, my model attempts to make judgments of the form, “for Player to have played 70% of the computer’s moves when his expectation was 61% over 200 relevant turns in his nine games is a deviation, hence 740-1 odds against the null hypothesis of no cheating.” In my case I can test the Bernoulli-projected by generating (say) 10,000 random nine-game performances out of the 400-odd games (which make 800 game-plays) in the training data for a nearby skill level, and comparing how many actual agreement percentages with the computer fall within of their projections with -intervals from normal distribution. These tests reveal that the projected given by my current model needs to be divided by about .

This would still allow claiming 2.61 of “actual” deviation, for 220-1 odds. However, it can be argued that random subsets of nine games by various players are different in kind from nine games by the same player, which is why I am expanding the data-taking of performances by players. Supplementing this error analysis by other statistical methods of gauging confidence may be needed—note that this paper by Louis Lyons unabashedly presents the panoply of methods applicable to particle physics.

The Littlewood and Look-Elsewhere Problems

Two other problems being discussed have analogies in the chess work. One is that if I run my analysis on a tournament with over 220 players, I am likely to find a performance that my model would judge to be a 220-1 outlier. This is an example of Littlewood’s Law, named for John E. Littlewood who famously collaborated with Godfrey H. Hardy. Hence the situation requires other distinguishing features to support an accusation of cheating, such as plausible physical evidence or observation of wrongdoing. To be sure, the experimentalists have accounted for these factors.

The related look-elsewhere problem, however, is being said to reduce the ATLAS deviation from to an effective . This also has an analogy in my chess work. Suppose I have two 9-game tournaments with much the same roster of 200 players. Between the tournaments there are 10 ways I can take 9 consecutive games to observe. This is not the same as having 2,000 independent trials to range over, but it greatly improves the odds of 220-1 events or even 740-1 events being merely expected by normal chance.

In the Higgs case the spread effect comes from sliding scales of experimental parameters that dovetail with trying to pinpoint the mass of the particle. Here is an example by physicist and blogger Matt Strassler.

Even with these problems seemingly tamed, there are still many cases where reported phenomena “disappear” on further probing. This should happen only of the time, but seems to happen more often. The chess-playing quantum physicist Tommaso Dorigo detailed one recent important such case on his blog here. The explanation is that this was “really” only a deviation, which happens by chance 32% of the time, or 16% in a one-sided case.

The high rate of disappearing significance overall is still puzzling. Its extent in human sciences was detailed exactly one year ago in a disturbing article by Jonah Lehrer for The New Yorker. If 250 researchers try the same experiment, one would expect 2 or so of them to get deviations (in either direction). The world will then see 2 or so published papers from them, but nary a peep from the 248 who failed and gave up quickly and forgot about it. Thus a significant result may appear independently confirmed when it was actually just by chance, and those failing to reproduce it will then peep up loudly. The effect is equally pernicious with 250 different experiments, especially given a fair chance of a lower-confidence positive from a test deemed related enough to corroborate the original.

Does Nature Play Games?

In physics, however, high-energy experiments are limited, and the problem of vanishing significance makes us consider stranger possibilities. We will not be the first. A cosmic conspiracy hypothesis involving the LHC and Higgs itself was featured in the New York Times in October 2009.

Our thought merely asks whether situations of the kind introduced by Christos Papadimitriou in the paper “Games Against Nature” are “real.” This famous paper is considered one of the forerunners of interactive protocols.

In one of several problems treated in the paper, Papadimitriou varied a standard model of random network faults by allowing the probability of failure in a graph edge to depend also on the current vertex of a “Runner” on the graph. He showed that determining whether Runner has a 50% chance of reaching a goal node when the graph is allowed to conspire against him in this way is complete for . We ask simply,

Can Nature do this?

That is, can Nature alter probabilities of events after seeing information that is not directly local to but involves some goal ? Is the computational power needed to do this available? Note that the word “after” is suspect, but the time-symmetry of many processes involving particles also enhances the plausibility of the question.

We invite those who know more about the pertinent aspects of physics and information theory to comment. For support at least by allusion, however, we note an article in the Nov. 16, 2011 issue of Nature by Gilles Brassard on quantum protocols for attempting to prove spatial position that fail under unexpected attacks. We say unexpected because a paper last year had seemed to prove their security, but this was broken as we noted here. The paper surveyed by Brassard proves instead that in some general settings no secure protocols can exist; a followup paper is here.

Admittedly with little more than surmise, we are prompted to ask: can nature deliver unexpected attacks on the protocols involved in experiments? Can it induce us to accept false conclusions with high probability, in the manner of a “cheating prover”? Does it have the computational capability to do so, in real time?

Open Problems

To state the more positive side of our question, is there a general computational way to “extract independence” from experimental data to maximize confidence in the results?

Does the Higgs boson exist? We note a poll mentioned here of several leading physicists before this month’s rumors, with widely varying and amusing responses.

Update: The results announced at CERN were fairly close to rumors and expectations; for summaries and reactions with varying degrees of intensity see Tommaso Dorigo, Peter Gibbs, Matt Strassler, Quantum Diaries, Peter Woit, Luboš Motl. MSNBC Cosmic Log has a nice simple summary of facts and statistical issues. We also thank commenter Surya for this note which was our first alert about the Higgs news.

Update 6 July 2012: The Higgs inched across the Discovery line in time for the 4th of July. John Huth wrote a humorous post for the blog “Quantum Diaries” channeling Mark Twain. This snippet expresses a point of my original post:

[The CERN presenter], who seemed quite earnest, moved to what I gathered was the ‘punch-line,’ as I could sense the rapt attention of the pilgrims in the room. When he combined the results of two kinds of fireworks, lo-and-behold!, a magic barrier was crossed.

Now, dear reader, you don’t need to know my opinion of statistics, but I will tell you that there is something called a ‘five-sigma’ effect. This manifestation of statistics is deemed by the cognoscenti to be a ‘discovery.’ My guide seemed to be glued to the screen, hanging on every word this gentleman uttered. When I inquired about the meaning of this ‘five-sigma’ miracle, he told me that if it was 4.9, it didn’t count. I was rather amazed that such a small fraction divides a miracle from a non-miracle, but he said it was thus.

Humour aside, it’s convincing to me, hail to the two teams and everyone who built the collider! Meanwhile I described a concrete situation to which my concerns about probabilities can be transferred here.

[Updated links post-press-conference, expanded sentence in intro on Higgsless models, July 6 update.]