The Thousand Percent Solution
Boeing Dreamliner’s battery of problems
Ray LaHood was until recently the top FAA administrator, the head of the U.S. agency that oversees air safety for the United States. He said at a meeting of the U.S. Conference of Mayors:
“Those planes aren’t going to fly until we are 1,000% sure that they are safe to fly.”
Today I want to talk about percents, beliefs, tests, and predictions.
I just flew in last week on a Boeing 777 from Dubai. It was a fifteen hour and 53 minute flight, made possible by the brilliant engineering of Boeing. The ability to fly that far reliably, on two engines, in reasonable comfort is a remarkable achievement. Yes I was in business class—thanks are due to the group I was visiting, the Qatar Computing Research Institute (QCRI). Boeing has long been one of the leaders in the design of safe planes.
Recently they have had troubles with their new plane, the 787 Dreamliner. It is an innovative plane, with great fuel efficiency through weight reduction based on extensive use of carbon fiber and other light materials.
As you no doubt know it has a small, but significant, problem: its batteries have caught on fire. One instance happened while the plane was on the ground, while another fire caused an emergency landing. I personally like my planes to avoid fires, and I personally prefer them not to make forced landings. I was once on a DC-10 from Rome to Newark that lost major hydraulics and had to make a forced landing in Paris, but that is another story.
LaHood’s comment that he was going to be 1,000% sure is a doubly scary statement to me. As an engineer and scientist I know that a percent is always a ratio from to . You can have or or , but you cannot have anything larger than when talking about belief. It is not possible. Yes you can have a 200% increase in your intake of chocolate—a great idea—but not when it refers to certainty.
The British sometimes write percent as two words as “per cent.” We, in the US, almost always write “percent.” But one word or two, it must be below one hundred percent. Seems like scary to me for the head of an engineering agency to use the term 1,000%.
The other scary part to saying 1,000% reminds me of the famous presidential race in the U.S. in 1972. On July 25, 1972, George McGovern, the Democratic Presidential candidate, discovered his vice presidential candidate had once had electroshock therapy for clinical depression. McGovern quickly made a now famous remark to the press that he was
“1000 percent behind Tom Eagleton, and I have no intention of dropping him from the ticket.”
Eagleton was soon dropped, replaced by Sargent Shriver, and McGovern was defeated by one of the largest landslides in history.
A Billion To One
The real fundamental issue is not the use of a silly phrase like “1000%,” but rather:
how do we test things to be sure they will work?
Prior to the 787′s fires, the prediction by Boeing was that a fire would happen at most once in about one billion flight hours. That seems pretty small, a minor risk, with many other risks of flying well ahead of the battery issue. But they were off by a factor of over .
The question mathematically is how can we calculate the probability of some event , and determine that it will occur in real-life with probability much less than one in a billion? In mathematics we can calculate low probabilities, but that seems to me very different from how to estimate real-life probabilities.
Let’s look at how one might calculate a low probability. Suppose that the event is of the form:
If we can estimate that each of the events occur about of the time, then we can conclude that ‘s probability is less that one in a billion. Of course this assumes that the events are independent events. If they are not independent, then we are unable to make this calculation.
In science and engineering such claims of independence are one way to calculate and claim very low probabilities. The fundamental issue with any such claim is that the events may be related. They may not be independent, and hence our calculation could be way off.
Note, testing whether or not two events are independent is as difficult as calculating the original probability. So what else can one do?
There is a method of stress-testing that is used in industry, and it was used by Boeing’s engineers. The idea is to calculate the probability that something fails by first running it in a more-stringent environment. A simple computer example would be to heat up a processor chip and see how long it takes to fail. The assumption is that if it lasts hours before failing in a heated state, it will last hours in a normal state. The hope is that this gives , and can therefore help compute very low probability events.
The trouble with these methods—independence and stress-tests—is that they do not seem to be based on a formal mathematical theory. The key question is, is there some way to make these methods into a formal theory? I do not know.
A Googol To One
Often what one estimates is not directly the probability of failure in any one instance, but rather the mean time between failures (MTBF). In this case the MTBF would be expressed in units of flight hours, including time on the ground. Two advantages are that the MTBF can often be estimated without explicit reference to probabilities, and the time intervals have units and values that appeal to human reasoning.
Ken notes that the largest system to acquire a reputable MTBF estimate in recent years is the universe, specifically the pocket of the cosmos that includes Earth. This is a by-product of the measurement of the Higgs boson mass coming in between 125 and 126 GeV, significantly under the border of 129-to-130 GeV above which the Higgs field would not have a lower energy configuration to possibly transit to. Last month a speaker’s reference to “many tens of billions of years” as the expected time before transition was re-quoted as “about ten billion” and then “a few billion” years. Happily we can cite an authoritative physics site’s estimate from a noted paper that the MTBF of space under the Higgs field is on the order of
The standard idea of inferring odds by inverting the MTBF then says the chance of space blowing up this calendar year is about one-in-a-googol. We still wonder about how these calculations are done, and what assumptions about independence and randomness of events such as those affecting the Higgs field are made. Estimates analogous to the above “stress tests” have generally used total concentrated energy as the main factor estimating the “” ratio, but we wonder about other possible factors.
A Billion to One Again?
Closer to home, the unlikeliest event in poker may be a hand of 4 aces losing to a royal flush. Note that this can happen with a single 52-card deck if at least one ace is a “community card” that can be shared in different hands, such as in Texas Hold’Em poker. The odds of this have been cited as 2.7 billion to one against, yet it happened (video) in 2008 in the world’s premier poker tournament, the World Series of Poker (WSOP):
Have 2.7 billion deals been played in the history of the WSOP? An estimate based on a thousand tables, each playing about 200 deals a day, puts the number on the order of a few million. The “1-in-2.7 billion” odds figure comes from our source for the above photo from ESPN HD coverage, but we dispute it. For the 5 community cards to include two aces is roughly 1-in-25. For the other 3 cards to include one of the 8 cards to help a royal flush is then almost a 50% chance. Of the remaining 2 cards, one must be one of the other 3 cards that allow for a possible royal flush, just under a 1/8 chance. Hence it seems about 1-in-500 community spreads allow this to happen. Then we need one hand to have the other two aces; figuring 10 two-card hands that’s about (2/5)*(1/50) = 1/125, and ditto for the other two royal-flush cards. This gives a rough Enrico Fermi-style estimate of 1-in-500*125*125 = about 8 million, right on the order of the number of WSOP hands.
Of more importance, if you hold the two aces, and have bet confidently since before the “flop” of the first 3 community cards, and someone across the table hasn’t folded, the odds are probably higher than 1-in-125 that the other guy has the flush cards. Such conditional probabilities play hob with risk analysis. When are we in a conditional-probability situation? Can we know?
What about the 787? Are you ready to fly in it soon? What about the estimation of very low probability events?