Dick Karp was the leadoff speaker this Saturday at the FOCS Theory Day
in Atlanta. He was followed by Mihalis Yannakakis, Noga Alon, and Manny Blum. Sounds to me like a lineup for a baseball team that is in the World Series. I can almost hear an announcer saying:

Now batting cleanup, Mannnnny Blummmmm.

And the crowd goes wild.

Today I thought I would write a summary of what Dick and the others said Saturday. The talks were webcast, but perhaps not all of you watched them. In any event I will add some additional comments that I hope you enjoy.

All of these speakers always give great talks, and this day was no exception. I did notice that Karp and Blum both have titles that ask a question. Is this meaningful?

I cannot resist one story about Dick. Years ago he visited Atlanta to give the keynote at a conference. Rich DeMillo was the program chair, and he and I went out to the airport to pick Karp up. We planned to eat lunch, and then get Dick to the venue for his address. During the lunch we had a wonderful time chatting with Karp. Finally, at one point Dick said that he had a small issue that he was worried about. We of course asked what it was—we were prepared to help him in anyway possible.

Dick explained that he made his slides on the long airplane ride from SFO to Atlanta. In those days talks were made by writing on plastic transparencies with colored pens; my personal favorite were vis-a-vis markers. Dick said that he was concerned about the new type of pens that he had used. He did not use vis-a-vis, but he had used a new brand. We said that was fine. But he added that he was a bit concerned how well the slides would project, since they looked pretty pale. The pens apparently were not as good as old reliable vis-a-vis.

After lunch we jumped in Rich’s car, raced over to the conference hotel, and quickly found the room where Dick’s talk was going to be held. We immediately checked out the projector and more importantly Dick’s slides. He was right—the colors were extremely pale and the slides were nearly impossible to read. But it was way too late to redo them, since the talk was just about to start. Some of the audience were already entering the room.

We went forward—there was no other option. Rich gave a wonderful introduction of Karp, there was a long applause, and then Dick started his talk. He gave a great presentation. Whether the slides are perfect or not, Dick has a way of speaking and presenting his ideas that transcends everything else. Whether the red on a slide, for example, was pale or dark was unimportant. When Dick finished there was again a long applause. His talk was a huge success, but I believe he did throw the new pens away.

Let’s move on to discuss the talks, which were all quite different styles. Yet each was a masterpiece given by one of the leaders of the field.

What Makes an Algorithm Great?

Karp started by quoting me, twice. I was a bit embarrassed—and secretly pleased. The first quote was that “Algorithms are Tiny,” which I have discussed earlier. By the way the ideas in that discussion are joint with Lenore Blum.

• Karp then began to discuss great algorithms based on various
  criteria. The first list was historical:
  • The Positional Number system of Alkarismi.
  • The Chinese Remainder Theorem—which is really an algorithm.
  • The Euclidean Algorithm.
  • Gaussian Elimination.

  Impossible to argue with any on his list. Without positional numbers, I believe, that one could make the case that almost no modern algorithm is possible. The Chinese Remainder Theorem is one of my favorites, and the other two are clearly great.

• Karp then talked about several algorithms that he said were based on great ideas.
  They were,
  • Linear Programming
  • Primality Testing
  • Fast Matrix Product
  • TSP
  • Integer Factoring

  The famous simplex method for LP is still one of the best ways to solve linear programs. Karp pointed out that the ellipsoid method was a great theoretic result, but the interior-point methods were the first to solve LP’s in a way which was both in polynomial-time and practical. Karp gave a nice survey of the work on the TSP: he pointed out that Nicos Christofides’ wonderful -approximation algorithm for the metric TSP is still the best known. If you somehow do not know it check it out. It was discovered in 1976—that’s a lot of FOCS’s ago. Dick also pointed out that Volker Strassen’s famous fast matrix product algorithm was a surprise to the mathematical community.

• Karp is an expert in so many things, but perhaps his roots are in combinatorial optimization. So he went into some detail on the great algorithms from that area.
  • Minimum Spanning Tree
  • Shortest Path in Graphs
  • For-Fulkerson Max-flow Algorithm
  • Gale-Shapley Stable Marriage
  • General Matching in Graphs

  In 1965 Jack Edmonds’ paper titled “Paths, Trees, and Flowers” not only showed how to do matching for non- bipartite graphs, but also laid out an informal basis of what later became P, NP, and co-NP. An incredible achievement for that point in time.

• Karp next discussed randomized algorithms. I sometimes think we could not have a FOCS meeting if we disallowed the use of randomization.
  • Primality Testing
  • Volume of Convex Bodies
  • Counting Matchings
  • Min-Cut of Graphs
  • String Matching
  • Hashing

  All of these algorithms are great, and Karp spent some details on the volume algorithm of Alan Frieze and Ravi Kannan. He said nothing about the beautiful string matching algorithm that is due to Michael Rabin and himself. He was modest, but I am under no constraint. Their matching algorithm is one of the examples of the immense power of randomness. Their algorithm is theoretically fast and is practical. Actually it is more than practical, it is I believe the algorithm of choice for most packages. A wonderful algorithm, that would make my personal top ten.

• Karp talked next about heuristic algorithms. These, he said, present an important challenge to theory. The central question, of course, is why do they work so well in practice?
  • Local Search
  • Shotgun Sequencing Algorithm
  • Simulated Annealing

  Dick stated that Myers’ great work on sequencing was critical to the effort that first sequenced the human genome at Celera—then a company in competition with the NIH funded labs. What is so neat about Myers’ algorithm is that it required him to understand the structure of the human genome. The genome is not random, has structure, and that structure makes what Gene did so difficult.

• Karp and complexity theory:
  • NLOG is Closed Under Complement
  • Undirected Connectivity is in DLOG
  • Space is More Powerful Than Time

  Karp agreed with my earlier discussion that even complexity theory is really all about algorithms. He selected the above as the some of the most outstanding ones. John Hopcroft, Wolfgang Paul, and Leslie Valiant proved that deterministic time is in deterministic space . Their proof relies on several algorithmic ideas: the block-respecting method and the pebbling game strategy.

• Karp only had time to list a number of great algorithms that show that theory can have impact on the “real-world.” All of the algorithms in his list have changed the world.
  • Fast Fourier Transform
  • RSA Encryption
  • Miller-Rabin Primality Test
  • Reed-Solomon Codes
  • Lempel-Ziv Compression
  • Page Rank of Google
  • Consistent Hashing of Akamai
  • Viterbi and Hidden Markov Models
  • Smith-Waterman
  • Spectral Low Rank Approximation Algorithms
  • Binary Decision Diagrams

  Without these algorithms the world would stop: no web search, no digital music or movies, no security, and on and on.

• Karp finally gave a short list of algorithms from the area of information transfer.
  • Digital Fountain Codes
  • CDMA
  • Compressive Sensing of Images

  The first is a breakthrough method of Michael Luby that is a near optimal erasure code. In many situations information may be lost, rather than corrupted. Since it is lost the decoder knows that it is missing, which allows for a whole different type of error correcting code. These codes are extremely powerful: they are easy to encode and decode, and need relatively few extra bits of redundancy. Compressive Sensing is a relatively recent idea that is based on deep mathematics, yet is rapidly changing image compression. The creators are David Donoho, Emmanuel Candès, Justin Romberg, and Terence Tao—see this for many papers and articles from the area.

The talk was wonderful. Wonderful. In one hour we were carried from ancient times, past the dawn of modern computational complexity, then to the modern era, and the present—with a peek at the future. A masterful talk by the master. Later in the meeting Alon said that he could not imagine doing what Karp did. I completely agree.

Computational Aspects of Equilibria.

Yannakakis spoke about equilibria phenomena in game theory and economic theory. He started by saying that he had two hours worth of slides, but kindly finished nicely on time: close to the von Neumann Bound. John von Neumann once defined the perfect length of a talk as a micro-century, which works out to just about minutes.

• Yannakakis first explained how many systems can evolve over time to a stable state. He spoke at some length about the classic example of neural nets and a 1982 theorem due to John Hopfield that proves that they always converge to a local minimum. The argument is based on showing that a certain potential function decreases and eventually reaches a local minimum.
• Yannakakis then switched to a discussion that was mostly about various types of games. He discussed in detail the famous result of John Nash, proved in 1950, that any non-zero sum game always has at least one mixed equilibria point.
• Yannakakis’ key point was that Nash’s proof that mixed equilibria always exist was based on the famous Brouwer Fixed Point Theorem (BFPT). This theorem that is named after Luitzen Brouwer is also, as Mihalis said, used to prove that a whole host of other game/economic systems have a stable equilibrium point.

• Yannakakis then showed the power of modern complexity theory. Just because the BFPT is used to prove something does not rule out the possibility that there is a proof that avoids it. But, this is not the case. He introduced three complexity classes: , , and . They capture respectively: general -player games, -player games, and finally games that have pure equilibria. A beautiful point is that he could use these classes to show that BFPT is equivalent to Nash’s famous theorem for multiple player games, and thus it is essential for the proof. The power of modern complexity should not be underestimated. Without the crisp notion of a “complexity class” I cannot see how any result like this could even be stated—let alone proved. Terrific.
• Yannakakis also pointed out that solutions to multiple player games may not be rational numbers, in general. This was, by the way, known to Nash back in 1950. The problem with this is that then there may be no way to write down the exact solution to the problem. This plays havoc with complexity theory. Mihalis gave the analogy to the sum of square root problem: Is the following true,

  

  where the are natural numbers and is a rational number. It is a long standing open problem whether or not this can be solved in polynomial time. See this for a discussion that I had earlier on this topic.

• Yannakakis finally gave us the usual news about the complexity classes: not much is known about their power. Oh well. The following diagram summarizes all that is known about them: (an arrow denotes inclusion)
  

His talk summarized quite neatly the relationship between equilibria problems and certain complexity classes. I wish that Yannakakis had more time to give more details—perhaps another talk.

Disjoint Paths, Isoperimetric Problems, and Graph Eigenvalues.

Alon started by pointing out that his first FOCS paper was at the . At that meeting, in 1984, he spoke on eigenvalues and expanders—he said he seemed to be stuck. He then pulled out a small piece of paper and proceeded to read,

I would like to thank the many FOCS program committees for their hard work and effort—except for the FOCS program committees of 1989, 2003, and 2006.

Even one of the greatest theorists in the world gets papers rejected from FOCS. This got a big laugh, yet I think it actually raises a serious question about the role of conferences. I know that is being discussed both on-line and off, but let’s move on to the rest of his talk and leave that discussion for another time and place.

Noga really gave highlights of two problems:

• Alon first discussed a kind of routing game. He imagines that you have some fixed node degree expander graph. The game is this: you will be given a series of requests of the form:

  

  where this means that you are to find a set of edge disjoint paths from to for each . He proves that on a special type of expander that not only can this be done for such that but that it can be done on-line. That is there is an algorithm that selects the path from to without knowing the requests that will follow,

  

  This seems amazing to me. See the paper joint with Michael Capalbo for full details on how this works.

• Alon then spoke on spines. Imagine dimension space divided into unit cubes. You are to select a subset of each cube so that this “spine” does not allow any non-trivial path that moves from cube to cube. In a sense the spines are a kind of geometric separator. Since separators play a key role in many parts of theory, it may come as no surprise that spines also are very useful. A deep result of Guy Kindler, Ryan O’Donnell, Anup Rao, and Avi Wigderson shows that a spine can have surface area bounded by where is the dimension of the space. However, the shape of their spine is essentially not known; it cannot be seen. The discussion here may help understand more about spines and their applications. Noga’s main result is a new uniform proof that solves this and other spine problems.
• Alon also had a simple piece of advice for all of us. He pointed out that we all use search engines to try to find mathematics that can help us in our research The trouble is that it is impossible to type in mathematical formulas into search engines. His suggestion is: give all your lemmas meaningful names. This may allow people to find your work, and then reference it. He gave an example of one of his lemmas:

  Lemma: (A simple discrete vertex isoperimetric inequality on the
  Dirichlet boundary condition)

When I Googled this “name” the second hit was his paper—not too bad.

A wonderful talk, with light touches, and beautiful results; even though it contained many technical parts, I believe that most came away with the basic ideas that Noga wanted to get across.

Can (Theoretical Computer) Science Come to Grips With Consciousness?

Blum spoke about a life long quest, that started when he was six years old, to understand consciousness. Blum has a record of looking at old problems in new ways, of looking at new problems in his own way, of creating whole fields of study, and thus we should take his ideas on what is “consciousness” very seriously. Let’s call this the: What is consciousness problem. (WICP).

It is hard to really do justice to his wonderful talk, but I think there are a few high level points that I can get right:

• Blum states that now may be the first time that real progress can be made on the WICP. This is due to the confluence of two events. The ability of researchers to use FMRI machines to watch a person’s brain as they think is relatively recent development. Second, is the maturing of a powerful theory called the Global Workspace Theory, which was created by Bernie Baars. I will not explain the theory, since I do not understand it. But Blum says that it has a very computational flavor, which may mean that theory can make an important contribution to the WICP.
• Blum explained that when he was a graduate student he worked with some of the greats such as Warren McCulloch and Walter Pitts. They are famous for their notion of the McCulloch-Pitts neuron, which they proved could be the basis of a universal computing device. Today we study threshold functions and the complexity class that are modern versions of their pioneering work.
• Blum pointed out a simple fact about the eye that I found fascinating: In the dark take a light source, and quickly move it around in front of you. You will see a path of light. This happens, Manny explained, because the eye responds only to motion. Then, keep the light source fixed and now move your head around. You will not see a path of light: you will see just a point of light. Somehow the eye and brain together can tell the difference between these two cases. Terrific.
• Blum then shifted into a more theory oriented part of his talk. He explained what he called templates. They are his way of modeling how we solve problems. His stated goal is quite ambitious; if we could understand WICP he thinks that we might be able to make robots/agents that learn. The template model is a kind of tree. For example, suppose that you are trying to prove a theorem. Then, the root of the tree would contain a “hint” or some high-level idea that gets you thinking in the right way about your theorem. Below that would be more precise yet still informal pieces of information. Eventually, the leaves of the tree would contain more formal pieces that make up the actual proof.
• Blum had a simple but great piece of advice to us all. If you are faced with a problem that you cannot solve, then modify the problem. Change it. Try another problem. He argued that this often may be interesting by itself, and sometimes may shed light on the original problem. Great advice.
• Blum ended with a pretty puzzle. Is there a irrational number so that

  

  for a rational ? The answer is yes, and he explained how his templates could guide you to the solution to his problem.

A wonderful talk, from one of the great visionaries of the field.

Open Problems

Before turning to some open problems I want to thank the following people for making Theory day such a great success: Dani Denton, Milena Mihail, Dan Spielman, and Robin Thomas. Thanks for all your hard work.

Robin pointed out that it was also ACO’s anniversary, and all of the talks should be online later this week.

Now some open problems that each speaker implicitly raised:

Blum: What do you think about WICP? Can theory help us understand WICP? Is the Global Workspace Theory one we can contribute to?
]]> http://rjlipton.wordpress.com/2009/10/27/highlights-of-focs-theory-day/feed/ 17 lipton all classes http://rjlipton.wordpress.com/2009/10/22/helping-wall-street-cheat-with-theory/ http://rjlipton.wordpress.com/2009/10/22/helping-wall-street-cheat-with-theory/#comments Thu, 22 Oct 2009 12:09:50 +0000 rjlipton http://rjlipton.wordpress.com/?p=3754 ]]>

How to create “bad” derivatives that cannot be detected

Sanjeev Arora is the one of the most creative theorists in the world. He helped create the concept of Probabilistically Checkable Proofs along with Uriel Feige, Shafi Goldwasser, Carsten Lund, László Lovász, Rajeev Motwani, Shmuel Safra, Madhu Sudan, and Mario Szegedy. On his own he found a wonderful approximation algorithm for the Euclidean TSP, and has done many other terrific things. I was at Princeton when we hired him, and it was clear back then that he was special.

Today I had planned to talk about some of the papers of FOCS 2009, but will instead talk about a new paper of Arora with Boaz Barak, Markus Brunnermeier, and Rong Ge (ABBG) on finance. I really like their paper, and it seems to me to be one of those seminal papers that could launch a whole subfield.

Sanjeev is currently the PI of a large NSF project looking into intractability and P=NP. He is also the creator and current head of the theory committee that has and continues to try to work with NSF to get more support for theory. The previous head was Richard Karp who did an excellent job running the committee for a number of years.

I feel like saying stop the presses. But this is not being printed so I guess that does not apply anymore. In one of my favorite movies, The Paper, there is a scene where the main character an editor of a newspaper, played by Michael Keaton, is told by one of his reporters, played by Randy Quaid, “you got to say it.” They are standing next to the high speed printing system. Papers are streaming by at a huge rate, and they want to stop the printing and change the lead story, since the the paper being printed has gotten it completely wrong. Quaid says again to Keaton you’ve got to say it. Finally, Keaton yells “stop the presses.” Of course nothing happens right away, next See the movie if you want to know more. I really like this flick.

As newspapers go out of business and news becomes electronic, what will we get to say in the future? Re-boot the servers?

Well that’s what I am doing today. So let’s turn to the world of finance and the paper of ABBG.

Let’s Play Cards

I am no expert on finance, so to explain ABBG I will use an analogy of playing a card game.

Imagine that there are two players Alice and Bob who wish to play a simple game of cards. Alice has a standard deck of playing cards and she is the dealer. Suppose she shuffles the cards and then deals out a “hand” of one card to Bob. He wins if the hand is a red card; otherwise, he loses. If Alice shuffles fairly, Bob has exactly a chance of winning—it is a fair, if boring game. I could make the game more realistic, but that would change nothing except make the analysis more involved.

The problem with the game is the simple phrase: If Alice shuffles fairly. What happens if Alice cheats? She could cheat in a number of ways; here are some that I have arranged in increasing order of skill required by Alice:

• The deck might be a stacked deck. Suppose that Alice has changed the deck so that it consists of only black cards. Then, Bob will have no chance of winning. Or she could have removed one red card and replaced it with a black one, then Bob’s chance of winning is reduced to . Note, Alice requires no special skills; she can be shuffling away and let Bob cut the deck as much as he likes. The game is a still losing one for Bob.
• The deck might be a marked deck. In this case Alice cheats by having the deck contain all the usual cards, but the backs of the cards are “marked” so that she can tell whether or not a card is a red or a black card. Then, she just shuffles away until she sees that the top card is black, she then stops and deals the top card to Bob.
• The deck might be a real deck. In this case Alice cheats by not shuffling the deck properly. She could have originally arranged that the red cards so they are on the top of the deck. With a few careful shuffles of the right kind, she can vastly lower Bob’s chance of winning. Even worse if she is a really good “card mechanic” she can show Bob the deck—that is she can fan out the faces of the cards and show them to Bob. The cards will look to be in a random order. But if she can execute the correct type of shuffles, then she can force that Bob always gets a black card. Essentially, she undoes the “random” order of the cards by performing careful shuffles and moves a black card to the top. This requires quite a bit of skill: a word of advice, never play cards with magicians.

Let’s Play Derivatives

Again there are two players who I will still call Alice and Bob. Alice is now creating financial derivatives and trying to sell them to Bob and to others. Before I even explain what a derivative is you probably can guess the high level insight: Alice creating the derivative is the same as Alice dealing the cards. If Alice creates them fairly, then Bob is okay. However, just as Alice can cheat at cards, she can cheat at derivatives. The reason Alice can cheat is the same as before: Alice may have extra information that she can exploit to cheat Bob. In a sense there is no surprise that Alice could cheat at cards, but that Alice could cheat at derivatives is the main result of ABBG.

With all due respect I do not mean to diminish the brilliant insight of ABBG—I really like their paper. My goal is to explain that what is happening with derivatives is very much the same as cheating at cards.

However, there are a number of interesting technical differences between cheating at cards and cheating at derivatives. One of them, by the way, is that the stakes are vastly larger. People play card games for stakes in the thousands or even hundred of thousand dollars, but banks “play derivatives” for billions.

The ABBG Result As A Card Game

My usual disclaimer: read their paper for the details and full statements of their results. I would like to give a high level view of their basic model, what they assume, and what they prove.

As I understand the key idea is that Alice has a collection of assets . She then creates derivatives where each is a random subset of size of the assets. Each asset will then either fail or succeed. The derivative pays off if at least one-half of the assets in it succeed; otherwise, it does not pay off. If Bob buys the derivative , then he wins if a majority of his assets succeed; otherwise, he loses.

Clearly, Alice and Bob are playing a type of game. Here is a way to model this as a card game. Alice has a deck of cards of the assets and she deals Bob a hand that consists of cards. That is his derivative. Later on, it is revealed which assets fail and which succeed, and Alice and Bob can determine who won the game.

In the derivative model Alice actually creates a number of derivatives at once. I must admit that I am not sure why this is done, but apparently it is the way that derivatives work. More on this later on.

Let’s model this as a card type of game. We will use cards to stand for assets and hands to stand for derivatives. Each card has a label on the back and a color red/black on the front. From Alice’s perspective, red stands for failure and black for success. Thus, for Alice, a winning hand is one with a majority of black cards. We will also assume that there are an equal number of red and black cards in the deck.

Alice has copies of a special deck of cards. Each deck has cards. All of these decks are identical. Alice shuffles the first one and deals out a hand of cards—this is the first derivative. She then takes the second deck and deals out another hand, and so on until all hands have been dealt.

These hands define the derivatives. Note, it is important that assets can be in more than one derivative, since Alice may wish to sell more than derivatives. Thus, it is critical that Alice has multiple copies of the deck.

If Alice has no idea which asset with fail or succeed in the future, this is a fine game. However, if Alice knows that some of the assets are more likely to fail than others, then she can cheat. The ABBG paper calls these lemons. Note, since the back of the cards are marked with the name of the asset it is like the cards are marked.

Even if Alice does not have a perfect mapping from an asset to its color, Alice could possibly cheat. She can do that if she has some partial information about the color of the cards. She does this by the way she deals out the cards to make the hands. Her goal is to get more black cards into hands, so she will win those hands. Even some partial information here is enough to greatly increase the odds in her favor.

The ABBG Result

In the ABBG paper the type of theorems they prove rely on two key points about the abilities of Alice and Bob.

The first concerns Alice. She must have some partial information about the lemons, the color of the cards, if she is going to cheat Bob. The exact number of lemons and how much they affect the cost to Bob is a technical point. I suggest you read their paper to get the exact results.

The second concerns Bob. If he is going to be unable to detect that Alice is cheating he must be computationally limited. In any cheating scam the person being cheated must be limited in their ability to detect the cheating, or it will be discovered. In the examples of ways that Alice can cheat at cards, Bob must not notice certain things. In one case, he must not notice that the deck is all black cards, in another that the deck has been marked, and in the last that the deck is in a pre-arranged order. His ability to detect Alice cheating in these ways gets harder: the first is easy—just look at the deck, while the last is quite hard—it is hard to notice that she is executing perfect shuffles. Notice that the more skill Alice uses to cheat, the harder it is for Bob to detect it. In similar ways, ABBG demonstrates clever skillful ways of cheating that makes it extremely hard for Bob to detect.

The ABBG results depend on Bob being limited in his computational ability. In particular, Bob must not be able to detect that Alice has dealt the hands in a non-random manner. The only way that Bob can detect this is their main theorem; in order for Bob to discover that Alice has cheated him in constructing the derivatives he must be able to discover that a graph has a dense subgraph. The intuition behind this result is that if Alice deals the hands, that is creates the derivatives, in a non-random manner the structure of the sets will define a graph that deviates enough from randomness to leave a proof of her cheating. However, the key insight is that detecting this is an intractable problem.

In summary, the ABBG phenomena is:

1. With enough lemons Alice can create non-random derivatives that will generate extra cost for Bob;
2. With limited computational ability Bob will be unable to detect that Alice has acted in a non-random manner.

The exact statement of this result is technical, but I believe that the above captures the key insights that they have made. Here is an example, in their terminology of the type of results that they prove:

Theorem: When , , an subgraph will generate an extra lemon cost of at least .

What this is saying is: Alice can cheat and cause a certain amount of extra loss over fair play. For the meaning of the symbols see ABBG, but the symbols denote the main parameters of their model: the number of assets, the size of the derivative, the number of lemons, and so on.

Is ABBG Really A Problem?

The last thing Wall Street seems to need is advice from theorists on how to cheat. They seem to have done quite well on their own in the past and in recent times. Just say Bernie Madoff—if you type “bernie” into Google he immediately comes up—I feel sorry for anyone named Bernie.

The ABBG paper has nevertheless caused a quite stir among finance bloggers. For example look at this, this, and this.

The core of the cheating described in ABBG, in our terminology, is that Alice deals out more black cards than she would if the dealing was completely random. Thus, Bob can only detect that Alice is cheating if he is able to discover that she is dealing in this way. The clever insight of the ABBG paper is that this detection is equivalent to detecting a dense subgraph in a graph.

The conventional wisdom is that this is a hard problem. You all know that I have some doubts about this, but even with those aside this is a reasonable issue.

• A problem can be hard in theory, but might be easy in practice. One needs to know the size of the graphs that are used in practice, since even worst case algorithms could do well.
• Another possibility is that there might be good approximation algorithms that work on the graphs ABBG uses.
• Finally, the problems they create are “planted” versions of the dense subgraph problem. I think that this version of the subgraph problem could be simpler.

Overall I think that I would not want to have to solve these problems. But the simple claim that they are “intractable” is one that the authors, I am sure, would agree is a claim that needs to be carefully checked.

Solutions?

One thing we know is that when there is a trust problem in a digital system often we can overcome the trust issue. Sometimes this requires quite simple ideas, sometimes this requires modern cryptography, and sometimes something in between. I think there could be a number of ways to make it so Alice cannot cheat when creating the derivatives.

One idea that comes to mind is some protocol that forces her to really use proper random bits in constructing the derivatives. You could imagine that Bob is protected in this way. Another approach might be to let Bob select his own random derivative. This would not allow Alice to cheat, but would allow Bob to cheat. If he was the one with knowledge about the assets, then he could cheat Alice.

The key insight seems to be that this is a crypto problem that we have seen before: having two or more parties agree on a random string. I think that this can be done. Perhaps the most interesting question is can some special properties of derivatives be used to make the protocol more efficient.

Open Problems

Are there other types of ways to cheat in markets? Also one neat question might be how to develop a protocol that will allow both parties to be sure that all is well? I expect that we will see many more papers on this topic.

]]> http://rjlipton.wordpress.com/2009/10/22/helping-wall-street-cheat-with-theory/feed/ 22 lipton images http://rjlipton.wordpress.com/2009/10/18/happy-fiftieth-birthday-focs/ http://rjlipton.wordpress.com/2009/10/18/happy-fiftieth-birthday-focs/#comments Sun, 18 Oct 2009 18:08:39 +0000 rjlipton http://rjlipton.wordpress.com/?p=3731 ]]>

A look back at some the early days of the FOCS conference

Alvy Ray Smith is a famous researcher who has won many awards for his seminal work in computer graphics. He is a theorist also: he has had several papers published at FOCS, for example.

Today I thought, since FOCS’s anniversary is soon, that it would be fun to look back at the first conferences and see what researchers were doing. Below is a singing tribute to FOCS:

The first observation is that they were not attending FOCS. The conference we call FOCS has been around for fifty years, but it has gone through two other names before becoming FOCS. The name was changed twice: In 1960 until 1965 it was the “Symposium on Switching Circuit Theory and Logical Design;” then, from 1966 to 1974 it was the “Symposium on Switching and Automata Theory;” and finally it switched to the “Foundations of Computer Science.” Hopefully, this is the last name, but who knows?

When the name was: Switching and Automata Theory, it had the “bad” abbreviation as SWAT. I recall the business meeting where we discussed changing the name to something better. I believe—I could be wrong—that Emily Friedman made two key contributions.

First, she suggested the name, FOCS, and also she helped design the “fox” logo. Emily was very active in the 1970’s and did some quite nice work on program schema and various aspects of language theory. Alvy Smith designed the famous cover that has used for years on the proceedings.

Speaking of covers of proceedings, when I was the program chair for STOC I was very excited about the honor. I especially liked the idea that the program chair got to select the color of the proceedings, since I thought this would be fun. Finally the big day came, we had selected the papers, the accept and reject letters had been sent, and it was time to pick the color for the cover. The print person at ACM called me to ask about my choice of colors. I told him that I wanted the cover to be black with white letters, which I thought would be dramatic and unique. He told me that he would select a color and hung up the phone—really. I did not get to choose. Oh well.

Let’s now turn to look at what the field was up to years ago.

Symposium on Switching Circuit Theory and Logical Design

Here are the papers from the 1963 conference:

• Infinite sequences and finite machines: Muller, David E.

• Two-sided finite-state transductions: by Elgot, C. C.; Mezei, J. E.
• On computability by certain classes of restricted turing machines: by Fischer, Patrick C.
• Bilateral threshold nets: by Frazer, W. D.
• Threshold gate realizations of logical functions with don’t cares: by Coates, C. L.; Lewis, P. M.
• On the analysis of functional symmetry: Arnold, R. F.; Lawler, E. L.
• The minimal synthesis of tree structures: by Lawler, E. L.
• Determining the best ordering of variables in cascade switching circuits: by Levien, Roger E.
• Sequential circuit synthesis using input delays: by Eichelberger, Edward B.
• Finite automata and badly timed elements: by McNaughton, Robert
• Demonstrating Hazards in sequential relay circuits: by Booth, Theodore M.
• Logical design theory of NOR gate networks with no complemented inputs: McCluskey, E. J.

• A survey of asynchronous logic: Comparing various definitions and models for asynchronous switching circuits: Miller, R. E.

Symposium on Switching and Automata Theory

Here are the papers from the 1966 conference:

• Context-free language processing in time : by Daniel Younger.
• Syntax directed transduction: by Lewis, P. M.; Stearns, R. E.
• Simple deterministic languages: by Korenjak, A. J.; Hopcroft, J. E
• One-way stack automata (Extended abstract): by Ginsburg, Seymour; Greibach, Sheila A.; Harrison, Michael A.
• Real-time computation by n-dimensional iterative arrays of finite-state machines: by Cole, Stephen N.
• The recognition problem for the set of perfect squares: by Cobham, Alan
• Roots of star events: by Brzozowski, J. A.
• Subdirect decompositions of transformation graphs: by Yoeli, M.; Ablow, C. M
• Pair algebra and its application: by Liu, C. L.
• Graphs of affine transformations, with applications to sequential circuits: by Gill, Arthur
• A method for the combined row-column reduction of flow tables: by Grasselli, A.; Luccio, F.
• Standard minimum transition time secondary assignments for asynchronous circuits: by Epley, Donald L.
• A row assignment for delay-free realizations of flow tables without essential hazards: by Unger, Stephen H.
• Synthesis of multiple sequential machines: by Smith, Edward J.; Kohavi, Zvi
• Realization of sequential machined with threshold elements: by Hadlock, F. O.; Coates, C. L
• Fast Simulation of Nondeterministic Turing Machines: by Reading, R. U.
• The application of threshold logic to the design of sequential machines: by Masters, Gilbert M.; Mattson, Richard L.
• Minimization and convexity in threshold logic: by Dertouzos, Michael L.; Fluhr, Zachary C.
• On minimal modulo 2 sums of products for switching functions: by Even, S.; Kohavi, I.; Paz, A
• Conjunctive encoding of Boolean matrices: by Stover, D. R.; Epley, D. L.
• Asynchronous propagation-limited logic: by Goldberg, J.; Stone, H. S.
• The synthesis of multipurpose logic devices: by King, W. Frank
• The universal logic block (ULB) and its application to logic design: by Forslund, D. C.
• Statistical properties of random digital sequences: by Booth, Taylor L.
• Reconsider the state minimization problem for stochastic finite state systems: by Ott, Gene H.
• An application of coding algebra to the design of a digital multiplexing system using linear sequential circuits: by Helm, H. A.
• Automorphism groups and quotients of strongly connected automata and monadic algebras: by Bayer, R.
• On the automorphism group of a reduced automaton: by Paul, Manfred

Some Observations

One issue that earlier researchers worried about was the copyright issue. You may see that papers are called “extended abstracts” or “preliminary version.” This was so that journals would allow the authors to publish a final version again. There were relatively few papers, and there were no parallel sessions.

The topics of course were of course different from what we think about today. Yet not too far from things that we are interested in today. It is very clear that exact results were the goal. I think almost no papers were about approximations—in any sense of the word. The field had not discovered that one way to make progress on hard problems was to lower the bar. Since getting the answer was often too hard, try to get an approximation. This lesson would not arrive for a few years.

The papers also just looked different, since they were typed not typeset. Take a look at some of them. You will see handwritten symbols:

SWAT 1971

Here are a few selected papers from the 1971 conference. You will notice that now the topics are very close to current ones. For example, in FOCS 2009 there is a paper by Nikhil Bansal and Ryan Williams that is on boolean matrix product, which is related to the Fischer and Meyer paper below.

Matching: This is the famous paper of John Hopcroft and Richard Karp called: A Algorithm For Maximum Matching In Bipartite Graphs. The special symbols in this and other papers were written in by hand: in the abstract the symbol was left out and the running time was claimed to be order instead of : note the extra space around the last . The things we had to do before .

DFS: This is the famous paper by Robert Tarjan, no Endre as he added later on, called: Depth-First Search And Linear Graph Algorithms. Tarjan used a special font on his typed papers that we could all tell was from Stanford. Note also he calls it a “working paper.”

Boolean Matrix Product: This is a neat paper by Mike Fischer and Albert Meyer called: Boolean Matrix Multiplication and Transitive Closure. In this paper they showed that fast matrix multiplication can be used to compute boolean matrix products: also they count bit operations not arithmetic operations.

Description Size: This is another neat paper by Mike and Albert, although this time it’s a “Meyer and Fischer” paper called: Economy of Description By Automata, Grammars, And Formal Systems. The main results show that more powerful devices can tremendously affect the size of the description of a set. For example, there is an exponential gap between deterministic finite state automata and non-deterministic automata; even more impressive there is a double exponential gap between deterministic finite automata and pushdown automata. I believe I was asked about this earlier and this is the place to look for such results.

Real Computation: This is a paper that perhaps should be better known. It is by Fred Abramson and is called: Effective Computation Over The Real Numbers. Fred extends Turing Machines to have values that are real numbers. This paper was probably a bit ahead of its time; I will discuss the real P=NP question of Lenore Blum, Michael Shub and Stephen Smale in the future.

Open Problems

I hope the FOCS meeting continues to be an important meeting for another fifty years and then fifty more. I do wonder what topics will be important for FOCS 2019 or FOCS 2029? Here are possible papers that could appear in the FOCS 2019:

1. Cryptography Without Assumptions: by Alice Brown, Bob Green, John Lime, Hila Mustard, Lincoln Violet, Chuck Yellow,
2. An Space Approximation Algorithm in the River-Streaming Model for the iBrain Optimization Problem: by Carol Smith and Fred Thomas;
3. The Extended Generalized Riemann Hypothesis Implies The Scott Quantum Hierarchy Collapses: by Ryan Browodski and Sue Lin-Jun.

What do you think? What are your predictions?

]]> http://rjlipton.wordpress.com/2009/10/18/happy-fiftieth-birthday-focs/feed/ 10 lipton images-2 images-3 images Picture 1 http://rjlipton.wordpress.com/2009/10/15/multi-party-protocols-and-focs-2009/ http://rjlipton.wordpress.com/2009/10/15/multi-party-protocols-and-focs-2009/#comments Thu, 15 Oct 2009 22:29:07 +0000 rjlipton http://rjlipton.wordpress.com/?p=3692 ]]>

Multi-party protocols, bounded width programs, concentration theorems, and FOCS 2009

Ashok Chandra is a famous theorist who has done seminal work in the many different areas. His early work was in program schema, an area that was central to theory in the 1970’s. Chandra then moved into other areas of theory and among other great things co-invented the notion of Turing Machine Alternation.

Later in his career, while still an active researcher, he began to manage research groups: first a small one at IBM Yorktown, then a much larger one at IBM Almaden, and later groups at other major companies including Microsoft. As a manager, one example of his leadership was the invention by his group at Almaden of that little trackpoint device that plays the role of a mouse. The trackpointer eventually appeared on the keyboards of IBM’s Thinkpad laptops—now called Lenovo laptops—a wonderful example of successful technology transfer.

Today I want to talk about mutli-party protocols, an area that Chandra helped create in the 1980’s. Then, I will connect this work with a new paper that is about to appear in FOCS in days. This is the FOCS and I am looking forward to attending it, even though I will not get any Delta miles for this one—it is in Atlanta.

I still remember vividly meeting Ashok for the first time: it was at the Seattle airport in 1974 at the taxi stand. We were both on our way to the sixth STOC conference—it was my first STOC. I knew of Ashok, since I had read his beautiful papers on program schema theory. Somehow we figured out that we were both going to the same hotel and we shared a cab. Our conversation in the cab made me feel I was welcome as a new member of the theory community: I have never forgotten his kindness.

At that STOC Chandra presented some of his recent work on program schema, “Degrees of Translatability and Canonical Forms in Program Schemas: Part I.” Also presented at the conference were these results, among others:

• Leslie Valiant: The Decidability of Equivalence for Deterministic Finite-Turn Pushdown Automata.
• John Gill: Computational Complexity of Probabilistic Turing Machines.
• Vaughan Pratt, Michael Rabin, Larry Stockmeyer: A Characterization of the Power of Vector Machines.
• Stephen Cook, Robert Reckhow: On the Lengths of Proofs in the Propositional Calculus.
• Robert Endre Tarjan: Testing Graph Connectivity.
• Allan Borodin, Stephen Cook: On the Number of Additions to Compute Specific Polynomials.
• David Dobkin, Richard Lipton: On Some Generalizations of Binary Search.

The topics then were very different from today: then, there was no quantum theory, no game theory, no learning theory, and many complexity classes had yet to be invented. Yet many of the topics are still being studied today—perhaps a discussion for another day.

Let’s now turn to multi-party protocols and eventually to concentration theorems.

Multi-Party Protocols

Chandra, Merrick Furst, and I discovered the basic idea of multi-party protocols because we guessed wrong. We were working on proving lower bounds on bounded width branching programs, when we saw that a lower bound on multi-party protocols would imply a non-linear lower bound on the length of such programs.

Let me back up, one summer Merrick Furst and I were invited to spend some time at IBM Yorktown by Ashok. The three of us started to try to prove a lower bound on the size of bounded width programs.

A bounded width program operates as follows. Suppose that the input is . I like to think of a bounded width program as a series of “boxes.” Each one takes a state from the left, reads an input which is the same for the box each time, and passes a state to the right. The first box is special and gets a start state from the left; the last box is special too and passes the final state to the right. The total number of states is fixed independent of the length of the computation. Note, also while the boxes read the same input each time, an input can be read by different boxes. The length, the number of boxes, is the size of the bounded width program.

The reason this model is called bounded width is that the number of states is fixed independent of the number of inputs. The power of the model comes from the ability to read inputs more than once. Note, with that ability the model could not compute even some very simple functions.

We guessed that there should be simple problems that required length that is super-linear in the number of inputs, . This guess was right. What we got completely wrong, like most others, was that we thought the length might even be super-polynomial for some simple functions. David Barrington’s famous theorem proved that this was false,

Theorem: For any boolean function in , the length of a bounded width program for that function is polynomial.

The connection between bounded width programs and protocols is simple. Suppose that there are boxes, which we wish to show is too few to compute some function. Divide the boxes into three groups: , and . The boxes in are the first boxes, the boxes in are the next boxes, and are the last —I fixed a typo here thanks to Andy D. Think of three players , , and . They each do not have enough bits to determine the function if it depends on all the bits. Yet any two players together have enough bits, possibly, to determine the function: this follows because,

Thus, we needed to somehow argue that they needed to move more than a fixed number of bits from one to another to compute the given function. This was the hard part.

IBM Yorktown is in the middle of nowhere. Well it is in the middle of a beautiful area, but then there were no lunch places nearby. So every day we ate in the IBM cafeteria, which was a fine place to eat. However, one day Merrick and I decided to take off and have lunch somewhere else. We had a long lunch “hour.” A really long hour. When we got back to IBM Ashok was a bit upset: where have we been? Luckily a theory talk was just about to start and we all went to hear the talk.

Talks are a great place to think. There is no interruption, which was especially true back then since there were no wireless devices. I used the time to try to put together all the pieces that we had discovered. Suddenly I saw a new idea that might be the step we were missing.

As soon as the talk was over I outlined the idea that I had to Ashok and Merrick. They were excited, and to my relief all was forgotten about the long lunch. Working that afternoon we soon had a complete outline of the proof. There were, as usual, lots of details to work out, but the proof would eventually all work. I still think we made progress on this result because of the long lunch and the quiet time of the talk. The result appeared in STOC 1983.

We were excited that we had a lower bound, but we missed two boats. Missing one is pretty bad, but two was tough. The first was our protocol lower bound : this was later vastly improved by much better technology, and the second was that we missed thinking about upper bounds. I do not know if we would have found Barrington’s beautiful result; however, we never even thought about upper bounds. Oh well.

Concentration Results

Let’s jump forward to the present. There is a terrific book, published this year, by Devdatt Dubhashi and Alessandro Panconesi called Concentration of Measure for the Analysis of Randomized Algorithms. I just got a copy and it is beautifully written, contains all the major concentration results, and is a must to have on your desk.

Of course research stops for no one, and there are two beautiful papers that contain new concentration theorems that will appear in the next FOCS. I am sure they will be included in the second edition of Dubhashi and Panconesi.

The first paper is: A Probability inequality using typical moments and Concentration Results by Ravi Kannan. The second paper is: A Probabilistic Inequality with Applications to Threshold Direct-product Theorems by Falk Unger.

The theme of both these papers, and the book, is to prove theorems that show that a sum of random variables is likely to be near its mean. More precisely, if

are random variables, a concentration theorem tries to get sharp bounds on

In general a sum of random variables will not have concentration unless there is some additional property. To see this just consider the case where they all are if a fair coin is heads and otherwise. Clearly, this sum is or and does not satisfy any meaningful concentration theorem.

One approach to get a concentration theorem is to assume that the random variables are independent, but this is often too restrictive for many applications. Look at their book for the myriad number of ways to get concentration theorems.

Another approach is to assume that the random variables satisfy certain constraints on the conditional expectations. This is what Ravi is able to improve, since the usual theorems assume “worst-case” bounds on these conditional expectations. Ravi can weaken these to “average” bounds. I think this is a brilliant insight—one that could have far reaching consequences. For example, Ravi can prove: (It is a corollary in his paper.)

Theorem: Suppose that are real-valued random variables satisfying the SNC condition. Let . Suppose that is a positive real with

for all . There is an absolute positive constant such that

where .

See his full paper for what SNC is and the role of in that condition.

Multi-Party Protocols Again

Unger’s concentration paper is quite interesting also. I will not be able to do it justice so again please take a look at the full paper. His result has to do with results that are quite close to multi-party protocols. In particular, he has an explicit application to this area.

His theorem on protocols is quite technical, but the flavor is an amplification type result. Suppose that players can compute some boolean function with probability . Then, he can bound how well they can compute multiple independent copies

where the inputs are all independent uniform random bits. Then, he can get exponential bounds on how well a protocol can do even if it only has to get a majority of the functions right.

Theorem: Let be a function such that no -party protocol which also uses bits of communication can compute better than with probability , for , when the inputs are chosen uniformly at random. Chose such that and . Then for any -party protocol with uniformly random inputs and output bits, which uses bits of communication, it holds: The probability that the output of agrees with on at least positions is at most

Here the term is order times certain other terms. The point is that their protocol cannot easily compute the function multiple times without using many bits of communication.

Open Problems

One open question is to find additional applications of these new concentration theorems. Can the average idea of Kannan work in other situations? Are there further concentration theorems yet to be discovered?

]]> http://rjlipton.wordpress.com/2009/10/15/multi-party-protocols-and-focs-2009/feed/ 5 lipton images http://rjlipton.wordpress.com/2009/10/11/magical-results-and-pnp/ http://rjlipton.wordpress.com/2009/10/11/magical-results-and-pnp/#comments Sun, 11 Oct 2009 20:03:47 +0000 rjlipton http://rjlipton.wordpress.com/?p=3678 ]]>

Which is more likely: intractable problems or magical results?

Martin Kruskal was a famous mathematician who won many awards for his work in diverse areas of applied mathematics. He is perhaps best known, among his peers, for his important joint work with Norman Zabusky; work which helped start the soliton revolution. He is also known among magicians, for a brilliant illusion that is called the “Kruskal Count.” See this memorial page about Kruskal for a list of his accomplishments.

Today I want to talk about some other magical aspects of mathematics. One of the great things about mathematics is that often there are results that seem to be “impossible.” Magical.

I was at Princeton’s Computer Science department at the same time that Kruskal was in their Mathematics department. I still remember making a “small” contribution to Princeton’s attempt to lure Michael Rabin away from Harvard. As is often the case when recruiting a senior super-star like Rabin, they had a group dinner at a quite good local Chinese restaurant. Since I knew Michael well I was invited to join them.

We had our own private room at the restaurant. It was a lovely meal: delicious food and great conversation. When the dinner ended, after we all had read our fortune cookies, Michael and others stood up and began to say their goodnights to each other.

I was still sitting next to Martin, who was in charge of our dinner. He proceeded to pull out a hand-calculator, and started to figure out what we each owed as our share of the bill. I was a bit surprised, since in computer science the tradition was to have the department pay for the whole dinner—after all it was an official recruiting function. But each department has its own rules.

Martin started to calculate, after tip, what each of the group of twelve owed for their dinner. I started to get my wallet out so I could pay my share, when I realized that Martin had included Rabin as one who would be paying. I gently suggested to Martin that perhaps we could treat our “guest” and divide not by 12 but by 11. Martin liked this idea very much, and re-calculated the new figure that we all owed.

Rabin never did accept an offer to come to Princeton, but I like to think that the switch to dividing by 11 instead of 12 could not have hurt.

Kruskal’s Count

Here is the trick, or the illusion as my magician friends prefer to say.

The trick uses an ordinary deck of cards. Each card has a value: an ace is , all picture cards are , and the rest have their face value. Thus, a king of diamonds is and a three of spades is . The deck is shuffled, really shuffled. No trick there.

You are the magician and let Paul be the person you are trying to impress with your mind reading abilities. You tell Paul to pick a secret number that is between and and to not tell you the number. Then, as you deal the cards, one by one, you ask Paul to do a simple count: For example, if his secret number was , then as you show the cards Paul should count to and see what the value of the second card is. That is his new secret number. He then does the same thing as the next cards are revealed. At all times he is counting and replacing his secret with a new secret. Paul does all of this in his head, keeping his secrets to himself.

When the deck runs out, you think hard. Then, you announce the secret that is in Paul’s head. Amazing.

How It Works

The critical part of the illusion is that you cannot guess what Paul’s original number was. That would be real mind-reading. If you can do that, then you should head directly to Las Vegas and play high-stakes poker.

The key is that you, the magician, pick a number , and do the same process as Paul does. The brilliant insight of Kruskal is that there is a good chance that at some point you and Paul will select the exact same card. Once this happens, then you and Paul are in synch and will continue to remain in synch. So at the end you just announce what your secret is and it will be the same as Paul’s with about probability. Very neat.

Magical Results

There are many other magical results throughout theory. I plan a longer post on a list of my favorite ones. For now I want to discuss one of the great discoveries of Computer Science, that is one can turn “lemons into lemonade.” If there are hard functions, “the lemons,” that cannot be computed efficiently, then magical things can happen, the “lemonade.” Turning lemons into lemonade is one of the key insights of modern cryptography.

I will not even begin to attempt to give a survey of the many great results in this area, but I do have a point that is related to my favorite question: “can P=NP be true?”

Close you eyes for a moment, and then take a deep breath. I am going to ask you to imagine two different scenarios:

In the first one, suppose Alicex is an alien from Mars and has just arrived on Earth. She knows some mathematics, but no modern complexity theory or cryptography—Mars has not yet been able to develop it. She is told that the following two things are possible here on Earth:

1. She can send a perfectly secret message back to her friends on Mars; even though Alicex has no secret bits in common with her friends back there. On Mars they have one-time pads, but nothing better for sending secret messages. Alicex is surprised.
2. She is also told that we, I mean Andrew Wiles, has been able to prove Fermat’s Last Theorem:

   

   with implies that . Of course Alicex calls this problem something else: the “Gozzetx Problem.” It is named after the famous Martian mathematician Fredx Gozzetx, whose elegant proof that every planar graph has a four coloring was one of the great achievements of early Martian mathematics. Every Martian high school student knows his famous proof.

   This is a welcome news. But then Alicex is told that she can convince her friends back home on Mars that she knows a proof of this great theorem. And she can do this without giving away the actual proof. This is very neat, since there is a prize in Martian money for a proof. She trusts her friends, but that’s a lot of money; even with their recent inflation is still enough to buy what we call a mocha latte. Alicex is surprised.

In the second one, suppose that Bobx is also an alien from Mars and has just arrived on Earth. Bobx like Alicex knows some mathematics, but no modern complexity theory or cryptography. We tell Bobx that the following is possible here on Earth: there is an algorithm that can factor any integer in polynomial time. Bobx is surprised.

Open Problems

Which alien is more surprised? Alicex or Bobx?

One of the reasons I have doubts about PNP is simple. I often wonder which is more surprising? That there could be an intelligent algorithm to solve a “hard” problem like factoring, or that there are magical results of modern cryptography? This is one reason I find the P=NP question so puzzling. Why is it so easy to believe that there is no clever algorithm for SAT, but that it is possible to do:

• Public-Key encryption?
• Signatures?
• Zero-knowledge proofs?
• And on and on?

These seems like magic to me. What do you think?

]]> http://rjlipton.wordpress.com/2009/10/11/magical-results-and-pnp/feed/ 25 lipton kruskal http://rjlipton.wordpress.com/2009/10/07/the-surprising-power-of-rational-functions/ http://rjlipton.wordpress.com/2009/10/07/the-surprising-power-of-rational-functions/#comments Thu, 08 Oct 2009 02:25:47 +0000 rjlipton http://rjlipton.wordpress.com/?p=3655 ]]>

Newman’s theorem on rational approximations and complexity theory

Donald Newman was not a theorist, but was a mathematician who worked on many topics during his career. One of his results is a lovely theorem that shows that the approximation of continuous functions by rational functions can be very different from the approximation by polynomials.

Today I want to talk about this famous theorem, and discuss some applications to theory. If you do not know this theorem you may enjoy seeing it. Even if you do know it, you may enjoy seeing some thoughts I have on a potential application of this theorem.

I never had the honor of meeting Newman, but I just read a nice tribute to him, while preparing this, which contained a number of interesting stories. One explained how he once introduced the great Paul Erdös to a seminar audience in a “novel” way—you will have to read it yourself. Another was on Newman’s performance on the Putnam mathematics competition—he was clearly a very fast and powerful problem solver. I quote:

“Ordinarily, six or seven solutions are sufficient for a student to win the competition. In his freshman year at CCNY, Newman won the Putnam with nine correct solutions. As a sophomore, he again won the competition with an unprecedented perfect 12 solutions, a fete (sic) he repeated in his junior year. He didn’t bother taking the exam as a senior.”

I recall taking the Putnam exam as a freshman—I did okay—nothing like Newman. I did worse as a sophomore, even worse as a junior, and finally was thrown off the team as a senior. For me, the more mathematics I learned the worse I got. Oh well.

I will not discuss whether or not the “world is digital or not.” I think the last post generated a lot of fun comments—many at my expense. I make mistakes and perhaps that post was one. If so I apologize to you all; I hope that the many comments, while most disagreed with me, were interesting to you. I somehow did not get my point across. So I will fold up my tent on that problem for a bit—no pun intended—and move on to Newman’s classic theorem. Wait, I now see how to explain my point stop, I better not go there today.

Newman’s Theorem

It has long been known that any continuous function on the interval , or any compact interval will do, can be approximated arbitrarily well by a polynomial. There are many proofs of this fundamental fact, many refinements, and many generalizations.

The function , the absolute value of , while continuous on the interval is not easy to approximate by a polynomial. To get to an error bound of roughly requires a polynomial of degree . The reason, at least the intuition, is that the function has a sharp corner: at zero its derivative is undefined. This behavior makes it hard for a low degree polynomial to “fit” the absolute value function.

What Newman did in 1962—the paper was published in 1964—is to prove that one can do exponentially better with rational functions instead of polynomials in approximating the absolute value function. I am not an expert in approximation theory, but I believe that this result came as a surprise. See Herbert Stahl’s paper, for instance. Another surprise in mathematics.

Let’s now turn to Newman’s Theorem. We need some notation first. Let and . Then,

Theorem: For all and ,

where is the rational function,

The theorem is remarkable in a number of respects: First, it shows that the degree of the approximating rational function can be much smaller than the degree of any approximating polynomial. Previously, to get an error bound of order a polynomial had to have degree order ; now the degrees of the numerator and denominator are only order . This is a huge decrease. Second, the rational function is not just constructive, but is expressible in a simple closed form. Very neat.

Both of these are wonderful for applications—as we will discuss in the next section.

Applications

I knew the Newman theorem back when I was a graduate student, yet I never could find an application of it to some theory problem. The theorem seemed to be perfect: great error bound and constructive. Yet I never found any “use” of his theorem.

It took a great insight from Richard Beigel, Nick Reingold, and Daniel Spielman to realize that Newman’s theorem could be used to solve an important open problem from complexity theory. They showed in a brilliant paper that the complexity class PP is closed under intersection and union. This is one of the great proofs in complexity theory. It is very sad that Reingold passed away about a year ago.

Later Mitsunori Ogihara used similar ideas to prove that the PL “hierarchy” collapses. There are now other uses of Newman’s theorem: for example, it is used in parts of learning theory. However, I will not attempt an exhaustive survey of its applications.

The PP result solved a well known open problem that had been raised as soon as PP was defined as a complexity class. Recall PP is the complexity class that contains sets that are accepted by a polynomial-time bounded probabilistic Turing machine that accepts with probability . It trivial that this class is closed under complements, but closure under intersection/union is not obvious at all.

The PP Proof

Here is a high level view of their proof. They need to detect the sign of an integer: the sign of is and the sign of is . The problem is that they need to do this with a low degree polynomial, which is not obvious how to do it. Newman’s theorem comes to the rescue.

Suppose that there is a rational function that approximates well, which of course there is by Newman’s theorem. Then, for not too near , must well approximate the sign of . The problem is that their Turing machine can “compute” polynomials, but not rational functions—they cannot divide in their model. So they use the following cool, but trivial, trick:

There is much more to their proof, but this is the key use of Newman’s theorem. They do need to modify his function to one that works over the integers in an exponential size range. Here are the polynomials they use; note, as is often the case in computer science not only degree but the size of coefficients is also important. In particular they use the following polynomials:

Some of the key properties are:

1. The degree of is ;
2. Each coefficient of is ;
3. If are integers in , then ;
4. If are integers in , then , , and .

Matrices

I have several open problems that concern the behavior of matrices. I noticed that the following simple theorem can be proved using Newman’s theorem. This is likely to be known, but I will present the statement here with an outline of the proof.

Lemma: Suppose that for all in the interval , where and are polynomials. Suppose also that is an by matrix with all its eigenvalues in . Then,

Recall, is the sum of the diagonal entries of the matrix .

What is lemma says is: if an inequality holds for an interval that contains the eigenvalues of a matrix, then a related inequality holds for the traces of the matrices.

The proof in the case that is diagonalizable is fairly straightforward. Suppose that where is a diagonal matrix. Then, the key insight is that

The last equality follows from a basic property of the trace of a matrix.

The lemma then follows by applying the inequality on and to each entry of the diagonal matrix ; the bound is clearly times .

The proof in the case that is not diagonalizable follows by a trick that I believe is due to Richard Bellman. His insight is that given any there is a that is diagonalizable and is within of . Then, one lets tend to and concludes the result by continuity. This trick, for example, gives a very short proof of the fact that any matrix satisfies its own characteristic polynomial. First prove this for diagonalizable matrices, which is simple, and then use the Bellman trick.

Theorem: Suppose that is an by matrix with all its eigenvalues in the interval , and with . Let be the rational function from Newman’s theorem. Then,

where is the sum of the absolute values of the eigenvalues of .

The proof follows from the lemma and Newman’s theorem using and the absolute value function for .

Open Problems

An obvious open problem is to find other applications of Newman’s theorem. Can you use the matrix version? I had an idea to use the matrix version on a learning problem, but that has not worked out yet. So I will get back to that in the future.

]]> http://rjlipton.wordpress.com/2009/10/07/the-surprising-power-of-rational-functions/feed/ 10 lipton images http://rjlipton.wordpress.com/2009/10/04/the-world-is-digital/ http://rjlipton.wordpress.com/2009/10/04/the-world-is-digital/#comments Mon, 05 Oct 2009 01:29:57 +0000 rjlipton http://rjlipton.wordpress.com/?p=3585 ]]>

One of the biggest insights of computer science is that the world is digital

Alan Kay is a brilliant computer scientist who has won the Turing Award, along with many other honors. He is famous for creating the Dynabook concept—the idea that became the laptop, and for creating many programming languages, such as Smalltalk.

Today I want to talk about one of the biggest results in all of Computer Science. No, it is not the Halting Problem, it is not the P=NP question, it is not that Linear Programming is in P, nor any of the many other beautiful results of theory. It is, in my opinion, the realization that the world is digital (WID).

This summer I was at an NSF workshop on Computational Thinking—an idea that Jeannette Wing is behind. During the workshop I said:

In my opinion, one of the key insights in all of Computer Science, is that the world is digital.

I also added that this was a relatively recent result, and it has had a tremendous impact on the world.

As soon as I said this, Bill Wulf and Alan Kay both jumped all over me, and continued their comments into the next coffee break. They must have thought that I was crazy or worse—perhaps they still do. Their position was that WID had been known forever—back to the Greeks—therefore it was not new, nor was it a deep insight. I disagreed. It’s hard to argue with the past president of the National Academy of Engineering, Bill; and with a Turing Award winner, Alan. But I tried. I did not get them to budge, not at all.

I still think I am right. I will explain why and I hope that you might agree with me, but either way let me know what you think.

The World is Digital

The more original a discovery, the more obvious it seems afterwards—Arthur Koestler, journalist, novelist, social philosopher, and science writer.

The fact that all things can be represented by sequences of zeroes and ones sounds like a simple idea, but I claim that it is deep and profound.

Look around you. You may see, like I do, beautiful trees outside in the backyard. You may hear wonderful sounds: birds chirping; music playing classical or rock or hip hop or another genre. You may be reading a book or watching a show on a HD-TV and seeing images of people that are almost lifelike.

Yet no matter how beautiful the images are, no matter how complex the sounds are, they all can be reduced to sequences of bits. The world truly is digital. In a sense, 0 and 1 represent computer science’s basic atomic particles. Information consists of arrangements of 0’s and 1’s. Nothing else is needed. Computer science differs from modern physics, while Physics uses hundreds of “basic particles” to build the world—we only need two.

I believe that the discovery that all things can be viewed as digital is as revolutionary as the notion that the world is made of atomic particles. But, the discovery that the WID took half a century longer to discover than even the simplest atomic particles.

Electrons, for example, were discovered by Joseph Thomson in 1897, long before we knew that the WID. Some may scoff and say that it’s obvious that the WID. But, I think I can argue that is not so. Yes, particle physics is a deep theory; however, Computer Science’s insight that everything can be represented digitally is profound.

Who Knew What When?

One way to argue, I believe, that WID is profound is that it is a recent discovery. I believe that if it was “obvious,” then it would have been known for a long time.

The Ancient Greeks: Greats like Euclid did not know the WID. In all of his famous Elements, there is not one section on boolean functions.

Yet, Alan and Bill said to me that the ancient Greeks already knew that the WID. Of course, the Greeks knew that the square root of was not a rational number. Somehow, Alan and Bill felt that this showed that they knew that the WID. I think the Greeks knew many wonderful things but that they were in the dark with respect to the WID—at least that is my opinion.

A kind of atomic theory was known, without proof, to the Greeks. That the world is made of discrete atoms is not the same I would argue as the WID. They did not make the key jump: that is they did not realize that any object of any kind could be represented as a binary string. It seems obvious to us today, but it was not centuries ago.

The Classic Mathematicians: The great mathematicians such as Augustin Cauchy, Joseph Fourier, Carl Gauss, David Hilbert, Pierre-Simon Laplace, and Isaac Newton did not know the WID. They mainly studied continuous systems, and when they did work on discrete systems, they worked with the integers. But I do not believe they ever thought about boolean functions, and nor
do I think that they knew the WID.

Lewis Carroll: Greats like Lewis Carroll—Charles Dodgson—did not know that the WID. While he is famous for his Alice In Wonderland series of books, during his later years he spent a great deal of time working on symbolic logic. He was especially interested in the theory of syllogisms: a set of premises that lead to a conclusion. Here is one of his examples:

Babies are illogical
Nobody is despised who can manage a crocodile
Illogical persons are despised

He then concluded that

Babies cannot manage crocodiles

Lewis Carroll used a version of Venn diagrams to help his readers understand his logical examples; these diagrams are named after the mathematician John Venn who introduced them in 1880. Carroll worked out a method of squares within squares that worked up to letters—where a letter is a “proposition.”

What I find interesting is that Carroll did not see that he was working with boolean logic. His examples and notation are fairly complex, and show, I believe, again that the central nature of bits and boolean values was not understood by him nor his contemporaries.

Gödel: Greats like Kurt Gödel did not know the WID. If he had, he would not have needed to invent his elaborate scheme for numbering proofs. If he had realized that proofs were only a finite collection of symbols, he could have assigned each symbol a unique binary string. That’s all he needed to do: no primes, no fancy facts about primes, just binary strings. Indeed, I never present his complex numbering system when I teach Gödel’s Theorem; I just explain that proofs and formulas are binary strings.

Markov: Greats like the Russian mathematician Andrey Markov did not know the WID. He invented a model of computation that he called “normal algorithms.” Unlike Turing Machines, his model was based on string rewriting rules. I remember reading his 1950 book on his theory of algorithms. One of the striking parts of the book, to me, was the introduction. His model needed to restrict the set of symbols to a finite set of symbols—otherwise, his rewriting rules would not be finite. Today, since we know that the WID this is obviously no issue. However, when he wrote his book he was concerned about the restriction to a finite alphabet. He talked. at length, about the infinite number of ways that one might be able to write a script letter such an “a”:

But, he reasoned that while there might be an infinite number of script letter “a”s, the restriction to a fixed size alphabet was just fine.

That Markov felt, in 1950, that he had to discuss the restriction to a finite alphabet shows me that he was unaware that the WID. Today if I had to argue why a finite alphabet suffices to describe any script letter, I would point out that we can just represent the letter as a digital image. Let the image be by pixels, where is some huge number—if is large enough I would argue that if a human can tell two script letters apart, then they would have different digital representations.

How I Came to Know the World is Digital

I remember when it finally began to dawn on me that the world is digital. I would like to say that I always knew, or at least that it came to me in a flash one day. But that would be a lie. It took me years to realize it. It’s one of those ideas or notions that came slowly to me. However, once I saw it, I wondered how I could have not always known it.

At Carnegie Mellon, where I got my Ph.D., we were never taught that the WID. My professors, like Albert Meyer and Alan Perlis, never told me. Meyer was then, and is now a brilliant researcher—he never told me the WID. Perlis was, at the time, the chair of the entire department of Computer Science, yet he never told me either.

When I was an assistant professor of Computer Science at Yale University in 1976, I got my first inkling that the world could be viewed as digital. I was on the phone with a friend, Bob Sedgewick, who at the time was an assistant professor at Brown University. He was telling me how a newly purchased printer worked.

Now you have to go back 30 years to recall that printers were once very different from the beautiful laser/ink jet printers of today. In those days a printer was a typewriter on steroids. In case you have never used one, here is how a typewriter worked. There was a keyboard like today’s laptops. But when you pressed a key, a piece of metal hit an ink ribbon and then hit your paper. In this way each letter was printed.

Printers did the same—again each letter had a piece of metal that was pressed onto the paper. The user was limited to the font and symbols which were built into the printer. Some printers had clever arrangements, so the printer could go fast. Some had the metal letters arranged on wheels; others had them arranged in even more complex configurations. In any event, typewriters and printers used the impact of a piece of metal onto an ink ribbon to leave a mark on paper. The only difference between printers and typewriters was speed.

Back to my story: Bob was telling me about a new kind of printer that used dots to represent each letter or symbol on the page. He explained how the page was coded as an array of 0’s and 1’s. Where a 1 occurred, the printer placed ink on the page; a 0 instructed the printer to leave a blank on the spot. The way the ink was placed onto the paper was not laser-based or ink-jet based by the way. This early printer used a more primitive technology; however, the page was represented as an array of bits—it was digital.

I have to say, I was surprised. I knew about Gödel numbers. I knew about binary numbers. I knew a lot of computer science—I thought. However, I somehow was surprised that the printer was just placing bits onto a piece of paper. I did not see that printed words and images could be represented solely by dots. Really.

What Bob explained was that he had written a program that created an array of 0’s and 1’s. Then, his program sent the array to the printer which created a printed page. He explained, further, that the program could create text or images or anything in this manner. This was a major shift from pieces of metal striking a ribbon and putting ink onto a page.

Typewriters and printers had been limited to print only the images that were built into the metal keys or balls. Now with the digital approach of using bits as a way to control the printing process, a whole new vista opened up. Any picture, any image, any font could, in principle, be represented, or printed, without changing the hardware.

This was a huge breakthrough.

The Chicken and The Egg Argument

I always show these posts to Subruk before I post them; otherwise, they would be a mess. This time he raised an interesting point that we decided that he should make—that I would not change my thoughts to conform to his insight. I am doing this partly because I do not completely agree with his point, and partly because I think it is a great insight that he should make himself. The plan is that he will add a comment to this once it is out there for you to read.

I would, however, like to address an issue that is related to his point, but please read his comment.

Mathematics is driven by practice and also drives practice. There are many examples of deep mathematics that was created to solve very practical problems. For example, the theory of Fourier Series was driven by the need to solve certain differential equations. Many other examples come to mind: geometry—construction of temples, calculus–astronomy, probability theory—life insurance tables, and on and on.

On the other hand, many times mathematics has been ahead of practice. Godfrey Hardy was actually proud that his favorite area, number theory, was pure—that it had no possible applications. Of course we can only speculate what he would think of modern cryptography’s use of number theory as a basis of encryption. Other areas of mathematics were also developed because they were beautiful. Many examples come to mind: abstract measure theory, set theory, algebraic geometry, infinite dimensional geometry—I am sure the list could go on and on. Sometimes these areas have eventually been used to solve real problems, and sometimes they remain beautiful areas that seem to be without any practical application.

My point is simple. Even before computers existed mathematicians could have worked on binary strings, on boolean functions, and on many of the areas that we work on today. They did not, in my opinion, because they did not understand the power of bits; they did not know that the WID.

A final point. If they had worked on boolean functions years ago, one wonders where theory would be today? Curiously, the difficult foundational issues that plagued early analysis, for example, would have been completely moot. Early mathematicians struggled with the basic question of “what is a real number?” If they had worked on boolean problems, these difficult problems would have been completely avoided.

Open Problems

Is the “world is digital” one of the great insights of Computer Science? Or do you agree with Kay and Wulf?

]]> http://rjlipton.wordpress.com/2009/10/04/the-world-is-digital/feed/ 65 lipton images pic