Книга: Coders at Work: Reflections on the craft of programming

Peter Norvig

Peter Norvig

Peter Norvig is a broad thinker and a hacker at heart. He once wrote a program to find in Google’s search logs series of three consecutive searches by the same user that, when put together, made a haiku (one of the most memorable: “java ECC / java elliptical curve / playboy faq”).

On his web site Norvig has links to the usual stuff: books and papers he’s written, slides from talks he’s given, and various bits of his code. But there are also links to items he’s had published in McSweeney’s Quarterly Concern, his witty recounting of writing a program to generate the world’s longest palindromic sentence, and his “Gettysburg Powerpoint Presentation,” a send-up of Microsoft’s PowerPoint software, which has been cited by Edward Tufte and which appears on the first page of results if you Google “PowerPoint.”

He is now the Director of Research at Google, after having been the Director of Search Quality. Prior to that he had been the head of the Computational Sciences Division at NASA Ames Research Center and before that, an early employee at the late-’90s Internet startup Junglee. He won the NASA Exceptional Achievement Award in 2001 and is a Fellow of the American Association for Artificial Intelligence and the Association for Computing Machinery.

Between Google, NASA, and Junglee, Norvig has experience with both the “hacker” and “engineer” approaches to building software and talks in this interview about the advantages and disadvantages of each. As a former computer science professor and now an insider at one of the biggest industrial software shops in the world, he also has an interesting perspective on the relation between academic computer science and industrial practice.

Other topics in our conversation included how programming has changed in recent years, why no design technique can make up for not knowing what you’re doing, and why NASA might be better off with less-reliable but cheaper software.

Seibel: When did you start programming?

Norvig: In high school. We had a PDP-8, I think it was, at my high school, and there was a class I took—we started in BASIC programming and I went from there.

Seibel: What year would that have been?

Norvig: I graduated high school in ’74, so it must have been ’72 or ’73. I remember a couple of things, going back to those early days. I remember the teacher trying to teach shuffling of a deck of cards. Her algorithm was, use a random-number generator to pick two locations and then swap them and keep a bit vector that said, these were swapped, and keep going until they’re all swapped. I remember my reaction being, “That’s stupid. That’s gotta be the stupidest thing in the world. It could take forever because there could be one pair that you never happen to choose.” I didn’t know enough then to say it’s n squared when it could have been order n. But I knew that that was just wrong. Then I was able to come up with, I think, the Knuth algorithm of swapping from 0 to 52 and then from 0 to 51 and so on—an order n algorithm. And I remember the teacher defending her approach. That helped me think, “Well, maybe I have an aptitude for this programming stuff.” It also helped me say, “Maybe teachers don’t really know everything.”

Seibel: Was it as soon as she described it that you just said, “Wow, this is wrong”? Or did you play with it for a while and then say, “Gosh it seems like we’re doing a lot of work here”?

Norvig: I think I noticed right away. It’s hard to know what I was really thinking back then but I think right away I noticed there’s a finite possibility that this might not terminate. I’m not sure I knew as much about the expected runtime.

I also remember finding my father’s back issues of Scientific American in the attic and going through them. There was this article by Christopher Strachey on software engineering in which he said that people are going to use these higher-order languages. And he had invented this language that there was never a compiler for—it was a paper language. And he said, “I’m going to write a checkers program using this language.” I remember reading that—it was the first nontrivial program I had ever read because in school we were just learning how to shuffle and so on. I read it again recently and the first thing I noticed was that there’s a bug in it. And it was great because you figure this is Christopher Strachey, he should know what he’s doing, and it’s Scientific American, they’ve got editors and so on—they should probably get those bugs out. But in the prose it says there’s a function makemove which takes a board position and returns a move and then you look in the code and there’s make-move and it takes a board position and an extra parameter. Apparently they wrote the prose first and then they wrote the implementation. And they found out you can’t search infinitely deep so they added an extra parameter which was depth of search and you recurse down to a certain level and then you stop. They had added that in afterwards and hadn’t gone back and fixed the documentation.

Seibel: So that was the first interesting code you read; what was the first interesting program you wrote?

Norvig: I guess it was probably the Game of Life. It was actually an assignment for the class. I quickly did the assignment, and of course then we didn’t have the nice display screens. I didn’t have my 30-inch monitor—I had the teletype with the yellow paper. I said, “This is a waste, printing out one small field”—they probably wanted us to do a ten-by-ten field—“and then printing out the next one and the next one.” So I said, “Let’s print out five generations in a row.” And I remember that you couldn’t have threedimensional arrays in BASIC, and for some reason I couldn’t even have a bunch of two-dimensional arrays. It’s like they ran out of memory or something. So I had to figure out how am I going to have five or six across of these two-dimensional arrays, and that’s when I discovered bit fields.

Seibel: So given the constraints on memory, you rolled your own storage for that much data. Had you been taught about bit arrays and figured out how to apply them, or did you go digging through the manual and discovered, “Oh, look, here’s this PEEK and POKE” or whatever it was?

Norvig: Well, I was storing a zero or one in each of these locations, and I needed to store more stuff somewhere, and I said, “Oh, store other numbers there.” In fact, I don’t even remember if I did bit store. I might’ve done digits—decimal rather than binary—because the binary stuff hadn’t really been exposed to us in an interesting way. And then I got to add in things, like, is it repeating and if so, with what cycle. You couldn’t do that when you were just keeping one previous generation.

Seibel: When you were coming up as a programmer, did you do things specifically to improve your skill as a programmer, or did you just program?

Norvig: I think I just programmed. Certainly I would do things because they were fun. Especially when I was a grad student and I was less beholden to schedules. I’d say, “Oh, here’s an interesting problem. Let’s see if I can solve that.” Not because it’s progress on my thesis, but just because it was fun.

Seibel: And you studied computers in college but didn’t major in computer science, right?

Norvig: When I started, the computer classes were part of the appliedmath department. By the time I graduated there actually was a computer science department, but I stuck with math as my major. It felt like doing all the requirements for a computer-science major was like majoring in IBM. You had to learn their assembly language, you had to learn their 360 operating system, and so on. That didn’t seem like much fun. There were some courses that I liked and I took those, but I didn’t want to go through all the requirements.

After college I worked for two years, for a software company in Cambridge. And after two years I said, “It took me four years to get sick of school and only two years to get sick of work, maybe I like school twice as much.”

Seibel: What were you doing for them?

Norvig: Their main product was a software-design tool set, and they also did software consulting of various kinds. The founders had worked at Draper Labs in Cambridge on the Apollo mission and other things like that; they had Air Force connections and they were a government contractor. They had this idea of how software should be designed. I never believed in the whole idea, but it was fun.

I remember one of the projects we had there, at this company, was to write a flowchart drawer. The idea was that it would parse your program and generate a flowchart for it. Which was perfect because that’s the way people always use flowcharts. You’re supposed to write them ahead of time but you really never do—you write them after the fact. And it was clever in that it had a sort of partial grammar so you could take a program that wasn’t even quite syntactically correct and it would gloss over the parts that it couldn’t parse. It would have to know how to parse the IF statements because those made different blocks and so on but the other stuff it just said, “Well, whatever is there just throw it into a block.” We got this contract to generate this thing and they specified that they wanted to run it on a Unix system. So we borrowed a machine at MIT and used all the Unix tools, yacc and stuff, for the compiler. And at the last minute they said, “No, we’re going to install it on a VMS system.” So all of the sudden, yacc wasn’t there. But we said, “That’s OK, we don’t need it—we just needed it to generate the tables and that’s done.”

Seibel: As long as your grammar never changes, you’re OK.

Norvig: Right, and so we delivered it and they were happy and then—of course—the grammar changed. And we didn’t have access to any Unix machines anymore. So I ended up having to patch the grammar by understanding the tables and saying, “Here’s a jump to this other state—OK, I’ll invent a new state here and jump to that one instead.”

Seibel: And was that really the right solution—did you ever consider just writing a new parser?

Norvig: I probably should have. But, you know, it was just this one little fix.

Seibel: And you didn’t get caught in the trap of every three weeks they come around with a new change to the grammar?

Norvig: Well, then I went off to grad school. Somebody else had the problem and I don’t know what happened.

Seibel: Not your problem anymore. So you got your PhD. Is there anything that you wish you had done differently about how you learned to program?

Norvig: I ended up in industrial settings, so maybe I would’ve wished to have done more of that earlier on. I did learn to do it, but I was in school and in grad school for a long time. That was a lot of fun, too, so I don’t have any regrets.

Seibel: What were the things you had to learn about industrial programming?

Norvig: About having schedules, and keeping team members and customers and managers happy. When you’re a grad student, you don’t have to do that; just show up to your adviser every now and then.

I guess the biggest change was going from one person to a team and figuring out how those kinds of interactions work. That’s something you don’t normally get in school. I guess some of the schools are starting to bring that into the curriculum more. When I was in school, working as a team was called cheating.

Seibel: For people who are going into industry, are there other skills, beyond just the ability to write code, that people should develop?

Norvig: Getting along with people is the main thing. Being able to understand the customer is important: to know what it is you want to build and whether what you have is right. Being able to interact with them and then interact with your teammates. And interact with people higher up in the company and your customers when you go out and see them. There are all different social relations and they require different skills.

Seibel: Has programming become a more social activity than it used to be?

Norvig: I think so. I think the computer used to be more segregated. And in the old days it was mostly batch processing, so the interface was so much simpler. It was possible to do this kind of waterfall design where you said, “The input is going to be this deck of cards, and the output is going to be a report that has this number in this column.”

It probably wasn’t a very good idea to specify it that way. Probably right from the start you should have had more interaction with the customer. But it seemed more separable. Now everything seems more fluid and interactive so it makes more sense to say, “Rather than have a complete specification from the start, just get customers in the room and start brainstorming.”

Seibel: And do you remember any particular aha! moments where you noticed the difference between working on something by yourself and working on a team?

Norvig: I don’t know if it was so much moments, but just this realization that you can’t do everything yourself. I think a lot of programming is being able to keep as much as you can inside your head, but that only goes so far, at least in my head. Then you have to rely on other people to have the right abstractions so that you can use what they have. I started thinking about it in terms of, “How is this likely done?” rather than, “I know how this was done because I did it.” If I were to have done this, how would I have done it? I hope that it’s like that, and if it’s not, figure out why not, and then figure out how to use it.

Seibel: Do you think that learning to work on teams that way also enables you to actually work on bigger things even by yourself when you’re sort of a team spread across time?

Norvig: I think that’s true and that’s certainly something I see in the younger programmers that are coming out now. Another difference between now and then is it seems like it’s much more assembling pieces now rather than writing everything from scratch. Now, for a school assignment, someone says, “OK, I needed a website, so I used Ruby on Rails for this, and I used Drupal for that part, and then I had this Python script, and then I downloaded this statistical routine,” and it’s all scripting to put together these pieces rather than writing everything from scratch. So I think understanding interfaces and how they go together is more important than all the details of the insides of these packages.

Seibel: And do you think that changes the kind of people who can be successful at programming now?

Norvig: I think the people that are really successful are the same—at least that’s what I see around here. But, yeah, it is a little bit more of, “Can I quickly get an understanding of what I need,” and less of, “I need complete understanding.” I think some of it is bravado, this willingness to say, “I’m just going to go ahead and do it,” the fearlessness of saying, “I don’t understand everything that’s going on, but I went into the documentation and I learned these three things. I tried it and it worked, so I’m just going to go ahead.” That gets you to a certain point, but I think to really be a good programmer, you can’t just do that. You have to understand a little bit more, and say, “Is it safe, what I’m doing here? Or what are the failure cases? Sure, I tried it once and it worked, but is it always going to work? How do I write test cases to show that and to understand it a little better, and then once I’ve done that, can I extract what I’ve done and publish a new tool that other people can use because I’ve put these pieces together in a certain way.”

Seibel: How did you like to work on a team when you were a programmer? Is it better to take the problem and split it up so everybody gets their piece? Or do you like the XP model of pair-program everything and everybody owns all the code collectively?

Norvig: I guess it’s more break it up. Steve Yegge’s got this “Good Agile, Bad Agile” piece. I think he’s about right. Ten percent of the time it is a really good idea to sit down together because you want that shared understanding. I think most of the time you’re not going to be as effective.

If you have two good programmers, it’s better for them to work independently and then debug each other’s work afterwards rather than to say, “We’ll take a 50 percent hit just for that added set of eyes.”

I think it is important to get together when you’re figuring out what it is you want to do both in terms of brainstorming what is the problem we’re trying to solve and what is the functionality going to be here? You don’t even know what the product is before you start. That you really want to do together. Then you get to the point of saying, “OK, now we know what we want to do. How are we going to divide it up?” That you want to do together. Once you have a pretty good idea, I think you’re better off spending most of the time on your own. You want feedback, so I think you should require every piece of code to be reviewed by another set of eyes, but it doesn’t have to be real-time, when you’re writing it.

I remember the IBM master-programmer idea, and that just seemed like the dumbest thing I have ever heard of. Why would anybody want to subject themselves to being a gopher for the one real programmer?

Seibel: I’m surprised you think the master-programmer model is such a dumb idea. In your “Teach Yourself Programming in Ten Years” essay you make the point that programming is a skill that, like many skills, probably takes about a decade to really master. And lots of crafts had master/journeymen/apprentice kind of hierarchies. So maybe nobody wants to be the apprentice, but maybe it isn’t crazy to say that somebody who’s been through that decade-long learning experience should be doing different work than someone who’s fresh out of school.

Norvig: I think the best part of the apprentice approach is that you get to watch the master, and I would like to see more of that. So I guess that’s another use of pair programming. I can see that it’d be really good, if you were inexperienced, to watch somebody who’s much more experienced. Particularly for the types of things that aren’t taught as much, like debugging skills. Anybody can learn algorithms and so on, but they don’t really teach debugging and watching someone, and saying, “Wow, I never thought of doing that,” that’s really useful.

But I think part of the reasons why you had master and apprentice is because the materials were rarer. When you were doing goldsmithing, there’s only so much gold. Or when the surgeon’s operating, there’s only one heart, and so you want the best person on that and you want the other guys just helping. With coding, it’s not like that. You’ve got plenty of terminals. You’ve got plenty of keyboards. You don’t have to ration it.

Seibel: Speaking of things that aren’t taught as much, you’ve been both an academic and in industry; do you feel like academic computer science and industrial programming meet in the right place?

Norvig: It’s a big question. I don’t think there’s a lot of waste in computer science curriculum. I think that it’s mostly very good stuff to know. I think going to school is useful, but it’s not everything that you need to be successful in the industry or to build systems. I do think that curriculum in many schools has been slow to adapt. There are a number of places where that comes into play: working in a team is not taught so much in school. This idea of being able to put the pieces together is not really taught there, but somehow the kids pick it up anyway, so maybe that’s OK. At Google we’re certainly interested in this large-scale cloud computing, parallel computing, and so forth. That’s not taught so much, although I think there’s a lot of interest in it. So I think they lag behind a little bit, but I think it’s still useful.

Seibel: And are there any areas where academics are out in front of industry? Where industry is ignoring good stuff about how we ought to build software.

Norvig: I think to some degree. Probably the best example was the model checking where Intel wasn’t really paying much attention and then they had this big recall and they lost a lot of money because they had a bug in their multiply. And then they started to pay attention, and they went to an academic and said, “What can you do to help us?” So there actually was something there. And now it’s an integral part of what they do, so that was a good example. It seems like programming languages, probably not so much. There’s a lot of work going on but you don’t see a big impact of the newer programming languages. Operating systems, a little bit. We’re supporting this RAD Lab at Berkeley with Dave Patterson and so on. They have some good ideas—how to make reliable systems. But it’s certainly the case that industry has the larger, bigger problems. They may not have all the answers to them—but they’re hitting them harder than in the university setting.

Seibel: So you don’t think there are any ideas floating around academia that haven’t been picked up by industry simply because people resist certain kinds of change—maybe a generation of self-taught PHP programmers are never going to warm to Haskell even if it might be a better way to write software?

Norvig: I guess I’m pretty skeptical. I think that if there were real advantages, people would be taking advantage of them. I don’t think it’s a perfect information market where everything instantly moves to the optimal solution, but I think it approximates that. Academics may be not seeing the whole problem that the industry has to deal with. And part of it is an education problem, but if you have a bunch of programmers who don’t understand what a monad is and haven’t taken courses in category theory, there’s a gap.

And part of it’s this legacy of we’ve got all these systems and you can’t just throw them out all at once, so there’s a transition. I’m sure there are places where industry should be more forward-looking about saying, “Sure, we can’t make that transition today, but we should have a plan to say where are we going to be in ten years? It’s not going to be where we are now and how do we get there?”

But you want improvements in areas that are going to make a big impact. And I think a lot of the times the level that programming languages are looking at is maybe too low a level to have as big an impact as language designers think it’s going to. So if they say, “Oh look, in my shiny new language, these six lines of code become two lines of code.” Well yeah, that’s nice, and I guess that makes you more productive and it’s easier to debug and maintain and so on. But maybe the code that you write is just a small part of the whole production system, and really the big headache is updating your data every day, and scraping the Web and getting this new data in, and putting it in the right format. So you have to remember that you’re solving a very small part of the overall problem and that means there has to be a big barrier to make it worthwhile to make a switch.

Seibel: So leaving aside the utility language research, you feel like we have come a ways since computer science was like majoring in IBM.

Norvig: Yeah. I think it’s a good curriculum now, and it’s depressing that it’s not being taken up by many students. Enrollments are down. Certainly there is a class of people who just love computers or computer design so much that that’s where they end up. We’re holding onto that group. But then there’s a bunch of the best and the brightest who are going into physics or biology or something because those are the hottest fields. And then there’s a bunch who are saying, “Well, I kind of like computers but there’s no future in it because all of the jobs are outsourced to India anyway, so I’m going to do prelaw or something else so I can get a job.” And I think that’s a shame. I think they’ve been misinformed.

Seibel: You mean it’s a shame because you think a lot of those people would enjoy being programmers, or because we need them?

Norvig: Both. Many people could enjoy lots of different things and if they enjoy two equally I don’t want to say that they have to do computer science. But I think there is a mismatch. We need more good people and I think that they can have a big impact on the world and if that’s what they want to accomplish, rationally we should be allocating more of the top people into computer science than we are now.

Seibel: In one of his papers Dijkstra talks about how computer science is a branch of mathematics and how computer-science students shouldn’t even touch a computer for the first n years of their education and should instead learn to manipulate systems of formal symbols. How much mathematics do you think is required to be a competent programmer?

Norvig: I don’t think you need the full Dijkstra level. And it’s a particular type of math that he focuses on. It’s discrete, logical proofs. I guess I’m coming from an area where that’s less important and it’s more probabilistic rather than logical. I rarely have a program that I can prove correct.

Is Google correct? Well, type in these words, you get back ten pages. If it crashes, then it’s incorrect but if it gives you back these ten links rather than those ten links, there’s no saying that one is right. You can have opinions on which are better than the other, but you can’t go beyond that. I think that’s quite different than what he had in mind. I think once you start solving these types of problems or the problem of having a robot car navigate through a city without hitting anybody, the logical proof gets thrown out pretty quickly.

Seibel: So is there any essential skill needed to be a good programmer? Different domains obviously have different requirements but ultimately is there some commonality to writing code regardless of the domain?

Norvig: You’ve got to be able to make progress and then improve on it. That’s all you need to be able to do in life. You’ve got to have some idea and say, “Here’s the direction to go,” and then be able to say, “Now I’ve got to refine it.” And refinement can be, “I didn’t quite get it right; there are cases I didn’t handle,” or it can be, “Now that I understand it better, I’m going to write a tool so that it’s more abstract and next time I can write a system like this more easily.” Which I guess is that level of introspection to say, “Where was I going? How did I get there? Is there a better way to get there?”

Seibel: So do you think that that skill—essentially make it; debug it; iterate—is a kind of thinking that lots of people should learn, even people who aren’t going to ultimately be programmers? If you were making a grade-school or junior-high or high-school curriculum, would you want everyone to be exposed to the idea of programming? Or is it too specialized a skill?

Norvig: I think it’s specialized. I think it’s one example of this type of thinking. But I’d be just as happy if you brought other examples like some type of mechanical problem. “Here’s a bunch of pieces. How can I move some water from here to here and get it into this cup?” It doesn’t have to be manipulating lines of code. It could be manipulating many kinds of pieces and seeing how they work together.

Seibel: And how far down should programmers go? In “Teach Yourself Programming in Ten Years,” you talk about knowing how long it takes to execute an instruction vs. reading from disk and so on. Do we still need to learn assembly?

Norvig: I don’t know. Knuth said, do everything in assembly, because it’s just too inefficient to write in C. I don’t agree with that. I think you want to know enough to know what instructions are inefficient, but that’s no longer at the level of individual instructions. It’s not just that this was a threeinstruction sequence rather than a two-instruction sequence. It’s, did you have a page fault or a cache miss? I don’t think we need to know assembly language. You need architecture. You should understand what assembly language is, and you should understand that there’s a memory hierarchy and missing from one level of the hierarchy to the next is a big performance penalty. But I think you can understand that at the abstract level.

Seibel: Are there any books that you think all programmers should read?

Norvig: I think there are a lot of choices. I don’t think there’s only one path. You’ve got to read some algorithm book. You can’t just pick these things out and paste them together. It could be Knuth, or it could be the Cormen, Leiserson, and Rivest. And there are others. Sally Goldman’s here now. She has a new book out that’s a more practical take on algorithms. I think that’s pretty interesting. So you need one of those. You need something on the ideas of abstraction. I like Abelson and Sussman. There are others.

You need to know your language well. Read the reference. Read the books that tell you both the mechanics of language and the whole enterprise of debugging and testing: Code Complete or some equivalent of that. But I think there are a lot of different paths. I don’t want to say you have to read one set of books.

Seibel: Though your job now doesn’t entail a lot of programming you still write programs for the essays on your web site. When you’re writing these little programs, how do you approach it?

Norvig: I think one of the most important things is being able to keep everything in your head at once. If you can do that you have a much better chance of being successful. That makes a small program easier. For a bigger program, you need extra tools to be able to handle that.

It’s also important to know what you’re doing. When I wrote my Sudoku solver, some bloggers commented on that. They said, “Look at the contrast—here’s Norvig’s Sudoku thing and then there’s this other guy, whose name I’ve forgotten, one of these test-driven design gurus. He starts off and he says, “Well, I’m going to do Sudoku and I’m going to have this class and first thing I’m going to do is write a bunch of tests.” But then he never got anywhere. He had five different blog posts and in each one he wrote a little bit more and wrote lots of tests but he never got anything working because he didn’t know how to solve the problem.

I actually knew—from AI—that, well, there’s this field of constraint propagation—I know how that works. There’s this field of recursive search—I know how that works. And I could see, right from the start, you put these two together, and you could solve this Sudoku thing. He didn’t know that so he was sort of blundering in the dark even though all his code “worked” because he had all these test cases.

Then bloggers were arguing back and forth about what this means. I don’t think it means much of anything—I think test-driven design is great. I do that a lot more than I used to do. But you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution.

Seibel: So then the question is, how should he have known that? Should he have gone and gotten a PhD and specialized in artificial intelligence? You can’t know every algorithm. These days you have Google, but finding the right approach to a problem is a little different than finding a web framework.

Norvig: How do you know what you don’t know?

Seibel: Exactly.

Norvig: So I guess it’s two parts. One is to recognize that maybe there is a known solution to this. You could say, “Well, nobody could possibly know how to do this, so just exploring randomly is as good as everything else.” That’s one possibility. The other possibility is, “Well, probably somebody does know how to do this. I just don’t know what the words are for it, so I have to discover those.” I guess that’s partly just intuition and saying, “It seems like the kind of thing that should be in the body of knowledge from AI.” And then you have to figure out, how do I find it? And probably he could’ve done a search on Sudoku and found it that way. Maybe he thought that was cheating. I don’t know.

Seibel: So let’s say that is cheating—say you were the first person ever to try and solve Sudoku. The techniques that you ended up using would still have been out there waiting to be applied.

Norvig: Let’s say I wanted to solve some problem in biology. I wouldn’t know what the best algorithms were for doing gene sequencing or whatever. But I’d have a pretty good idea that there were such algorithms. Then I could start looking around. At another level, some of these things are pretty fundamental—if you don’t know what dynamic programming is, then you’re at a severe disadvantage. It’s going to come up time and time again. If you don’t know this idea of search in general—that you can make a choice and backtrack when you don’t need it. These are all ideas from the ’60s. It was only a few years into programming that people discovered these things. It seems like that’s the type of thing that everyone should know. Some things that were discovered last year, not everybody should know.

Seibel: So should programmers go back and read all the old papers?

Norvig: No, because there are lots of false starts and lots of mergers where two different fields develop completely different technology and terminology, and then they discover they were really doing the same thing. I think you’d rather have a story from the modern point of view rather than have to follow all the steps. But you should have them. I don’t know what the best books are for that since I picked it up the hard way, piecemeal.

Seibel: So back to designing software. What about when you’re working on bigger programs, where you’re not going to be able to just remember how all the code fits together? Then how do you design it?

Norvig: I think you want to have good documentation at the level of overall system design. What’s the thing supposed to do and how’s it going to do it? Documentation for every method is usually more tedious than it needs to be. Most of the time it just duplicates what you could read from the name of the function and the parameters. But the overall design of what’s going to do what, that’s really important to lay out first. It’s got to be something that everybody understands and it’s also got to be the right choice. One of the most important things for having a successful project is having people that have enough experience that they build the right thing. And barring that, if it’s something that you haven’t built before, that you don’t know how to do, then the next best thing you can do is to be flexible enough that if you build the wrong thing you can adjust.

Seibel: How much do you think you can sit down and figure out how something ought to work, assuming it’s not something that you’ve built before? Do you need to start writing code in order to really understand what the problem is?

Norvig: One way to think about it is going backwards. You want to get to an end state where you have something that’s good and for some problems there’s roughly one thing that’s good. For other problems there are roughly millions and you can go in lots of different directions and they’d all be roughly the same. So I think it’s different depending on which type of those types of problems you have.

Then you want to think about what are the difficult choices vs. what are the easy ones. What’s going to come back to really screw you if you make the wrong architectural choice—if you hit built-in limitations or if you’re just building the wrong thing. At Google I think we run up against all these types of problems. There’s constantly a scaling problem. If you look at where we are today and say, we’ll build something that can handle ten times more than that, in a couple years you’ll have exceeded that and you have to throw it out and start all over again. But you want to at least make the right choice for the operating conditions that you’ve chosen—you’ll work for a billion up to ten billion web pages or something. So what does that mean in terms of how you distribute it over multiple machines? What kind of traffic are you going to have going back and forth? You have to have a convincing story at that level. Some of that you can do with calculations on the back of the envelope, some of that you can do with simulations, and some of that you have to predict the future.

Seibel: It seems for that kind of question you’ll be far more likely to answer correctly with either back-of-the-envelope calculations or simulation than writing code.

Norvig: Yeah, I think that’s right. Those are the kind of things where the calculations are probably a better approach. And then there are these issues of some vendor says they’re going to have a switch coming out next year that will handle ten times as much traffic; do you design to that? Do you believe them? Or do you design to what you have today? There are a lot of trade-offs there.

Then there are user-interface things where you just don’t know until you build it. You think this interaction will be great but then you show it to the user and half the users just can’t get it. Then you have to backtrack and come up with something new.

Seibel: Leaving aside designing user interactions, when is prototyping valuable? As opposed to just thinking about how something is going to work?

Norvig: I think it’s useful to imagine the solution, to see if it’s going to work. It’s useful to see if it feels comfortable. You want a set of tools that are going to help you build what you have to build now and are going to help you evolve the system over time. And if you start out prototyping and all the sudden it feels clunky, then maybe you’ve got the wrong set of primitives. It’d be good to know that as soon as possible.

Seibel: What about the idea of using tests to drive design?

Norvig: I see tests more as a way of correcting errors rather than as a way of design. This extreme approach of saying, “Well, the first thing you do is write a test that says I get the right answer at the end,” and then you run it and see that it fails, and then you say, “What do I need next?”—that doesn’t seem like the right way to design something to me.

It seems like only if it was so simple that the solution was preordained would that make sense. I think you have to think about it first. You have to say, “What are the pieces? How can I write tests for pieces until I know what some of them are?” And then, once you’ve done that, then it is good discipline to have tests for each of those pieces and to understand well how they interact with each other and the boundary cases and so on. Those should all have tests. But I don’t think you drive the whole design by saying, “This test has failed.”

The other thing I don’t like is a lot of the things we run up against at Google don’t fit this simple Boolean model of test. You look at these test suites and they have assertEqual and assertNotEqual and assertTrue and so on. And that’s useful but we also want to have assertAsFastAsPossible and assert over this large database of possible queries we get results whose score is precision value of such and such and recall value of such and such and we’d like to optimize that. And they don’t have these kinds of statistical or continuous values that you’re trying to optimize, rather than just having a Boolean “Is this right or wrong?”

Seibel: But ultimately all of those can get converted into Booleans—run a bunch of queries and capture all those values and see if they’re all within the tolerances that you want.

Norvig: You could. But you can tell, just from the methods that the test suites give you, that they aren’t set up to do that, they haven’t thought about that as a possibility. I’m surprised at how much this type of approach is accepted at Google—when I was at Junglee I remember having to teach the QA team about it. We were doing this shopping search and saying, “We want a test where on this query we want to get 80 percent right answers.” And so they’re saying, “Right! So if it’s a wrong answer it’s a bug, right?” And I said, “No, it’s OK to have one wrong answer as long at it’s not 80 percent.” So they say, “So a wrong answer’s not a bug?” It was like those were the only two possibilities. There wasn’t an idea that it’s more of a trade-off.

Seibel: But you are still a believer in unit tests. How should programmers think about testing?

Norvig: They should write lots of tests. They should think about different conditions. And I think you want to have more complex regression tests as well as the unit tests. And think about failure modes—I remember one of the great lessons I got about programming was when I showed up at the airport at Heathrow, and there was a power failure and none of the computers were working. But my plane was on time.

Somehow they had gotten print-outs of all the flights. I don’t know where—there must have been some computer off-site. I don’t know whether they printed them that morning or if they had a procedure of always printing them the night before and sending them over and every day when there is power they just throw them out. But somehow they were there and the people at the gates had a procedure for using the paper backup rather than using the computer system.

I thought that was a great lesson in software design. I think most programmers don’t think about, “How well does my program work when there’s no power?”

Seibel: How does Google work when there’s no power?

Norvig: Google does not work very well without power. But we have backup power and multiple data centers. And we do think in terms of, “How well does my piece work when the server it’s connecting to is down or when there are other sorts of failures?” Or, “I’m running my program on a thousand machines; what happens when one of them dies?” How does that computation get restarted somewhere else?

Seibel: Knuth has an essay about developing TeX where he talks about flipping over to this pure, destructive QA personality and doing his darnedest to break his own code. Do you think most developers are good at that?

Norvig: No. And I had an example of that in my spelling corrector. I had introduced a bug in the code that measured how well I was doing and simultaneously had made some minor change in the real code. I ran it and I got back a much better score for how well it was doing. And I believed it! If it had been a much worse score I would have never said, “Oh, this minor change to the real function must have made it much worse.” But I was willing to believe this minor change made the score much better rather than being skeptical and saying, “Nah, couldn’t have made that much difference, there must be something else wrong.”

Seibel: How do you avoid over-generalization and building more than you need and consequently wasting resources that way?

Norvig: It’s a battle. There are lots of battles around that. And, I’m probably not the best person to ask because I still like having elegant solutions rather than practical solutions. So I have to sort of fight with myself and say, “In my day job I can’t afford to think that way.” I have to say, “We’re out here to provide the solution that makes the most sense and if there’s a perfect solution out there, probably we can’t afford to do it.” We have to give up on that and say, “We’re just going to do what’s the most important now.” And I have to instill that upon myself and on the people I work with. There’s some saying in German about the perfect being the enemy of the good; I forget exactly where it comes from—every practical engineer has to learn that lesson.

Seibel: Why is it so tempting to solve a problem we don’t really have?

Norvig: You want to be clever and you want closure; you want to complete something and move on to something else. I think people are built to only handle a certain amount of stuff and you want to say, “This is completely done; I can put it out of my mind and then I can go on.” But you have to calculate, well, what’s the return on investment for solving it completely? There’s always this sort of S-shaped curve and by the time you get up to 80 or 90 percent completion, you’re starting to get diminishing returns. There are 100 other things you could be doing that are just at the bottom of the curve where you get much better returns. And at some point you have to say, “Enough is enough, let’s stop and go do something where we get a better return.”

Seibel: And how can programmers learn to better recognize where they are on that curve?

Norvig: I think you set the right environment, where it’s results-oriented. And I think people can train themselves. You want to optimize, but left to yourself you optimize your own sense of comfort and that’s different from what you really should be optimizing—some people would say return on investment for the company, others would say satisfaction of your customers. You have to think how much is it going to benefit the customer if I go from 95 percent to 100 percent on this feature vs. working on these ten other features that are at 0 percent.

At Google, I think it’s easy because we have this “launch early and often” philosophy. And because of the way the company is, for a number of reasons: one, most of our products we don’t charge any money for so it’s easy to say, well, go ahead and ship it; how much could they complain? The other one is we’re not stamping CDs and putting them in a box so if there’s something that’s not complete today or even if it has a bug, it’s not a disaster. Most of the software is on our servers so we can fix it tomorrow and everybody gets the update instantly. We don’t have this nightmare of installing updates. So it makes it easier for us to say, “We’re just going to launch things and get some feedback from the users and fix the stuff that needs to be fixed and don’t worry about the other stuff.”

Seibel: If you’re working on the design of a big system what are the tools you use—do you sit down with a pad of graph paper or a UML drawing tool?

Norvig: I never liked any of these UML-type tools. I always thought, “If you can’t do it in the language itself that’s a weakness of the language.” I think a lot of what you’re doing, you’re dealing at a higher level. At Google a lot of what we do is figuring out how to break things up and parallelize them. We’re going to necessarily run this on multiple machines but we’ve got so many users and, for many applications, so much data; how’s that going to work? So we’re thinking more at the level of machines and racks of machines rather than at the level of functions and interactions. Once you get that straightened out, then you can start diving into individual functions and methods.

Seibel: And so that level of description is just prose?

Norvig: Yeah, mostly. Sometimes people draw pictures. They’ll say, “We’ll have this server here that will be serving these kinds of requests and then it’s connected to this server and then we’ll use these various tools for storage and big distributed hash tables and other types of things. We’ll choose these three tools off the shelf and then we’ll argue about whether we have to build a new one; which of these existing ones works or do we need something else?”

Seibel: And how do you evaluate that kind of design?

Norvig: You show it to the people who’ve done it before. They say, “Oh, it looks like you’ll need a cache here—it’s going to be too slow but there should be a lot of repeat requests, so if you install a cache of this size here that should help a lot.” You have a design review where people look over it and say whether they think it makes sense and then you start building it and testing it.

Seibel: And you guys have formal design reviews? You worked at NASA where they had a very formal design review.

Norvig: Nothing formal like NASA. The stakes are lower for us because, as I say, it’s easy for us to have a failure and recover from that. At NASA usually the first failure is fatal so they were much more careful. We don’t worry about that much. It’s more of a consultation, I think, rather than a review.

There are people who officially read design documents and comment on them. You go through that and get your design approved. But it’s still much

Norvig: I think it was something that had always been done and so people accept it. Well, I shouldn’t say that completely. Some people it takes a while to get used to it. One of the typical failure cases is a new hire comes in and they’re not used to doing this kind of thing so they just start an experimental branch and they have all their code in there and you keep on telling them, “Gee, you don’t have any check-ins yet.” And they say, “Yeah, yeah, yeah, I’m just cleaning it up—I’ll check it in tomorrow.” And then another week goes by and another week goes by and eventually they have this one gigantic check-in. And then it’s a problem that too much time has gone by, it’s hard to evaluate it all at once, and some of the things they’re comparing against have changed out from underneath them. Then they see what a headache it is and they learn not to do that.

Seibel: So that’s on the coding side. Are there skills that the reviewers develop?

Norvig: There certainly are people who are known for being better reviewers than others. There’s a trade-off of when you submit a review—do you try to get somebody who will give you a lot of good feedback or do you try to get somebody who will just say “OK” as quickly as possible?

Seibel: So what makes the better reviewers better?

Norvig: Well, that they catch more things. Some of it is the trivial stuff of you indented the wrong number of spaces or whatever but some of it is, “I think this design would be cleaner if you moved this from here over to there.” So some people will do more of that and others won’t bother.

Seibel: Sort of related to that, does every good programmer turn into a good architect when they grow up? Or are there some people who are brilliant coders but only at a certain level and they should never be allowed to do bigger designs?

Norvig: I think different people have different skills. One of our best search people is by no means our best programmer, in terms of how the code looks. But if you say, “Here’s this new factor that we have—you know, how many times people click on this page after they’ve done such and such—how do we fold that into our search results?” He’ll say, “Oh, on line 427 there’s this variable alpha and you should take this new factor and raise it to the second power and multiply it by 1.5 and add it to alpha.” Then you experiment for a couple months trying different things and you find out he was right except it should have been 1.3 instead of 1.5.

Seibel: So that suggests he just has this very good mental model of how the software works.

Norvig: He understands the code perfectly. Other people can write code better but he understands all the implications of what goes where.

Seibel: Do you think those are related? It often seems that people who write the worst spaghetti code are the ones who can hold the most in their head—that’s the only way they could possibly write code like that.

Norvig: Yeah, I think that may be.

Seibel: So the reviews here are less formal here than at NASA. What are the other differences between the “engineering” and “hacker” ethos, in the best sense of both those words?

Norvig: One big difference is organizational structure and how software is accepted. Google was founded as a software company, and they went out and hired a CEO who has a PhD in computer science from Berkeley, hired a VP of Sales who has a computer engineering background—it’s throughout the whole company. At NASA they’re rocket scientists! They aren’t software guys. They say, “Software is this necessary evil. Straight line code, I can sort of understand; if it’s got a loop in it, that’s kind of iffy. Then if there’s a branch statement inside the loop, ooooh, that’s getting away from what I can solve with a differential equation in control theory.” So they’re distrustful.

Seibel: As well they should be!

Norvig: As well they should be, yeah. And they’re distrustful of innovation. So you can say, “Look at this great new prototype I have,” and they’ll say, “That’s fantastic; I’d love to fly that on my mission—as soon as it’s been proven on two other missions.” And you go to everybody else and they all say the same thing.

Don Goldin came in as NASA administrator and he said, “We’ve got to do this better, faster, cheaper. These space missions cost too much. It’d be better to run more missions and some of them would fail but overall we’d still get more done for the same amount of money.” And that was undeniably true. Unfortunately it was not politically true. It’s not OK to lose a spacecraft. Because the public understands very well, NASA lost a spacecraft. They don’t really know that there’s any difference between a $100 million spacecraft and a billion dollar spacecraft. It’s not like you get to lose ten of the $100 millions instead of one billion. So it was never quite true.

Seibel: What is the worst bug you’ve ever had to track down?

Norvig: Well, I guess the most consequential bugs I was involved with were not mine, but the ones I had to clean up after: the Mars program failures in ’98. One was foot-pounds vs. newtons. And the other was, we think, though we’re not 100 percent sure, prematurely shutting off the engines due to a software problem.

Seibel: I read one of the reports on the Mars Climate Orbiter—that was the one that was the foot-pounds vs. newtons problem—and you were the only computer scientist on that panel. Were you involved in talking to the software guys to figure out what the problem was?

Norvig: That was pretty easy, post hoc, because they knew the failure mode. From that they were able to back it out and it didn’t take long to figure that one out. Then there was this postmortem of why did it happen? And I think it was a combination of things. One was outsourcing. It was a joint effort between JPL in Pasadena and Lockheed-Martin in Colorado. There were two people on two different teams and they just weren’t sitting down and having lunch together. I’m convinced that if they had, they would have solved this problem. But instead, one guy sent an email saying, you know, “Something not quite right with these measurements, seems like we’re off by a little bit. It’s not very much, it’s probably OK, but—”

Seibel: That was all during the flight?

Norvig: Right. During the flight they had chance and chance to catch it. They knew something was wrong and they sent this email but they did not put it into the bug-tracking system. If they had, NASA has very good controls for bug tracking and at later points in the flight somebody would have had to OK it. Instead it was just an informal email that never got an answer back, and JPL said, “Oh, I guess Lockheed-Martin must have solved this problem.” And Lockheed says, “Oh, JPL’s not asking anymore—they must not be concerned.”

So it was this communications problem. It was also a software reuse problem. They have extremely good checks for the stuff that’s missioncritical, and on the previous mission the stuff that was recorded in footpounds was non–mission-critical—it was just a log that wasn’t used for navigation. So it had been classified as non–mission-critical. In the new mission they reused most of the stuff but they changed the navigation so that what was formerly a log file now became an input to the navigation.

Seibel: So the actual problem was that one side generated a data file in foot-pounds and that data file was fed into a piece of software that was calculating inputs to the actual navigation and was expecting newtons?

Norvig: Right. So essentially the other root cause was too many particles from the sun. The spacecraft is asymmetrical and it’s got these solar panels. Particles twist the spacecraft a little bit so you’ve got to fire the rockets to twist it back. So this new hire at Lockheed went to the rocket manufacturer, and they had all their specifications in foot-pounds, so he just said, “I’ll go with that,” and he recorded them that way, not knowing that NASA wanted them in metric.

Seibel: I was struck, reading that report, by the NASA attitude of, “Well, the problem was due to this software bug but we had so many other chances to notice that the ship wasn’t where we expected, and we should have. We should have fixed it anyway even though the numbers we were dealing with were totally bogus because of some stupid software glitch.” I thought that was admirable.

Norvig: Yeah, they were looking at the process.

Seibel: Is it actually common for there to be software bugs of that magnitude, which we never hear about because all the other processes keep everything online?

Norvig: Yeah, I think so. Look at all the software bugs on your computer. There are millions of them. But most of the time it doesn’t crash.

Seibel: Yet you hear about how the shuttle flight software costs $1,500 a line or something because of the care with which they write it and which is allegedly bug-free. Is that just a lie?

Norvig: No, that’s probably true. But I don’t know if it’s optimal. I think they might be better off with buggy software.

Seibel: With cheaper software and better operations?

Norvig: Yeah, because of the amount of training they have to give to these astronauts to be able to deal with the things the software just can’t do. They put these astronauts in simulators and give them all these situations and when things go bad you’ve got this screen and stuff is scrolling through it and you can’t pause the screen, you can’t go back, you can’t get a summary of what the important things are. The astronauts just have to be trained to know, “When I see this happening, here’s what’s really going on.” There are a hundred messages in a row saying, “This electrical thing has faulted,” and you train them to say, “OK, that must be because this one original one faulted and then there was a cascade downstream and all the other ones are reported.” Why can’t you do that in software rather than train the astronaut? They don’t try because they don’t want to mess with it.

Seibel: On a different topic, what are your preferred debugging techniques and tools? Print statements? Formal proofs? Symbolic debuggers?

Norvig: I think it’s a mix and it depends on where I am. Sometimes I’m using an IDE that has good tracing capability and sometimes I’m just using Emacs and don’t have all that. Certainly tracing and printing. And thinking. Writing smaller test cases and watching them go, and breaking the functionality down to see where the test case failed. And I’ve got to admit, I often end up rewriting. Sometimes I do that without ever finding the bug. I get to the point where I can just feel that it’s in this part here. I’m just not very comfortable about this part. It’s a mess. It really shouldn’t be that way. Rather than tweak it a little bit at a time, I’ll just throw away a couple hundred lines of code, rewrite it from scratch, and often then the bug is gone.

Sometimes I feel guilty about that. Is that a failure on my part? I didn’t understand what the bug was. I didn’t find the bug. I just dropped a bomb on the house and blew up all the bugs and built a new house. In some sense, the bug eluded me. But if it becomes the right solution, maybe it’s OK. You’ve done it faster than you would have by finding it.

Seibel: What about things like assertions or invariants? How formally do you think about those kinds of things when you’re coding?

Norvig: I guess I’m more on the informal side. I haven’t used languages where that’s a big part of the formal mechanism, other than just type declarations. Like loop invariants: I’ve always thought that was more trouble than it was worth. I occasionally have a problem where this loop doesn’t terminate, but mostly you don’t, and I just feel like it slows you down to do the formal part. And if you do have a problem, the debugger will tell you what loop you’re stuck inside. I guess if you’re writing high-dependability software that’s embedded in something that it’s really important that it doesn’t fail, then you really want to prove everything. But just in terms of getting the first version of the program running or debugging it, I’d rather move fast towards that than worry about the degree of formal specification you need later.

Seibel: Have you ever done anything explicit to try and learn from the bugs that you’ve created?

Norvig: Yeah, I think that’s pretty interesting and I wish I could do more with that. I’m actually in a discussion now to see if I can do an experiment, company-wide and then maybe for the world at large, to understand more some of these issues. How do you classify bugs, but also what are some factors in terms of productivity? How do you know? Is there a certain type of person? What are the factors of that person that makes them more productive? And I think it’s more interesting what the controllable factors are that make somebody do better. If giving them a bigger monitor increases productivity by such and such a percent, then you should probably do it.

Seibel: People are going to hate you when you discover that actually, really tiny monitors make people more productive.

Norvig: Right. And if providing quiet is important, then you should probably do that, but on the other hand, if providing communication between team members is important, then you should do that, and how do you balance those two?

I just started thinking about what’s the right way to do that. How do you set up an experiment? What do you track? Do we have numbers already that we could make use of just by adding in some kind of questionnaire? Do we have to set up an experiment?

Seibel: It’s often claimed that there are orders of magnitude differences in productivity between programmers. Yet I read somewhere some criticism of those claims, saying the studies that found that were done quite some time ago, and a lot of things have changed about programming since then that could have accounted for the differences—such as some people in the study were still using batch processing techniques while other people were using timesharing programming environments.

Norvig: I don’t think that’s all of it, because I think there were some differences within the same organization using more of the same tools. I also remember there were criticisms of some of the studies of finding correlation, but not knowing cause and effect. If you found that the programmers in the big corner offices were more productive, is that because you reward the good programmers with the offices, or is it because the offices makes them better? You can’t really come to a conclusion.

Seibel: Do you still enjoy programming as much as when you were starting out?

Norvig: Yeah, but it’s frustrating to not know everything. I’m not doing it as much, so I forget a little bit. And there are all these new things. I really should redesign my web site and it should have JavaScript on the client side. It should have PHP or something like that, and I just haven’t gotten up the momentum to learn all that stuff and be able to do it.

Seibel: Do you think programming is a young person’s game?

Norvig: I think it helps in some ways. We certainly have a range of people here that are exceptionally good at all levels and all ages. I think the advantage of being young is it’s important to grasp the whole program, the whole problem, in your head—being able to concentrate. And I think that’s easier when you’re younger, because your brain is better at it, or maybe it’s just that you have less distractions. If you have kids, and family, and so on, you just can’t devote as many consecutive hours as when you don’t. So that’s part of it. But on the other hand, you’ve got this range of experience, so you can make up for that in some ways by being able to do more because you know how to do it.

Seibel: One of the aspects of the modern style of programming, as you were saying, is that programmers have to absorb things quickly. How do you tackle the problem of understanding a big pile of code, none of which you’ve ever seen before?

Norvig: I think you do a mix of statically and dynamically. You start reading the code and trying to make sense of it and then you get some traces of what calls what, and where most of time is spent, and what’s the flow through it. Then try to do something. Say, “I’m going to make this trivial little change.” Or go to the issues database and say, “I’ll take this one.” In order to do that, I have to learn a little piece of it. There’s only a little piece, but you get that done and you move on to the next one.

Seibel: Have you ever done literate programming a la Knuth?

Norvig: I never used his tools per se. I’ve certainly written macros and so on. And I’ve used the Java docs and things like that. In many ways, Lisp programming encourages you to make your own system as you go, and so it ends up being literate in that way. You find your own macros for your own application-specific programming, and part of that is the documentation, part of it’s the data, and part of it’s the code, so I’ve certainly done that. Then more recently, in whatever language I’m using, whether it’s Java, or Python, or whatever, I’ve certainly been careful to write test cases, and document around that.

You look at Knuth’s original Literate Programming, and he was really trying to say, “What’s the best order for writing a book,” assuming that someone’s going to read the whole book and he wants it to be in a logical order. People don’t do that anymore. They don’t want to read a book. They want an index so they can say, “What’s the least amount of this book that I have to read? I just want to find the three paragraphs that I need. Show me that and then I’ll move on.” I think that’s a real change.

Seibel: I wonder if there isn’t a way to write modern-style literate programming. Certainly Knuth’s tools give you an index and beautiful crossreferencing. I wonder if perhaps a modern approach to literate programming just would organize the book differently—both as a whole program and as a bunch of pieces that you can understand in bits?

Norvig: I don’t know. I think he was solving a problem that doesn’t exist anymore to a large degree. Part of it was because he wanted to put it in a linear order rather than in a web-like or a searchable order. I think part of it was the limitations. I think he was using Pascal originally. And there it’s pretty strict in terms of what had to be declared first and not necessarily in the order you want. Modern languages are more free in that order, so I think it’s less of an issue now.

Seibel: You mentioned reading Strachey’s checkers code in Scientific American. And in your “Teach Yourself Programming in Ten Years” essay, you talk about the importance of reading code. What code did you read coming up?

Norvig: I read a lot of the Symbolics’s code, because that was available when I was at Berkeley.

Seibel: Was that just because it was available and was interesting? Or were you reading it to try to understand some behaviors you were observing?

Norvig: Both. I sometimes just tried to figure out how things work and sometimes I needed it to solve a problem.

Seibel: So when you’re reading just for general edification, how do you approach that?

Norvig: I think it’s probably interest-driven. “Gee, this file system allows you to read files across the net using the same protocol that you use locally on your machine—I wonder how it does that?” And so you’ll say, “Maybe it’s in the open function.” You look there, and say, “Oh, that calls this other thing.” And you look there, and eventually you say, “Oh, this is how it does it.”

Seibel: Have you read any of Knuth’s literate programs in book form?

Norvig: I’ve certainly picked up the books and flipped through them. I could say I glanced at them, but I haven’t studied them.

Seibel: What about The Art of Computer Programming? Some of the people I’ve talked to on this have absolutely read it from cover to cover. Some people have it on the shelf and use it as a reference. And some people just have it on the shelf.

Norvig: At one point I had it as my monitor stand because it was one of the biggest set of books I had, and it was just the right height. That was nice because it was always there, and I guess then I was more prone to use it as a reference because it was just right in front of me.

Seibel: But you had to lift up the monitor every time you wanted to look at it?

Norvig: No, I had the box set. You had to pull hard, but you could pull one of the box. Now I’m less likely to use any book for reference—I’m just likely to do a search.

Seibel: Just because it’s more convenient?

Norvig: It’s convenient. I think it’s also that I’m probably more goaloriented. Knuth is good if you say, “I want to know everything about this subject.” But usually I’m saying, “I want to know if A is better than B,” or, “I want to know the asymptotic complexity of this, and once I’ve got that, I don’t need all the details of how we got there.”

Seibel: As a programmer, do you consider yourself a scientist, an engineer, an artist, or a craftsman?

Norvig: Well, I know when you compare the various titles of books and so on, I always thought the “craft” was the right answer. So I thought art was a little pretentious because the purpose of art is to be beautiful or to have an emotional contact or emotional impact, and I don’t feel like that’s anything that I try to do. Certainly I want programs to be pretty in some ways, and sometimes I feel like I spend too much time doing that. I’ve been in a position where I’ve had the luxury to say, “Gee, I have time to go back and pretty this up a little bit.” And places where I’ve been able to write for a publication, you spend more time doing that than you would if it was just for your own professional growth.

But I don’t think of that as art. I think craft is really the right word for it. You can make a chair, and it’s good looking, but it’s mostly functional—it’s a chair.

Seibel: How do you recognize a great programmer, particularly when you’re hiring? You guys have hired a lot of programmers and you obviously try to hire really good programmers. How do you do it?

Norvig: We still don’t know.

Seibel: Google is somewhat famous for asking puzzle questions in interviews. Do you think that’s a good approach?

Norvig: I don’t think it’s important whether people can solve the puzzles or not. I don’t like the trick puzzle questions. I think it’s important to put them in a technical situation and not just chitchat and get a feeling if they’re a nice guy. Though it is important to have someone that you can get along with. But you really have to see, can they technically do what they said they can do. And there are a lot of different ways to demonstrate that. And many times you can see it from the r?sum?. I think our best signal is if somebody has worked with one of our employees before and the employee can vouch for them. But we still try to draw it out on-site during the interview. It’s more you want to get a feeling for how this person thinks and how they work together, so do they know the basic ideas? Can they say, “Well, in order to solve this, I need to know A, B, and C,” and they start putting it together. And I think you can demonstrate that while still failing on a puzzle. You can say, “Well, here’s how I attack this puzzle. Well, I first think about this. Then I do that. Then I do that, but geez, here’s this part I don’t quite understand.” For some people that little part clicks and for some it doesn’t. And you can do fine if it doesn’t click as long as you’ve demonstrated the basic competency and fluency in how you think about it. And then you really want to have people write code on the board if you’re interviewing them for a coding job. Because some people have forgotten or didn’t quite know and you could see that pretty quickly.

Seibel: So is that just a negative indicator? If they can’t write reasonable code, that’s a bad sign. But if they don’t stumble over that hurdle, it’s hard to tell whether they’re actually going to write really good code in a larger context.

Norvig: Right. You could tell to a certain degree but at other levels you can’t. And we’ve studied this pretty carefully because we’ve gotten a lot of applications and we look at it at two levels. One, we say, from all the r?sum?s we get, are we interviewing the right set of people? And then, from the interviews we get, are we hiring the right set of people?

Seibel: So how do you measure that? You don’t know about the one you didn’t talk to or didn’t hire.

Norvig: Yeah, so that’s hard. At both levels, you’ve only got half your sample, so it’s this biased problem, but I guess basically what we’re doing is saying, “Of the people that we interviewed and did well, what did their r?sum? look like,” and try to find more of those. Is having so many years of experience important? Is working on an open-source programming project important? How does that compare to winning a programming contest?

Seibel: Do you really take all these things and shove them into a database?

Norvig: Yes, we do, and when we’re doing the hiring, we have these scores that come up that say, “The r?sum? predictor says such and such, and the interview predictor says such and such.” We don’t take them as gospel, but they’re just another piece of input along with all the other feedback we have.

Seibel: Do the people who are doing the interviews have those numbers beforehand?

Norvig: No, we only see that when they’re in the hiring committee, once we’ve gathered all the feedback. One of the interesting things we found, when trying to predict how well somebody we’ve hired is going to perform when we evaluate them a year or two later, is one of the best indicators of success within the company was getting the worst possible score on one of your interviews. We rank people from one to four, and if you got a one on one of your interviews, that was a really good indicator of success.

Seibel: But you had to do well enough on something else that you actually got hired?

Norvig: Right, so that’s the thing. Ninety-nine percent of the people who got a one in one of their interviews we didn’t hire. But the rest of them, in order for us to hire them somebody else had to be so passionate that they pounded on the table and said, “I have to hire this person because I see something in him that’s so great, and this guy who thought he was no good is wrong, and I’ve got to stand up for him and put my reputation on the line.”

Seibel: So you’re surrounded by top-notch programmers here at Google. Given how pervasive computers and software are in our society, do you think everybody needs to understand a bit about programming just to get along in or understand the world they live in?

Norvig: You probably want an educated person to understand how software is made to the same degree they understand how a car is made. The other interesting thing is, how much does an informed citizen have to be a programmer? Certainly the average person now can do word processing and many of them can do spreadsheets, and so if you’re a little bit experienced with spreadsheets, you’re starting to be a programmer.

Lots of attempts at end-user programming, and programming for everyone, haven’t been very successful. I don’t know how easy it is. Is there a way of thinking that people have that we’ve gotten all the people that it’s easy to teach, and the other ones are going to be really hard, or is it just that we’ve missed the model and there’s some simple model in place of many people that could influence programming if we created it?

Seibel: A lot of people that I have talked to for this book, and elsewhere, got into computers both because they just enjoyed it and because they felt like it would change the world. Some of the folks I talked to for this book did that in the past and now are depressed by how little they feel the world has changed as a result. How do you feel about that?

Norvig: Well, I’m in the right place. We have hundreds of millions of users and we can make a difference for them, and we can launch new services for them quickly. I think that’s great. I can’t imagine anything else I could be doing to have that level of impact.

Оглавление книги