Книга: Coders at Work: Reflections on the craft of programming

Jamie Zawinski

Jamie Zawinski

Lisp hacker, early Netscape developer, and nightclub owner Jamie Zawinski, a.k.a. jwz, is a member of the select group of hackers who are as well known by their three-letter initials as by their full names.

Zawinski started working as a programmer as a teenager when he was hired to hack Lisp at a Carnegie Mellon artificial intelligence lab. After attending college just long enough to discover that he hated it, he worked in the Lisp and AI world for nearly a decade, getting a strange immersion in a fading hacker subculture when other programmers his age were growing up with microcomputers.

He worked at UC Berkeley for Peter Norvig, who has described him as “one of the of the best programmers I ever hired,” and later at Lucid, the Lisp company, where he ended up leading the development of Lucid Emacs, later renamed XEmacs, which eventually led to the great Emacs schism, one of the most famous open source forks.

In 1994 he finally left Lucid and the Lisp world to join Netscape, then a fledgling start-up, where he was one of the original developers of the Unix version of the Netscape browser and later of the Netscape mail reader.

In 1998 Zawinski was one of the prime movers, along with Brendan Eich, behind mozilla.org, the organization that took the Netscape browser open source. A year later, discouraged by the lack of progress toward a release, he quit the project and bought a San Francisco nightclub, the DNA Lounge, which he now runs. He is currently devoting his energies to battling the California Department of Alcoholic Beverage Control in an attempt to convert the club to an all-ages venue for live music.

In this interview we talked about, among other things, why C++ is an abomination, the joy of having millions of people use your software, and the importance of tinkering for budding programmers.

Seibel: How did you learn to program?

Zawinski: Wow, it was so long ago I can barely remember it. The first time I really used a computer in a programming context was probably like eighth grade, I think. We had some TRS-80s and we got to goof around with BASIC a little bit. I’m not sure there was even a class—I think it was just like an after-school thing. I remember there was no way to save programs so you’d just type them in from magazines and stuff like that. Then I guess I read a bunch of books. I remember reading books about languages that I had no way to run and writing programs on paper for languages that I’d only read about.

Seibel: What languages would that have been?

Zawinski: APL, I remember, was one of them. I read an article about it and thought it was really neat.

Seibel: Well, it saves having to have the fancy keyboard. When you were in high school did you have any classes on computers?

Zawinski: In high school I learned Fortran. That’s about it.

Seibel: And somehow you got exposed to Lisp.

Zawinski: I read a lot of science fiction. I thought AI was really neat; the computers are going to take over the world. So I learned a little bit about that. I had a friend in high school, Dan Zigmond, and we were trading books, so we both learned Lisp. One day he went to the Apple Users Group meeting at Carnegie Mellon—which was really just a software-trading situation—because he wanted to get free stuff. And he’s talking to some college student there who’s like, “Oh, here’s this 15-year-old who knows Lisp; that’s novel; you should go ask Scott Fahlman for a job.” So Dan did. And Fahlman gave him one. And then Dan said, “Oh, you should hire my friend too,” and that was me. So Fahlman hired us. I think his motivation had to be something along the lines of, Wow, here are two high school kids who are actually interested in this stuff; it doesn’t really do me much harm to let them hang out in the lab.” So we had basic grunt work—this set of stuff needs to be recompiled because there’s a new version of the compiler; go figure out how to do that. Which was pretty awesome. So there are the two of us—these two little kids—surrounded by all these grad students doing language and AI research.

Seibel: Was that the first chance you actually had to run Lisp, there at CMU.

Zawinski: I think so. I know at one point we were goofing around with XLISP, which ran on Macintoshes. But I think that was later. I learned how to program for real there using these PERQ workstations which were part of the Spice project, using Spice Lisp which became CMU Common Lisp. It was such an odd environment. We’d go to weekly meetings, learning how software development works just by listening in. But there were some really entertaining characters in that group. Like the guy who was sort of our manager—the one keeping an eye on us—Rob MacLachlan, was this giant blond-haired, barbarianlooking guy. Very intimidating-looking. And he didn’t talk much. I remember a lot of times I’d be sitting there—it was kind of an openplan cubicle kind of thing—working, doing something, writing some Lisp program. And he’d come shuffling in with his ceramic mug of beer, bare feet, and he’d just stand behind me. I’d say hi. And he’d grunt or say nothing. He’d just stand there watching me type. At some point I’d do something and he’d go, “Ptthh, wrong!” and he’d walk away. So that was kind of getting thrown in the deep end. It was like the Zen approach—the master hit me with a stick, now I must meditate.

Seibel: I emailed Fahlman and he said that you were talented and learned very fast. But he also mentioned that you were kind of undisciplined. As he put it, “We tried gently to teach him about working in a group with others and about writing code that you, or someone else, could understand a month from now.” Do you remember any of those lessons?

Zawinski: Not the learning of them, I guess. Certainly one of the most important things is writing code you can come back to later. But I’m about to be 39 and I was 15 at the time, so it’s all a little fuzzy.

Seibel: What year did that start?

Zawinski: That must have been ’84 or ’85. I think I started in the summer between 10th and 11th grade. After high school, at 4:00 or so I’d head over there and stay until eight or nine. I don’t think I did that every day but I was there a fair amount.

Seibel: And you very briefly went to CMU after you finished high school.

Zawinski: Yeah. What happened was, I hated high school. It was the worst time of my life. And when I was about to graduate I asked Fahlman if he’d hire me full-time and he said, “No, but I’ve got these friends who’ve got a startup; go talk to them.” Which was Expert Technologies—ETI. I guess he was on their board. They were making this expert system to automatically paginate the yellow pages. They were using Lisp and I knew a couple of the people already who had been in Fahlman’s group. They hired me and that was all going fine, and then about a year later I panicked: Oh my god, I completely lucked into both of these jobs; this is never going to happen again. Once I no longer work here I’m going to be flipping burgers if I don’t have a college degree, so what I ought to do is go get one of those.

The plan was that I’d be working part-time at ETI and then I’d be going to school part time. That turned into working full-time and going to school full-time and that lasted, I think, six weeks. Maybe it was nine weeks. I know it lasted long enough that I’d missed the add/drop period, so I didn’t get any of my money back. But not long enough that I actually got any grades. So it’s questionable whether I actually went.

It was just awful. When you’re in high school, everyone tells you, “There’s a lot of repetitive bullshit and standardized tests; it’ll all be better once you’re in college.” And then you get to your first year of college and they’re like, “Oh, no—it gets better when you’re in grad school.” So it’s just same shit, different day—I couldn’t take it. Getting up at eight in the morning, memorizing things. They wouldn’t let me opt out of this class called Introduction to Facilities where they teach you how to use a mouse. I was like, “I’ve been working at this university for a year and a half—I know how to use a mouse.” No way out of it—“It’s policy.” All kinds of stuff like that. I couldn’t take it. So I dropped out. And I’m glad I did.

Then I worked at ETI for four years or so until the company started evaporating. We were using TI Explorer Lisp machines at ETI so I spent a lot of my time, besides actually working on the expert system, just sort of messing around with user-interface stuff and learning how those machines worked from the bottom up. I loved them—I loved digging around in the operating system and just figuring out how it all fit together.

I’d written a bunch of code and there was some newsgroup where I posted that I was looking for a job and, oh, by the way, here’s a bunch of code. Peter Norvig saw it and scheduled an interview. My girlfriend at the time had moved out here to go to UC Berkeley, so I followed her out.

Seibel: Norvig was at Berkeley then?

Zawinski: Yeah. That was a very strange job. They had a whole bunch of grad students who’d been doing research on natural language understanding; they were basically linguists who did some programming. So they wanted someone to take these bits and pieces of code they’d left behind and integrate them into one thing that actually worked.

That was incredibly difficult because I didn’t have the background to understand what in the world they were doing. So this would happen a lot: I’d be looking at something; I’d be completely stuck. I have no idea what this means, where do I go from here, what do I have to read to understand this. So I’d ask Peter. He’d be nice about it—he’d say, “It totally makes sense that you don’t understand that yet. I’ll sit down and explain it to you Tuesday.” So now I’ve got nothing to do. So I spent a lot of time working on windows system stuff and poking around with screen savers and just the kind of UI stuff that I’d been doing for fun before.

After six or eight months of that it just felt like, wow, I’m really just wasting my time. I’m not doing anything for them, and I just felt like I was on vacation. There have been times when I was working really a lot when I’d look back at that and I’m like, “Why did you quit the vacation job? What is wrong with you? They were paying you to write screen savers!”

So I ended up going to work for Lucid, which was one of the two remaining Lisp-environment developers. The thing that really made me decide to leave was just this feeling that I wasn’t accomplishing anything. And I was surrounded by people who weren’t programmers. I’m still friends with some of them; they’re good folks, but they were linguists. They were much more interested in abstract things than solving problems. I wanted to be doing something that I could point to and say, “Look, I made this neat thing.”

Seibel: Your work at Lucid eventually gave rise to XEmacs, but when you went there originally were you working on Lisp stuff?

Zawinski: Yeah, one of the first projects I worked on was—I can’t even remember what the machine was, but it was this 16-processor parallel computer and we had this variant of Lucid Common Lisp with some control structures that would let you fork things out to different processors.

I worked a little bit on the back end of that to make the overhead of spawning a thread lower so you could do something like a parallel implementation of Fibonacci that wasn’t just completely swamped by the overhead of creating a new stack group for each thread. I really enjoyed that. It was the first time I’d gotten to use a fairly bizarre machine like that.

Before that I was bringing up Lisp on new machines. Which means basically someone’s already written the compiler back end for the new architecture and then they’ve compiled the bootstrap piece of code. So I’ve got this file full of binary, supposedly executable code for this other machine and now I’ve got to decipher their loader format so that I can write a little C program that will load that in, make the page executable, and jump to it. Then, hopefully, you get a Lisp prompt and at that point you can start loading things in by hand.

Which for every architecture was bizarre, because it’s never documented right. So it’s a matter of compiling a C program and then looking at it byte by byte—byte-editing it in Emacs. Let’s see what happens if I change this to a zero; does it stop running?

Seibel: When you say it wasn’t documented right, was it that it wasn’t documented correctly, or it wasn’t documented at all?

Zawinski: It was usually documented and it was usually wrong. Or maybe it was just three revisions behind—who knows? But at some point you tweak a bit and then it would no longer believe this was an executable file and you had to figure out what was going on there.

Seibel: So that’s something that comes up all the time, from the lowest-level systems programming to high-level APIs, where things just don’t work the way you expect or the way they are documented. How do you deal with that?

Zawinski: Well, you just come to expect it. The sooner you realize that your map is wrong, the sooner you’ll be able to figure out where it went wrong. In my case, I’m trying to produce an executable file. Well, I know the C compiler will produce one. Take the good one and start converting it into the bad one until it stops working. That’s primary tool of reverse engineering.

The hardest bug I’ve ever fixed, I think, was probably during that period at Lucid. I’d gotten to the point where it’s running the executable and it’s trying to bootstrap Lisp and it gets 500 instructions in and crashes. So there I am leaning on the S key, stepping through trying to figure out where it crashes. And it seems to be crashing at a different place each time. And it doesn’t make any sense. I’m reading the assembly output of this architecture I only barely understand. Finally I realize, “Oh my god, it’s doing something different when I step; maybe it’s timing-based.” Eventually I figure out that what’s going on is this is one of the early machines that did speculative execution. It would execute both sides of the branch. And GDB would always take the branch if you single-stepped past a branch instruction. There was a bug in GDB!

Seibel: Nice.

Zawinski: Right. So then that takes me down into, “Oh my god; now I’m trying to debug GDB, which I’ve never looked at before.” The way to get around that is you’re coming up to a branch instruction and you stop before the branch, set a break point on both sides, and continue. So that was how I proved that really was what was going on. Spent like a week trying to fix GDB; couldn’t figure it out. I assume a register was getting stomped somewhere, so it always thought there was a positive value in the branch check or something like that.

So I changed the step-by-instruction command to recognize when it was coming up on a branch instruction and just say, “No, don’t do that.” Then I can just lean on the S key and it would eventually stop and I’d set the break point by hand and continue. When you’re debugging something and then you figure out that not only is the map wrong but the tools are broken—that’s a good time.

Working on Lisp systems was especially weird because GDB was completely nonfunctional on Lisp code because it doesn’t have any debug info—it’s written by a compiler GDB has never heard of. I think on some platforms it laid out the stack frames in a way GDB didn’t understand. So GDB was pretty much an assembly stepper at that point. So you wanted to get out of the GDB world just as quickly as you could.

Seibel: And then you’d have a Lisp debugger and you’d be all set.

Zawinski: Right, yeah.

Seibel: So somewhere in there Lucid switched directions and said, “We’re going to make a C++ IDE”.

Zawinski: That had been begun before I started working there—it was in progress. And people started shifting over from the Lisp side to the Energize side, which is what the development environment was called. It was a really good product but it was two or three years too early. Nobody, at least on the Unix side, had any idea they wanted it yet. Everyone uses them now but we had to spend a lot of time explaining to people why this was better than vi and GCC. Anyway, I’d done a bit of Emacs stuff. I guess by that point I’d already rewritten the Emacs byte compiler because—why did I do that? Right, I’d written this Rolodex phone/address-book thing.

Seibel: Big Brother Database?

Zawinski: Yeah. And it was slow so I started digging into why it was slow and I realized, oh, it’s slow because the compiler sucks. So I rewrote the compiler, which was my first run-in with the intransigence of Stallman. So I knew a lot about Emacs.

Seibel: So the change to the byte compiler, did it change the bytecode format or did it just change the compiler?

Zawinski: It actually had a few options—I made some changes at the C layer, the byte-code interpreter, added a few new instructions that sped things up. But the compiler could be configured to emit either old-style byte-code or ones that took advantage of the new codes.

So I write a new compiler and Stallman’s response is, “I see no need for this change.” And I’m like, “What are you talking about? It generates way faster code.” Then his next response is, “Okay, uh, send me a diff and explain each line you changed.” “Well, I didn’t do that—I rewrote it because the old one was crap.” That was not OK. The only reason it ever got folded in was because I released it and thousands of people started using it and they loved it and they nagged him for two years and finally he put it in because he was tired of being nagged about it.

Seibel: Did you sign the papers assigning the copyright to the Free Software Foundation?

Zawinski: Oh yeah, I did that right away. I think that was probably the first thing in the email. It was like, send me a diff for each line and sign this. So I signed and said, “I can’t do the rest; can’t send you a diff; that’s ridiculous. It’s well documented; go take a look.” I don’t think he ever did.

There’s this myth that there was some legal issue between Lucid and FSF and that’s absolutely not true—we assigned copyrights for everything we did to them. It was convenient for them to pretend we hadn’t at certain times. Like, we actually submitted the paperwork multiple times because they’d be like, “Oh, oh, we seem to have lost it.” I think there was some kind of brouhaha with assignments and XEmacs much later, but that was way after my time.

Seibel: So you started with Lisp. But you obviously didn’t stick with it for your whole career. What came next?

Zawinski: Well, the next language I did any serious programming in after Lisp was C, which was kind of like going back to the assembly I programmed on an Apple II. It’s the PDP-11 assembler that thinks it’s a language. Which was, you know, unpleasant. I’d tried to avoid it for as long as possible. And C++ is just an abomination. Everything is wrong with it in every way. So I really tried to avoid using that as much as I could and do everything in C at Netscape. Which was pretty easy because we were targeting pretty small machines that didn’t run C++ programs well because C++ tends to bloat like crazy as soon as you start using any libraries. Plus the C++ compilers were all in flux—there were lots of incompatibility problems. So we just settled on ANSI C from the beginning and that served us pretty well. After that Java felt like going back to Lisp a bit in that there were concepts that the language wasn’t bending over backwards trying to make you avoid—that were comfortable again.

Seibel: Like what?

Zawinski: Memory management. That functions felt more like functions than subroutines. There was much more enforced modularity to it. It’s always tempting to throw in a goto in C code just because it’s easy.

Seibel: So these days it seems like you’re mostly doing C and Perl.

Zawinski: Well, I don’t really program very much anymore. Mostly I write stupid little Perl scripts to keep my servers running. I end up writing a lot of goofy things for getting album art for MP3s I have—that kind of thing. Just tiny brute-force throw-away programs.

Seibel: Do you like Perl or is it just handy?

Zawinski: Oh, I despise it. It’s a horrible language. But it is installed absolutely everywhere. Any computer you sit down on, you’re never going to have to talk someone through installing Perl to run your script. Perl is there already. That’s really the one and only thing that recommends it.

It has an OK collection of libraries. There’s often a library for doing the thing you want. And often it doesn’t work very well, but at least there’s something. The experience of writing something in Java and then trying to figure out—I myself have trouble installing Java on my computer—it’s horrible. I think Perl is a despicable language. If you use little enough of it, you can make it kind of look like C—or I guess more like JavaScript than like C. Its syntax is crazy, if you use it. Its data structures are a mess. There’s not a lot good about it.

Seibel: But not as bad as C++.

Zawinski: No, absolutely not. It’s for different things. There’s stuff that would be so much easier to write in Perl or any language like Perl than in C just because they’re text-oriented—all these so-called “scripting languages”. Which is a distinction I don’t really buy—“programming” versus “scripting”. I think that’s nonsense. But if what you’re doing is fundamentally manipulating text or launching programs, like running wget and pulling some HTML out and pattern-matching it, it’s going to be easier to do that in Perl than even Emacs Lisp.

Seibel: To say nothing of, Emacs Lisp is not going to be very suitable for command-line utilities.

Zawinski: Yeah, though I used to write just random little utilities in Emacs all the time. There was actually a point, early on in Netscape, where part of our build process involved running “emacs -batch” to manipulate some file. No one really appreciated that.

Seibel: No. I imagine they wouldn’t. What about XScreenSaver—do you still work on that?

Zawinski: I still write new screen savers every now and then just for kicks, and that’s all C.

Seibel: Do you use some kind of IDE for that?

Zawinski: I just use Emacs, mostly. Though recently, I ported XScreenSaver to OS X. The way I did that was I reimplemented Xlib in terms of Cocoa, the Mac graphics substrate, so I wouldn’t have to change the source code of all the screen savers. They’re still making X calls but I implemented the back end for each of those. And that was in Objective C, which actually is a pretty nice language. I enjoyed doing that. It definitely feels Java-like in the good ways but it also feels like C. Because it’s essentially C, you can still link directly with C code and just call the functions and not have to bend over backwards.

Seibel: At Lucid, leaving aside the politics of Emacs development, what technical stuff did you learn?

Zawinski: I definitely became a better programmer while I was there. Largely because that was really the smartest group of people I’ve been around. Everyone who worked there was brilliant. And it was just nice to be in that kind of environment where when someone says, “That’s nonsense,” or “We should do it this way,” you can just take their word for it, believe that they know what they were talking about. That was really nice. Not that I hadn’t been around smart people before. But it was just such a high-quality group of people there, consistently.

Seibel: And how big was the development team?

Zawinski: I think there were like 70 people at the company so probably; I don’t know, 40 or so on the development team. The Energize team was maybe 25 people, 20. It was divided up into pretty distinct areas. There were the folks working on the compiler side of things and the back-end database side of things. The GUI stuff that wasn’t Emacs. And then there was, at one point, me and two or three other people working on integrating Emacs with the environment. That eventually turned into mostly me working on mostly Emacs stuff, trying to make our Emacs 19 be usable, be an editor that doesn’t crash all the time, and actually runs all the Emacs packages that you expect it to run.

Seibel: So you wanted the Emacs included in your product to be a fully capable version of Emacs.

Zawinski: The original plan was that we wouldn’t include Emacs with our product. You have Emacs on your machine already and you have our product and they work together. And you had GCC on your machine already and our product, and they work together. I think one of the early code names for our product was something like Hitchhiker because the idea was that it would take all the tools that you already have and integrate them—make them talk to each other by providing this communication layer between them.

That didn’t work out at all. We ended up shipping our version of GCC and GDB because we couldn’t get the changes upstream fast enough, or at all in some cases. And same thing with Emacs. So we ended up shipping the whole thing. We ended up going down the path of, “Well, we’re replacing Emacs. Shit. I guess we have to do that so we better make it work.” One thing I spent a bunch of time on was making the vi emulation mode work.

Seibel: And that’s several weeks of your life you’re never going to get back.

Zawinski: That’s true, yeah. It was challenging. I think it ended up working OK. The real problem with that wasn’t so much that it was emulating vi wrong as that vi users quit and restart vi all the time. And no amount of coding on my part is going to get them out of that mindset. So they’re like, “I expected this to launch in half a second and it’s taking 14 seconds. That’s ridiculous. I can’t use this.”

Seibel: Why did you leave Lucid?

Zawinski: Lucid was done. There’d been a bunch of layoffs. I sent mail to a bunch of people I know saying, “Hey, looks like I’m going to need a new job soon” and one of those people was Marc Andreessen and he said, “Oh, funny you should mention that, because we just started a company last week.” And that was that.

Seibel: So you went to Netscape. What did you work on there?

Zawinski: I pretty much started right away doing the Unix side of the browser. There had been maybe a few days’ worth of code written on it so far. A little bit more of the Windows and Mac sides had been started. The model was a big pile of back-end code and then as small as possible a piece of front-end code for each of the three platforms.

Seibel: And was this all new code?

Zawinski: It was all new code. Most of the Netscape founders had been NCSA/Mosaic developers so they had written the various versions of NCSA/Mosaic, which was actually three different programs. And all six of those people were at Netscape. They weren’t reusing any code but they had written this program before.

Seibel: So they started with an empty disk and started typing?

Zawinski: Exactly. I never looked at the Mosaic code; still haven’t. We actually were sued over that at one point; the university claimed that we were reusing their code and I guess that was settled one way or the other. There’s always been that rumor that we started that way, but we didn’t.

And really, why would we? Everyone wants to write version two, right? You were figuring it out while you wrote it and now you’ve got a chance to throw that away and start over—of course you’re going to start over. It’s going to be better this time. And it was. With the design that the other ones had, there was basically no way to load images in parallel, things like that. And that was really important. So we had a better design for the back end.

Seibel: Yet that’s also a classic opportunity to fall into the secondsystem syndrome.

Zawinski: It is, it is.

Seibel: How did you guys avoid that?

Zawinski: We were so focused on deadline it was like religion. We were shipping a finished product in six months or we were going to die trying.

Seibel: How did you come up with that deadline?

Zawinski: Well, we looked around at the rest of the world and decided, if we’re not done in six months, someone’s going to beat us to it so we’re going to be done in six months.

Seibel: Given that you picked the date first, you had to rein in scope or quality. How did that work?

Zawinski: We spent a long time talking about features. Well, not a long time, but it seemed like a long time because we were living a week every day. We stripped features, definitely. We had a whiteboard; we scribbled ideas; we crossed them out. This was a group of like six or seven people. I don’t remember exactly the number. A bunch of smart, egotistical people sitting in a room yelling at each other for a week or so.

Seibel: Six or seven being the whole Netscape development team or the Unix development team?

Zawinski: That was the whole client team. There were also the server folks who were implementing their fork of Apache, basically. We didn’t talk to them much because we were busy. We had lunch with them, but that was it. So we figured out what we wanted to be in the thing and we divided up the work so that there were, I guess, no more than two people working on any part of the project. I was doing the Unix side and Lou Montulli did most of back-end network stuff. And Eric Bina was doing layout and Jon Mittelhauser and Chris Houck were doing the Windows front end and Aleks Toti? and Mark Lanett were doing the Mac front end for the pre–version 1.0 team. Those teams grew a little bit after that. But we’d have our meetings and then go back to our cubicles and be heads-down for 16 hours trying to make something work.

It was really a great environment. I really enjoyed it. Because everyone was so sure they were right, we fought constantly but it allowed us to communicate fast. Someone would lean over your cubicle and say, “What the fuck did you check in; that’s complete bullshit—you can’t do it that way. You’re an idiot.” And you’d say, “Fuck off!” and go look at it and fix it and check it in. We were very abrasive but we communicated fast because you didn’t have to go blow sunshine up someone’s ass and explain to them what you thought was wrong—you could say, “Hey, that’s a load of shit! I can’t use that.” And you’d hash it out very quickly. It was stressful but we got it done pretty quickly.

Seibel: Are the long hours and the intensity required to produce software quickly?

Zawinski: It’s certainly not healthy. I know we did it that way and it worked. So the way to answer that question is, is there another example of someone delivering a big piece of software that fast that’s of reasonable quality where they actually had dinner at home and slept during the night? Has that ever happened? I don’t actually know. Maybe it has.

But it’s not always about getting it done as quickly as possible. It also would be nice to not burn out after two years and be able to continue doing your job for ten. Which is not going to happen if you’re working 80-plus hours a week.

Seibel: What is the thing that you worked on that you were most proud of.

Zawinski: Really just the fact that we shipped it. The whole thing. I was very focused on my part, which was the user interface of the Unix front end. But really just that we shipped the thing at all and that people liked it. People converted immediately from NCSA Mosaic and were like, “Wow, this is the greatest thing ever.” We had the button for the What’s Cool page up in the toolbar and got to show the world these crazy web sites people had put up already. I mean, there were probably almost 200 of them! It’s not so much that I was proud of the code; just that it was done. In a lot of ways the code wasn’t very good because it was done very fast. But it got the job done. We shipped—that was the bottom line.

That first night when we put up the .96 beta, we were all sitting around the room watching the downloads with sound triggers hooked up to it—that was amazing. A month later two million people were running software I’d written. It was unbelievable. That definitely made it all worthwhile—that we’d had an impact on people’s lives; that their day was more fun or more pleasant or easier because of the work we’d done.

Seibel: After this relentless pace, at some point that has to start to catch up with you in terms of the quality of the code. How did you guys deal with that?

Zawinski: Well, the way we dealt with that was badly. There’s never a time to start over and rewrite it. And it’s never a good idea to start over and rewrite it.

Seibel: At some point you also worked on the mail reader, right?

Zawinski: In 2.0 Marc comes into my cubicle and says, “We need a mail reader.” And I’m like, “OK, that sounds cool. I’ve worked on mail readers before.” I was living in Berkeley and basically I didn’t come into the office for a couple weeks. I was spending the whole time sitting in cafes doodling, trying to figure out what I wanted in a mail reader. Making lists, crossing it off, trying to decide how long it would take me. What should the UI look like?

Then I came back and started coding. And then Marc comes in again and says, “Oh, so we hired this other guy who’s done mail stuff before. You guys should work together.” It’s this guy Terry Weissman, who was just fantastic—we worked together so well. And it was a completely different dynamic than it had been in the early days with the rest of the browser team.

We didn’t yell at each other at all. And the way we divided up labor, I can’t imagine how it possibly worked or could ever work for anyone. I had the basic design done and I’d started doing a little coding and every day or every couple of days we’d look at the list of features and I’d go, “Uhhh, maybe I’ll work on that,” and he’d go, “OK, I’ll work on that,” and then we’d go away.

Check-ins would happen and then we’d come back and he’d say, “Alright, I’m done with that, what are you doing?” “Uh, I’m working on this.” “OK, well, I’ll start on that then.” And we just sort of divided up the pieces. It worked out really well.

We had disagreements—I thought we had to toss filtering into folders because we just didn’t have time to do it right. And he was like, “No, no, I really think we ought to do that.” And I was like, “We don’t have time!” So he wrote it that night.

The other thing was, Terry and I rarely saw each other because he lived in Santa Cruz and I lived in Berkeley. We were about the same distance from work in opposite directions and because the two of us were the only two who ever needed to communicate, we were just like, “I won’t make you come in if you don’t make me come in.” “Deal!”

Seibel: Did you guys email a lot?

Zawinski: Yeah, constant email. This was before instant messaging—these days it probably all would have been IM because we were sending one-liner emails constantly. And we talked on the phone.

So we shipped 2.0 with the mail reader and it was well-received. Then we’re working on 2.1, which is the version of the mail reader that I’m starting to consider done—this is the one with all the stuff that we couldn’t ship the first time around. Terry and I are halfway through doing that and Marc comes in and says, “So we’re buying this company. And they make a mail-reader thing that’s kind of like what you guys did.” I’m like, “Oh. OK. Well, we have one of those.” And he says, “Well, yeah, but we’re growing really fast and it’s really hard to hire good people and sometimes the way you hire good people is you just acquire another company because then they’ve already been vetted for you.” “OK. What are these people going to be working on?” “They’re going to be working on your project.” “OK, that kind of sucks—I’m going to go work on something else.”

So basically they acquired this company, Collabra, and hired this whole management structure above me and Terry. Collabra has a product that they had shipped that was similar to what we had done in a lot of ways except it was Windows-only and it had utterly failed in the marketplace.

Then they won the start-up lottery and they got acquired by Netscape. And, basically, Netscape turned over the reins of the company to this company. So rather than just taking over the mail reader they ended up taking over the entire client division. Terry and I had been working on Netscape 2.1 when the Collabra acquisition happened and then the rewrite started. Then clearly their Netscape 3.0 was going to be extremely late and our 2.1 turned into 3.0 because it was time to ship something and we needed it to be a major version. So the 3.0 that they had begun working on became 4.0 which, as you know, is one of the biggest software disasters there has ever been. It basically killed the company. It took a long time to die, but that was it: the rewrite helmed by this company we’d acquired, who’d never accomplished much of anything, who disregarded all of our work and all of our success, went straight into second-system syndrome and brought us down.

They thought just by virtue of being here, they were bound for glory doing it their way. But when they were doing it their way, at their company, they failed. So when the people who had been successful said to them, “Look, really, don’t use C++; don’t use threads,” they said, “What are you talking about? You don’t know anything.”

Well, it was decisions like not using C++ and not using threads that made us ship the product on time. The other big thing was we always shipped all platforms simultaneously; that was another thing they thought was just stupid. “Oh, 90 percent of people are using Windows, so we’ll focus on the Windows side of things and then we’ll port it later.” Which is what many other failed companies have done. If you’re trying to ship a cross-platform product, history really shows that’s how you don’t do it. If you want it to really be cross-platform, you have to do them simultaneously. The porting thing results in a crappy product on the second platform.

Seibel: Was the 4.0 rewrite from scratch?

Zawinski: They didn’t start from scratch with a blank disk but they eventually replaced every line of code. And they used C++ from the beginning. Which I fought against so hard and, dammit, I was right. It bloated everything; it introduced all these compatibility problems because when you’re programming C++ no one can ever agree on which ten percent of the language is safe to use. There’s going to be one guy who decides, “I have to use templates.” And then you discover that there are no two compilers that implement templates the same way.

And when your background, your entire background, is writing code where multiplatform means both Windows 3.1 and Windows 95, you have no concept how big a deal that is. So it made the Unix side of things—which thankfully was no longer my problem—a disaster. It made the Mac side of things a disaster. It meant it was no longer possible to ship on low-end Windows boxes like Win16. We had to start cutting platforms out. Maybe it was time to do that, but it was a bad reason. It was unnecessary.

It really felt—this is my bitter, selfish way of looking at it—like Terry and I built this great thing and were punished for our success by having it handed over to idiots. That was a very unhappy time for me at Netscape. That began the period where I was only there waiting to vest.

Seibel: So you lasted there five years?

Zawinski: Yeah. A year past vesting, because just before vesting day mozilla.org started and that was really interesting again, so I stuck around for that.

Seibel: Did you ultimately get dragged into using C++?

Zawinski: Well, there was the Java thing. At one point, we were going to rewrite the browser in Java. We were like, “Yes! We’re going to get to ditch that 4.0 code base which is going to destroy our company and this is gonna actually work because, like, we know what we’re doing!”

And it didn’t work.

Seibel: Did it just not work because Java wasn’t ready?

Zawinski: No. We were all broken up into fairly well-defined groups again. There were three of us working on the mail reader. And we were done. We had a really nice mail reader that was fast and had a lot of really nice features and was better about saving your data—there were never any stalls where it was writing some big file. We took really good advantage of multithreading in Java, which was less painful than I had expected it to be. It was just really pleasant to work on. From the API we had designed we saw all these directions it could grow.

Except the one thing that it couldn’t do was display messages. Because what it did was, it generated HTML and to display HTML you need an HTML display layer, which wasn’t done and was never finished. The layout group just completely went down a rat hole and they were the reason that that project got canceled.

Seibel: So they were presumably wrestling with the—at that time—immature Java GUI technology.

Zawinski: I don’t think so. Because all the chrome worked. There was just this big blank rectangle in the middle of the window where we could only display plain text. They were being extremely academic about their project. They were trying to approach it from the DOM/DTD side of things. “Oh, well, what we need to do is add another abstraction layer here and have a delegate for this delegate for this delegate. And eventually a character will show up on the screen.”

Seibel: Overengineering seems to be a pet peeve of yours.

Zawinski: Yeah. At the end of the day, ship the fucking thing! It’s great to rewrite your code and make it cleaner and by the third time it’ll actually be pretty. But that’s not the point—you’re not here to write code; you’re here to ship products.

Seibel: Folks engaged in overengineering usually say, “Well, once I’ve got this framework in place everything will be easy after that. So I’ll actually save time by doing this.”

Zawinski: That is always the theory.

Seibel: And there are times when that theory is true, when someone has good sense and the framework isn’t too elaborate, and it does save time. Is there any way you can tell which side of the line you’re on?

Zawinski: I know it’s kind of a clich? but it comes back to worse is better. If you spend the time to build the perfect framework that’s going to do what you want and that’s going to carry you from release 1.0 through release 5.0 and everything’s going to be great; well guess what: release 1.0 is going to take you three years to ship and your competitor is going to ship their 1.0 in six months and now you’re out of the game. You never shipped your 1.0 because someone else ate your lunch.

Your competitor’s six-month 1.0 has crap code and they’re going to have to rewrite it in two years but, guess what: they can rewrite it because you don’t have a job anymore.

Seibel: There must have been times, perhaps on a shorter time frame, where you’ve ripped out a big chunk of code because you thought it would be faster to start over.

Zawinski: Yes, there are definitely times when you have to cut your losses. And this always feels wrong to me, but when you inherit code from someone else, sometimes it’s faster to write your own than to reuse theirs. Because it’s going to take a certain amount of time to understand their code and learn how to use it and understand it well enough to be able to debug it. Where if you started from scratch it would take less time. And it might only do 80 percent of what you need, but maybe that’s the 80 percent you actually need.

Seibel: Isn’t it exactly this thing—someone comes along and says, “I can’t understand this stuff. I’ll just rewrite it”—that leads to the endless rewriting you bemoan in open-source development?

Zawinski: Yeah. But there’s also another aspect of that which is, efficiency aside, it’s just more fun to write your own code than to figure out someone else’s. So it’s easy to understand why that happens. But the whole Linux/GNOME side of things is straddling this line between someone’s hobby and a product. Is this a research project where we’re deciding what desktops should look like and we’re experimenting? Or are we competing with Macintosh? Which is it? Hard to do both.

But even phrasing it that way makes it sounds like there’s someone who’s actually in charge making that decision, which isn’t true at all. All of this stuff just sort of happens. And one of the things that happens is everything get rewritten all the time and nothing’s ever finished. If you’re one of those developers, that’s fine because there’s always something to play around with if your hobby is messing around with your computer rather than it being a means to an end—being a tool you use to get whatever you’re actually interested in done.

Seibel: Speaking of messing around with a computer for its own sake, do you still enjoy programming?

Zawinski: Sometimes. I end up doing all the sysadmin crap, which I can’t stand—I’ve never liked it. I enjoy working on XScreenSaver because in some ways screen savers—the actual display modes rather than the XScreenSaver framework—are the perfect program because they almost always start from scratch and they do something pretty and there’s never a version 2.0. There’s very rarely a bug in a screen saver. It crashes—oh, there’s a divide-by-zero and you fix that.

But no one is ever going to ask for a new feature in a screen saver. “I wish it was more yellow.” You’re not going to get a bug report like that. It is what it is. So that’s why I’ve always written those for fun. They make this neat result and you don’t have to think about them too much. They don’t haunt you.

Seibel: And do you enjoy the puzzle of doing some math and figuring out geometry and graphics?

Zawinski: Yeah. What’s this abstract little equation going to look like if I display it this way? Or, how can I make these blobs move around that looks more organic and less rigid, like a computer normally moves things? Stuff like that. What do I do to these sine waves to make it look more like something bouncing?

And then I end up writing all these stupid little shell scripts—selfdefense stuff. I know I could do this by clicking on 30,000 web pages and doing it by hand, but why don’t I write this script—little timesaver things. Which barely feels like programming to me. I know to people who aren’t programmers, that seems like a black art.

I really enjoyed doing the Mac port of the XScreenSaver framework. That was actually writing a lot of new code that required thinking about APIs and the structure of the thing.

Seibel: Was that your APIs—how you were structuring your code?

Zawinski: Both. Both figuring out the existing APIs and figuring out the best way to build a layer between the X11 world and the Mac world. How should I structure that? Which of the Mac APIs is most appropriate? It was the first time in a long time that I’d done something like that and it was just like, “Wow, this is kind of fun. I think I might be kind of good at this.”

It had been forever because I got completely burned out on the software industry. That part of it I just couldn’t take anymore—the politics of it both in the corporate world and in the free-software world. I’d just had too much. I wanted to do something that didn’t involve arguing online about trivia. Or having my product destroyed by bureaucratic decisions that I had no input in.

Seibel: Are you ever tempted to go back and hack on Mozilla?

Zawinski: Nah. I just don’t want to be arguing with people and having pissing matches in Bugzilla anymore. That’s not fun. That kind of thing is necessary to build big products. If it’s something that requires more than one person working on it, which obviously Mozilla does, that’s the way you have to do it. But I don’t look forward to that kind of fight anymore. That’s been beaten out of me by too many years of it. And the other alternative, as a programmer, is go work for someone else. And I don’t have to do that, so I can’t. My first bad day I’d just leave. And were I to start my own company I couldn’t be a programmer—I’d have to run the company.

Seibel: Other than having two million people using your software, what about programming do you enjoy?

Zawinski: It’s a hard question. The problem-solving aspect of it, I guess. It’s not quite like it’s a puzzle—I don’t really play many puzzletype games. Just figuring out how to get from point A to point B—how to make the machine do what you want. That’s the basic element that the satisfaction of programming comes from.

Seibel: Do you find code beautiful? Is there an aesthetic beyond maintainability?

Zawinski: Yeah, definitely. Anything expressed just right, whether it’s being really concise or just capturing it—like anything, a really wellput-together sentence or a little doodle, a caricature that looks just like someone but only used four lines, that kind of thing—it’s the same sort of thing.

Seibel: Do you find that programming and writing are similar intellectual exercises?

Zawinski: In some ways, yeah. Programming is obviously much more rigid. But as far as the overall ability to express a thought, they’re very similar. Not rambling, having an idea in your head of what you’re trying to say, and then being concise about it. I think that kind of thinking is the overlap between programming and writing prose.

It feels like they use similar parts of my brain, but it’s hard to express exactly what it is. A lot of times I’ll read things that just look like bad code. Like most contracts. The really rigid style they use—it’s so repetitive. I look at that and I’m like, why can’t you break this out into a subroutine—which we call paragraphs. And the way they usually begin with definitions, like, so and so, referred to as blah blah blah.

Seibel: Lets talk a little bit about the nitty-gritty of programming. How do you design your code? How do you structure code? Maybe take your recent work on the OS X XScreenSaver as an example.

Zawinski: Well, first I messed around and made little demo programs that never ended up being used again. Just to figure out here’s how you put a window on the screen, and so on. Since I’m implementing X11, the first thing to do is pick one of the screen savers and make the list of all the X11 calls it makes.

Then I create stubs for each of those and then I slowly start filling them in, figuring out how am I going to implement this one, how am I going to implement this one.

At another level, on the Mac side of things, there’s the start-up code. How is the window getting on the screen? And at some point that’s going to call out to the X code. One of the trickier parts of that was really figuring out how to set up the build system to make that work in any kind of sane way. So a bunch of experimentation. Moving things around. At some point, maybe I’d had this piece of code on top and this piece of code being called by it. And then maybe those need to be turned inside out. So there’s a lot of cut-and-pasting until I kind of wrapped my head around a flow of control that seemed sensible. Then I went in and cleaned things up and put things in more appropriate files so this piece of code is together with this piece of code.

That was sort of the broad strokes, building the infrastructure. After that it was just a matter of moving on to the next screen saver; this one uses these three functions that the previous one hadn’t used before, so I’ve got to implement those. And each of those tasks was fairly straightforward. There were some that ended up being really tricky because the X11 API has a ton of options for putting text on the screen and moving rectangles around. So I kept having to make that piece of code hairier and hairier. But most of them were fairly straightforward.

Seibel: So for each of these X11 calls you’re writing an implementation. Did you ever find that you were accumulating lots of bits of very similar code?

Zawinski: Oh, yeah, definitely. Usually by the second or third time you’ve cut and pasted that piece of code it’s like, alright, time to stop cutting and pasting and put it in a subroutine.

Seibel: If you were doing something on the scale of writing a mail reader again, you mentioned starting with a few paragraphs of text and a list of features. Is that the finest granularity that you would get to before you would just start writing code?

Zawinski: Yeah. Maybe there’d be a vague description of the division between library and front end. But probably not. If I was working alone I wouldn’t bother with that because that part is just kind of obvious to me. And then the first thing I would do with something like that is either start at the top or at the bottom. So start with either, put a window on the screen that has some buttons on it, and then dig down and start building the stuff that those buttons do. Or you can start at the other side and start writing the thing that parses mailboxes and that saves mailboxes. Either way. Or both and meet in the middle.

I find that getting something on the screen as soon as possible really helps focus the problem for me. It helps me decide what to work on next. Because if you’re just looking at that big to-do list it’s like, eh, I don’t know which one I should do—does it matter which one I do? But if there’s something you can actually look at, even if it’s just the debug output of your mailbox parser, it’s like, OK, there! That’s something; what’s the next direction this needs to go in? OK, instead of just displaying a tree structure, now maybe I should be emitting HTML or something along those lines. Or parsing the headers in a more detailed way. You just look for the next thing to build on from there.

Seibel: Do you refactor to keep the internal structure of the code coherent? Or do you just have a very good sense at the beginning how it’s all going to fit together?

Zawinski: I usually have a pretty good sense of that. I don’t remember too many occasions where I thought, “Oh, I did this whole thing inside out. I’m going to have to move everything around.” That does happen sometimes.

When I’m just writing the first version of the program, I tend to put everything in one file. And then I start seeing structure in that file. Like there’s this block of things that are pretty similar. That’s a thousand lines now, so why don’t I move that into another file. And the API sort of builds up organically that way. The design process is definitely an ongoing thing; you never know what the design is until the program is done. So I prefer to get my feet wet as early as possible; get something on the screen so I can look at it sideways.

Also, once you start writing the code, you’re gonna realize, “No, that was a dumb idea. Why did I think that this module was going to be really easy when actually it’s way more complicated than I thought?”

Which is something you’re not going to clue into until you actually start writing code and you feel it getting away from you.

Seibel: What are the signs that something is getting away from you?

Zawinski: When you go into something and you have in your head, “Oh, this is going to take me half a day and it’s gonna be a chunk of code this size,” and then you start doing it and you get that sinking feeling like, “Oh, right, I need this other piece too; well, I’d better go off and do that. Oh, and that’s kind of a big problem.”

Seibel: I’ve noticed that one thing that separates good programmers from bad programmers is that good programmers are more facile at jumping between layers of abstraction—they can keep the layers distinct while making changes and choose the right layer to make changes in.

Zawinski: There’s definitely got to be some style in where you decide to put things—it can matter a lot down the line. Being able to just hack it out somewhere up near the user versus making a maybe larger change that may have repercussions down at the bottom—either of those can be the right answer and it is tricky to know which is which.

This change I need to make, is it really one little special case or are there eventually going to be 12 of these? I think one of the most important things, for me anyway, when building something from the ground up like that is, as quickly as possible, getting the program to a state that you, the programmer, can use it. Even a little bit. Because that tells you where to go next in a really visceral way. Once the thing’s on the screen and you’ve got the one button hooked up that does one thing, now you kind of know, which button is next. Obviously that’s a GUI-centric description of what I’m talking about.

Seibel: We talked a bit about some of the really hideous bugs you had to track down like that thing with GDB. But let’s talk a little bit more about debugging. For starters, what’s your preferred debugging tool? Print statements? Symbolic debuggers? Formal proofs of correctness?

Zawinski: That’s changed over the years a lot. When I was using the Lisp machines it was all about running the program and stopping it and exploring the data—there was an inspector tool that let you browse through memory and I changed it so basically the Lisp listener became an inspector. So anytime it printed out an object there was a context menu on it so you could click on this thing here and have that value returned. Just to make it easier to follow around chains of objects and that sort of thing. So early on that was how I thought about things. Getting down in the middle of the code and chasing it around and experimenting.

Then when I started writing C and using GDB inside of Emacs I kind of tried to keep up that same way of doing things. That was the model we built Energize around. And that just never seemed like it worked very well. And as time went by I gradually stopped even trying to use tools like that and just stick in print statements and run the thing again. Repeat until done. Especially when you get to more and more primitive environments like JavaScript and Perl and stuff like that, it’s the only choice you have—there aren’t any debuggers.

People these days seem confused about the notion of what a debugger is. “Oh, why would you need that? What does it do—put print statements in for you? I don’t understand. What are these strange words you use?” Mostly these days it’s print statements.

Seibel: How much of that was due to the differences between Lisp and C, as opposed to the tools—one difference is that in Lisp you can test small parts—you can call a small function you’re not sure is working right and then put a break in the middle of it and then inspect what’s going on. Whereas C it’s like, run the whole program in all of its complex glory and put a break point somewhere.

Zawinski: Lisp-like languages lend themselves more to that than C. Perl and Python and languages like that have a little more of the Lisp nature in that way but I still don’t see people doing it that way very much.

Seibel: But GDB does give you the ability to inspect stuff. What about it makes it not usable for you?

Zawinski: I always found it unpleasant. Part of it is just intrinsic to being C. Poking around in an array and now I’m looking at a bunch of numbers and now I have to go in there and cast the thing to whatever it really is. It just never managed that right, the way a better language would.

Seibel: Whereas in Lisp, if you’re looking at a Lisp array, they’ll just be printed as those things because it knows what they are.

Zawinski: Exactly. It always just seemed in GDB like bouncing up and down, the stack things would just get messed up. You’d go up the stack and things have changed out from under you and often that’s because GDB is malfunctioning in some way. Or, oh well, it was expecting this register to be here and you’re in a different stack frame, so that doesn’t count anymore.

It just always felt like I couldn’t really trust what the debugger was actually telling me. It would print something and, look, there’s a number. Is that true or not? I don’t know. And a lot of times you’d end up with no debug info. So you’re in a stack frame and it looks like it has no arguments and then I’d spend ten minutes trying to remember the register that argument zero goes in is. Then give up, relink and put in a print statement.

It seemed like as time went by the debugging facilities just kept getting worse and worse. But on the other hand now people are finally realizing that manual memory allocation is not the way to go; it kind of doesn’t matter as much any more because the sorts of really complicated bugs where you’d have to dig deep into data structures don’t really happen as often because those, often, in C anyway, were memory-corruption issues.

Seibel: Do you use assertions or other more or less formal ways of documenting or actually checking invariants?

Zawinski: We went back and forth about what to do about assertions in the Netscape code base. Obviously putting in assert statements is always a good idea for debugging and like you said, for documentation purposes. It expressed the intent. We did that a lot.

But then the question is, well, what happens when the assertion fails and you’re in production code; what do you do? And we sort of decided that what you do is return zero and hope it keeps going. Because having the browser go down is really bad—it’s worse than returning to the idle loop and maybe having leaked a bunch of memory or something. That’s going to upset people less than the alternative.

A lot of programmers have the instinct of, “You’ve got to present the error message!” No you don’t. No one cares about that. That sort of stuff is a lot easier to manage in languages like Java that actually have an exception system. Where, at the top loop of your idle state, you just catch everything and you’re done. No need to bother the user with telling them that some value was zero.

Seibel: Did you ever just step through a program—either to debug it or, as some people recommend, to just step through it once you’ve written it as a way of checking it?

Zawinski: No, not really. I only really do stepping when I’m debugging something. I guess sometimes to make sure I wrote it right. But not that often.

Seibel: So how do you go about debugging?

Zawinski: I just eyeball the code first. Read through it until I think, well, this can’t happen because that’s going on right there. And then I put in something to try and resolve that contradiction. Or if I read it and it looks fine then I’ll stop in the middle or something and see where it is. It depends. It’s hard to generalize about that.

Seibel: As far as the assertions—how formally do you think? Some people just use ad hoc assertions—here’s something that I think should be true here. And some people think very formally—functions have preconditions and postconditions and there are global invariants. Where are you on that scale?

Zawinski: I definitely don’t think about things in a mathematically provable way. I’m definitely more ad hoc than that. You know, it’s always helpful when you’ve got inputs to a function to at least have an idea in your head what their bounds are. Can this be an empty string? That sort of thing.

Seibel: Related to debugging, there’s testing. At Netscape, did you guys have a separate QA group or did you guys test everything yourselves?

Zawinski: We did both. We were all running it all the time, which is always your best front-line QA. Then we had a QA group and they had formal tests they went through. And every time there was a new release they’d go down the list and try this thing. Go to this page, click on this. You should see this. Or you shouldn’t see this.

Seibel: What about developer-level tests like unit tests?

Zawinski: Nah. We never did any of that. I did occasionally for some things. The date parser for mail headers had a gigantic set of test cases. Back then, at least, no one really paid a whole lot of attention to the standards. So you got all kinds of crap in the headers. And whatever you’re throwing at us, people are going to be annoyed if their mail sorts wrong. So I collected a whole bunch of examples online and just made stuff up and had this giant list of crappily formatted dates and the number I thought that should turn into. And every time I’d change the code I’d run through the tests and some of them would flip. Well, do I agree with that or not?

Seibel: Did that kind of thing get folded into any kind of automated testing?

Zawinski: No, when I was writing unit tests like that for my code they would basically only run when I ran them. We did a little bit of that later with Grendel, the Java rewrite, because it was just so much easier to write a unit test when you write a new class.

Seibel: In retrospect, do you think you suffered at all because of that? Would development have been easier or faster if you guys had been more disciplined about testing?

Zawinski: I don’t think so. I think it would have just slowed us down. There’s a lot to be said for just getting it right the first time. In the early days we were so focused on speed. We had to ship the thing even if it wasn’t perfect. We can ship it later and it would be higher quality but someone else might have eaten our lunch by then.

There’s bound to be stuff where this would have gone faster if we’d had unit tests or smaller modules or whatever. That all sounds great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, “We’ve got to go from zero to done in six weeks,” well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical. If there’s no unit test the customer isn’t going to complain about that. That’s an upstream issue.

I hope I don’t sound like I’m saying, “Testing is for chumps.” It’s not. It’s a matter of priorities. Are you trying to write good software or are you trying to be done by next week? You can’t do both. One of the jokes we made at Netscape a lot was, “We’re absolutely 100 percent committed to quality. We’re going to ship the highest-quality product we can on March 31st.”

Seibel: That leads to another topic, maintaining software. How do you tackle understanding a piece of code that you didn’t write?

Zawinski: I just dive in and start reading the code.

Seibel: So where do you start? Do you start at page one and read linearly?

Zawinski: Sometimes. The more common thing is learning how to use some new library or toolkit. If you’re lucky there’s some documentation. There’s an API. So you figure out the piece of it you might be interested in using. Or work out how that was implemented. Thread your way through. Or with something like Emacs, maybe start at the bottom. What are cons cells made of? How’s that look? And then skip around from there. Sometimes starting with the build system can give you an idea how things fit together. I always find that a good way to sort of immerse yourself in a piece of code is pick a task you want to accomplish and then try and do it.

With something like Emacs you might do that by taking an existing module and gutting it. OK, now I’ve got this piece of code. Rip out the part that actually does anything and now I’ve got the boilerplate. OK, now I know what a component of this system looks like and I can start putting my stuff back in. Sort of stripping it down to the frame.

Seibel: In Emacs you ended up rewriting the byte-code compiler and bits of the byte-code VM. And we’ve talked about how it’s more fun to rewrite stuff than to fix it, but it’s not always a good idea. I wonder how do you draw that line? Do you think that you chose to rewrite the whole compiler because it was really easier than fixing it more locally? Or was it just, “Hey! It’d be fun to write a compiler.”

Zawinski: It sort of just turned into a rewrite. It started with me just fixing it and trying to add optimizations to it. And then eventually there wasn’t any of the original left. I ended up using the same APIs until then they were gone. I think the byte-code compiler worked out fine. Partly because that was such an isolated module. There’s only one entry point: compile and save.

There was definitely a lot of stuff that I put into Lucid Emacs that was more gratuitous than that. Really, a lot of the stuff I did was motivated by wanting it to be more like Lisp machines. Wanting it to be more like the Emacs I was familiar with. Which really was the Lisp environment I was familiar with. So I put in a lot of stuff to try to make Emacs be a less half-assed Lisp in a lot of ways: there should be event objects instead of a list with a number in it. Having an event object be a list with a number in it—that’s just tasteless. It’s icky. And in retrospect, those changes were some of the biggest problems. Those kind of changes caused compatibility problems with third-party libraries.

Seibel: Of course you didn’t know there were going to be two Emacs at that point.

Zawinski: Sure. But even without that, even if there had only been one Emacs, there were still two Emacs—there was Emacs 18 and Emacs 19. There was still going to be a compatibility problem. In hindsight those were changes that if I’d realized what an impact it was going to make, I probably would have done that differently. Or spent a lot more time on making the old way work as well. That kind of thing.

Seibel: Earlier you said something about writing code in order to make it easier to read, which ties into maintenance. What are the characteristics that make code easier to read?

Zawinski: Well, comments obviously. Writing down what the assumptions are and what this does. If it’s building up a data structure, describing the layout of it. A lot of times I find that pretty helpful. Especially in writing Perl code when it’s like, uh, well, it’s a hash table and values are bunch of references to lists, because the data structures in Perl are just nuts. Do I need a right arrow here to get to this? I find examples like that to be helpful.

I always wish people would comment more, though the thing that makes me cringe is when the comment is the name of the function rephrased. Function’s called push_stack and the comment says, “This pushes to the stack.” Thank you.

You’ve got to say in the comment something that’s not there already. What’s it for? Either a higher-level or a lower-level description, depending on what’s most important. Sometimes the most important thing is, what is this for? Why would I use it? And sometimes the most important thing is, what’s the range of inputs that this expects?

Long variable names. I’m not a fan of Hungarian notation, but I think using actual English words to describe things, except for loop iterators, where it’s obvious. Just as much verbosity as possible, I guess.

Seibel: What about organization—ultimately there’s some linear organization but programs are not really linear. Do you organize your code top-down or bottom-up?

Zawinski: I usually end up putting the leaf nodes up at the top of the file—try to keep it basically structured that way. And then usually up at the top, document the API. What are the top-level entry points of this file, this module, whatever? With an object-y language, that’s done by the language for you. With C you’ve got to be a little more explicit about that. In C I do tend to try to have a .h file for every .c file that has all the externs for it. And anything that’s not exported in the .h file is static. And then I’ll go back and say, “Wait, I need to call that,” and I change it. But you’re doing that explicitly rather than just by accident.

Seibel: You put the leaves first in the file, but is that how you write? Do you build up from leaves?

Zawinski: Not always. Sometimes I start at the top and sometimes I start at the bottom. It depends. One way is, I know I’m going to need these building blocks and I’ll put those together first. Or another way of thinking about it is, you’ve sort of got an outline of it in your head and you dig down. I do it both ways.

Seibel: So suppose for the sake of argument that you were going to come out of retirement and build a development team. How would you organize it?

Zawinski: Well, I think you want to arrange for there to be no more than three or four people working really closely together on a day-today basis. Then that can scale up a lot. Say you’ve got a project where you can divide it up into twenty-five really distinct modules. Well, you can have twenty-five tiny teams—maybe that’s a little much. Say ten. And as long as they can coordinate with each other, I don’t think there’s a whole lot of limit to how big you can scale that. Eventually it just starts looking like multiple projects instead of like one project.

Seibel: So you’ve got multiple teams of up to four people. How do you coordinate the teams? Do you have one grand architect who’s managing the dependencies and mediating between those teams?

Zawinski: Well, there’s got to be agreement about what the interface between modules is. For that very modular approach to work at all, the interface between modules has to be clear and simple. Which, hopefully, means it won’t take too much screaming for everyone to agree on it and it won’t be too difficult to follow the module contract. I guess what I’m getting at is the best way to make interaction between modules be easy is to just make it be really simple. Make there be fewer ways for that to go wrong.

And what lines you divide on depends entirely on a project. With some kind of web app you’ve probably got the UI and you’ve got your database and the part that runs on the server and the part that runs on the machine behind the server. And if it’s a desktop application it’s similar division of labor. There’s file formats and GUI and basic command structure.

Seibel: How do you recognize talent?

Zawinski: That I don’t know. I’ve never really been the person who had to hire people. And when I’ve been involved in interviews I’ve always just felt like I had no idea. I can tell from the interview whether I’d get along with this person, but I can’t tell whether they’re any good or not just by talking to them. I always found that difficult.

Seibel: How about if they’re bad? Are there reliable clues then?

Zawinski: Sometimes. Normally I would think that someone who is a big fan of C++ templates—keep me away from that guy. But that might just be a snap judgment on my part. Maybe in the context they’ve used them, they actually work fine. Certainly with the folks I’ve worked with, ability to argue their point was important because we all ended up being a pretty argumentative bunch. With that environment, that helped a lot. That certainly doesn’t have anything to do with programming ability. That’s just interpersonal-dynamics stuff.

Seibel: And on a different team, that would actually be detrimental.

Zawinski: Yeah, absolutely.

Seibel: It sounds like at Netscape you guys divided things up so people owned different parts of the software. Some people think that’s really important. Other folks say it’s better for a team to collectively own all the code. Do you have an opinion on that?

Zawinski: I’ve done it both ways. They both have their merits. The idea that everyone should own all the code, I don’t think is really practical because there’s going to be too much of it. People are going to have to specialize; you need an expert sometimes. It’s just always going to work out that way. There’s always going to be the code you’re familiar with because you happened to write more of that module than some other guy did. Or there’s just going to be parts that resonate with you more. It’s certainly good for other people to have their hands in it, if only because you’re not going to be maintaining it yourself forever. It’s going to have to be handed off to someone else for one reason or another. And for that knowledge to be spread around is good. But it’s also good to have someone to blame. If everybody is responsible for it then there’s no one to put their foot down.

Seibel: Have you ever been a manager?

Zawinski: Not really. When I was doing the Emacs stuff at Lucid, there would be a lot of modules that were included in Lucid Emacs that were written by other people. Those people didn’t really work for me but it was a little bit like management. And a lot of those people were definitely less experienced and the way that worked out well was they were doing their favorite thing and I was basically giving them feedback: “Well, I want to include this but first I need this, this, and this from it.”

Seibel: And did you give them a free rein? You tell them you want X, Y, and Z and then they get to figure out how to do it?

Zawinski: Yeah. If I’m trying to decide whether to include this module in the thing that I’m going to ship, I’m going to have requirements about it. Does the damn thing work is really the bottom line there. So I would give them advice on, “I think you’re going to have better luck if you try it this rather than this way.” But I wanted it to work and I wanted to not have to be the one to write it. If they wanted to go do it in some crazy way but it worked, that was OK because that gave me point two: I didn’t have to write it. But mostly the feedback I was giving them was just, does it work and does it make sense.

Seibel: On the flip side, when you were the less-experienced programmer, what did your mentors do that was helpful?

Zawinski: I guess the most important thing is they’d recognize when it was time to level up. When I went to work for Fahlman I was given some silly little busy work. And eventually got given tasks that were a little more significant—not that they were significant at all really.

Seibel: I think you talked about Rob MacLachlan, who just hovered and said, “Wrong!” Was that balanced, perhaps, by someone else who was a little more nurturing?

Zawinski: Well, he wasn’t completely a cave man. He would actually tell me things, too. I know I ended up doing a lot of reading of code and asking questions. I think one thing that’s really important is to not be afraid of your ignorance. If you don’t understand how something works, ask someone who does. A lot of people are skittish about that. And that doesn’t help anybody. Not knowing something doesn’t mean you’re dumb—it just means you don’t know it yet.

Seibel: Did you read code mostly because it was something you were working on, or was it just something you wanted to know how it worked?

Zawinski: Yeah. Just poking around—“I wonder how that works.” The impulse to take things apart is a big part of what gets people into this line of work.

Seibel: Were you actually one of those kids who took toasters apart?

Zawinski: Yeah. I made a telephone and learned how to dial with a telegraph tapper that I made out of a tin can. When I was little I had these old books I got at a garage sale or something, like Boy’s OwnScience Book from the ’30s, and I remember getting a really big kick out of those. That was really hacker culture in the ’20s and ’30s where they’re showing how to wire up a telegraph between your room and the barn and how to make Leyden jars.

Seibel: That brings me to another of my standard questions: do you, as a programmer, think of yourself as a scientist or an engineer or an artist or a craftsman or something else?

Zawinski: Well, definitely not a scientist or engineer because those have very formal connotations. I don’t do a lot of math; I don’t draw blueprints; I don’t prove things. I guess somewhere between craftsman and artist, depending on what the project is. I write a lot of screen savers—that’s not craftsman; that’s making pretty pictures. Somewhere in that area.

Seibel: Do you feel like you taught yourself computer science or did you just learn to program?

Zawinski: Well, I certainly picked up a bunch of computer science over the years. But learning to program was the goal. Making the machine do something was the goal and the computer-science side of it was a means to an end.

Seibel: Did you ever feel that as a lack—did you ever wish you had been exposed to things in a more systematic way?

Zawinski: There were definitely times, especially at Lucid, where it’d be obvious that there’s this whole big black hole that these guys are talking about that I just completely missed because I never needed to know it. And then I’d pick up the terminology and have a basic idea what they’re talking about and maybe do a little bit of reading on it if it was something I needed to know. So there were definitely times, especially early on, where I felt like, “Oh my god, I don’t know anything.” It would just be embarrassing—but that was just being insecure. Being the young kid around all these people with PhDs—“Aaah, I don’t know anything! I’m an idiot! How did I bluff my way into this?”

Though my life certainly would have turned out very differently if I had spent a lot more time in school—it was a moment in time when I got to do the things I did.

Seibel: Did you ever feel the opposite, where you felt like the computer scientists around you just didn’t understand actual programming as well as you did?

Zawinski: I felt like that a lot, but really that’s not so much about thinking, “Wow, you guys have been barking up the wrong tree.” as, “Wow, we’re just not interested in the same things.” I don’t want to be a mathematician but I’m not going to criticize someone who is a mathematician.

It’s weird that people often confuse those two pursuits. People who are into very theoretical computer science are thought of in this same way as people who are shipping desktop applications. And they don’t really have a lot to do with each other.

Seibel: You’re largely self-taught. Do you have any advice for selftaught programmers?

Zawinski: That’s a really hard question because the world’s so different now. I always feel weird talking about, “Here’s what I did.” I don’t know if that was the right way to do it. But people always hear it as, “Be like me.”

I stumbled into this—it all just sort of happened. I made some decisions and they led to others and here we are.

Every now and then I get an email from someone that’s basically, “I want to be a programmer; what do I do?” Or, “Should I go to college or not?” How can I answer that? I would have had very strong opinions about this if you asked me in 1986. But someone today couldn’t take the same path that I took because that path doesn’t even exist anymore.

Ten years ago I would have said absolutely the first thing you have to do is learn assembly language. You have to learn how the machine actually works. Does that matter any more? I don’t even know. Maybe it does. But possibly not. If the way software is going to be ten years from now is all web applications or a piece of distributed code in some rented computing cluster that’s moving around between a dozen different Google servers and spawning other copies of itself and then merging back together once it’s got results, does anyone need to know assembly language any more? Is that so abstracted away that it doesn’t matter? I don’t know.

I was kind of freaked out when I realized that there are people graduating with CS degrees who’d never written C. They started in Java and they stayed there. That just seemed bizarre and wrong. But I don’t know. Maybe it’s not wrong. Maybe that’s the caveman thoughts: “Back in my day, we programmed with a nine-volt battery and a steady hand!”

Seibel: What about books? Are there particular computer-science or programming books that everyone should read?

Zawinski: I actually haven’t read very many of those. The one I always recommend is Structure and Interpretation of Computer Programs, which a lot of people are afraid of because it’s Lispy, but I think does a really good job of teaching programming without teaching a language. I think a lot of introductory-level stuff focuses on syntax and I definitely saw that in the classes I had in high school and in the intro classes at Carnegie-Mellon during my brief time there.

This is not teaching people to program; this is teaching people where the semicolon goes. That seems like the kind of thing that’s going to scare people away from it more than anything, because that’s not the interesting part. Not even to someone who knows what they’re doing.

There was another book—what was it called?—about debugging, written by someone from Microsoft. It was about how to use asserts effectively. I remember thinking that was a really good book, not because I learned anything from it, but because it was the book you wish your idiot coworker had read.

Then there was another book that everybody thought was the greatest thing ever in that same period—Design Patterns—which I just thought was crap. It was just like, programming via cut and paste. Rather than thinking through your task you looked through the recipe book and found something that maybe, kinda, sorta felt like it, and then just aped it. That’s not programming; that’s a coloring book. But a lot of people seemed to love it. Then in meetings they’d be tossing around all this terminology they got out of that book. Like, the inverse, reverse, double-back-flip pattern—whatever. Oh, you mean a loop? OK.

Seibel: Is there a key skill programmers must have?

Zawinski: Well, curiosity—taking things apart. Wanting to know what’s going on under the hood. I think that’s really the basis of it. Without that I don’t think you get very far. That’s your primary way of acquiring knowledge. Taking something apart and looking at it is how you learn to build your own. At least for me. I’ve read very few books about computers. My experience has been digging through source code or reference manuals. I’ve got a goal and, alright, to do this I need to know what this thing does and what this thing does. And I’ll just sort of random-walk through that until I find where I’m going.

Seibel: Have you read Knuth’s, The Art of Computer Programming?

Zawinski: I haven’t. And that’s one of those things where, I really probably should have. But I never did.

Seibel: It’s tough going—you need a lot of math to really grok it.

Zawinski: And I’m not a math person at all.

Seibel: That’s interesting. Lots of programmers come out of mathematics and lots of computer-science theory is very mathematical. So you’re an existence proof that it’s not absolutely necessary. How much math or mathy kind of thinking is necessary to be a good programmer?

Zawinski: Well, it depends on where you draw the line as to what’s mathy and what’s not. Is being good at pattern matching mathy? Having an understanding of orders of magnitude and combinatorics is important at a gut level. But I’m sure I would completely flunk if I had to take a basic intro quiz on that kind of stuff. It’s been so long since I’ve had to do anything formal like that.

Really the only math classes I had were in high school. I had algebra. A little bit of calculus. I wasn’t terribly good at it. I got through it but it didn’t really come naturally to me. I had a physics class in high school where we were doing mechanics and doing labs dragging blocks across sandpaper and stuff like that. I did terribly in that class and felt like an idiot because I actually enjoyed the class. I did the labs really well—the procedure was spot on—and then I just couldn’t do the math.

I’d get an answer that I knew was three orders of magnitude off. I’d show my work—I don’t know what I did wrong. I’d get half credit since the data was collected properly and I cleaned up afterwards. So math was never really my forte.

But I wouldn’t go so far as to say you don’t need that to be a programmer. There’s obviously different kinds of programming. Without people who are not like me none of this would exist. But I’ve always seen much more in common with writing prose than math. It feels like you’re writing a story and you’re trying to express a concept to a very dumb person—the computer—who has a limited vocabulary. You’ve got this concept you want to express and limited tools to express it with. What words do you use and what does your introductory and summary statement look like? That sort of thing.

The issue of taste really fits in there. You can have a piece of text describing something that describes it correctly or it can describe it well, with some flair. And the same thing’s true of programs. It can get the job done or it can also make sense, be put together well.

Seibel: And why does that matter? Is that just for the satisfaction of it or is tasteful code also better in some practical way?

Zawinski: To a large degree, tasteful and maintainable are similar. Or very closely related. One of the things that makes a piece of writing tasteful is if it’s structured in a way that’s easy to grasp. Are the facts loaded up at the front or are they scattered around? If you’re referring back—if you’re flipping through a book, can you figure out where in the book is the thing you kind of remember? “This was somewhere near the middle because that’s where he talked about this thing.” Or is it just scattered all through. And that’s the same sort of thing that goes on with programming a lot.

Seibel: Do you think the kind of people who can be successful at programming has changed?

Zawinski: Certainly these days it’s impossible to just write a program from scratch that doesn’t have any dependencies. The explosion of toolkits and libraries and frameworks and that sort of thing—even the most basic piece of software needs those these days. It’s just exploded. These days, everything’s got to be a web app. And that’s just a whole different way of going about it.

So, if anything, that makes the part of the skill set that is being able to dive into someone else’s code and figure out how to make use of it even more important. “I don’t understand this, so I’m going to write my own” worked better in the past. Whether it was ever a good idea or not, you could do it. It’s much harder to get away with that now.

Seibel: I wonder if the inclination to take things apart and understand everything also needs to be a little more tempered these days. If you try to take apart every piece of code you work with, it’ll never end—these days you’ve got to have a little capacity for saying, “I sort of understand how this works and I’m going to let it go at that until it becomes urgent that I understand it better.”

Zawinski: Yeah. My first instinct, because things work that way, is you’re breeding a generation of programmers who don’t understand anything about efficiency or what’s actually being allocated. When they realize, “Oh, my program’s getting gigantic,” what are they going to do? They’re not going to know where to start. That’s my first instinct because I’m a caveman. Really that probably doesn’t even matter because you’ll just throw more memory at it and it’ll be fine.

Seibel: Or perhaps people will actually learn a more sophisticated view of what all those things mean. Like, maybe it doesn’t really matter whether we allocated six bytes or four bytes here—what matters is whether we’ve sized this thing so it fits in one node of the cluster versus having to spill over onto a second node.

Zawinski: Right, exactly. I think programming has definitely changed in that sense. The things you had to focus on before were different. Before you would focus on counting bytes, and “How big are my objects? Maybe I should do something different here because that array header is really going to add up.” Things like that. No one is ever going to care about that stuff again. Tricks like XORing your forward and back pointers into the same word are voodoo—why would anyone do that; it’s crazy. But there’s this whole different set of skills now that were always around but come more to the front. People who can dig into an API and figure out which parts you need and which parts you don’t, is, I think, one of those important things now.

Seibel: If you were 13 today, would you be drawn to programming the way programming is today?

Zawinski: So hard to say. I don’t know any 13-year-olds. I don’t know what the world looks like. Things are harder to take apart now. There’s not going to be some 10-year-old who pops open his cell phone and figures out how the speaker works like I did with a phone when I was a little kid. There are no user-serviceable parts anymore.

I feel like that kind of tinkering, taking the back of the tape deck off and seeing how the gears fit together, that sort of exploration is what led to this for me. Aside from things like LEGO Mindstorms, I don’t think there’s a lot of opportunity for people to follow that path these days. But maybe I’m wrong—like I said, I don’t know any 13-year-olds. I don’t know what the toys are like. There’s a lot of video games; there’s a lot of things with remote controls. I haven’t seen any really good construction kinds of toys. Which seems sad.

Seibel: On the flip side, programming itself is much more accessible. You don’t have to master all the intricacies of assembly programming right off the bat just to make a computer do something neat.

Zawinski: Yeah. I imagine that today’s kids who are getting into programming start off building some web app or writing a Facebook plugin or something. Brad Fitzpatrick, who wrote LiveJournal, is a friend of mine. When he wrote LiveJournal he was goofing around and wrote this Perl script where he and his friends could say, “I’m going to get lunch.” The way he started out was he wrote a little Perl script and put it on a web server. Probably things will go more in that direction.

Оглавление книги