Книга: Coders at Work: Reflections on the craft of programming

Brad Fitzpatrick

Brad Fitzpatrick

Brad Fitzpatrick is the youngest person I interviewed and the only one who has never lived in a world without the Internet or personal computers. Born in 1980, he got an early start as a programmer, learning to program at age five on a homebuilt Apple II clone. By his teenage years the Internet revolution was in full swing and he was deep into it, building his first commercial web site while still in high school and starting work on the popular community site LiveJournal the summer before he went to college.

Keeping up with LiveJournal’s ever-growing popularity forced Fitzpatrick to learn the hard way about building scalable web sites and along the way he and the programmers at the company he founded, Danga Interactive, ended up building several pieces of open source software, including memcached, Perlbal, and MogileFS, which are now used on the servers of many of the world’s busiest web sites.

Fitzpatrick is a prototypical—if exceptionally accomplished—turn-of-the-century web programmer: his primary programming languages have been Perl and C, though he also works in Java, C++, Python, JavaScript, and C# as needed. And almost all the programming he does is somehow network-related, whether it’s building better back-end infrastructure for web sites, designing protocols and software to improve the way blog-reading software knows when blogs have been updated, or programming his cell phone to automatically open his garage door when he rides up on his motorcycle.

We talked about learning to program at the same age as he was reading Clifford the Big Red Dog, why he was glad he stayed in college while running LiveJournal, and how he learned not to be afraid of reading other people’s code.

Seibel: How did you become a programmer?

Fitzpatrick: My dad was working at Mostek. They made memory and he was into computers. He built an Apple II from spare parts, pretty much. He and my mom would sit around the TV soldering it all together. It took them months, just soldering it. Then he was able to get ROMs from work that they weren’t going to sell because they had a bit, or multiple bits, stuck high or stuck low. Somehow they got the Apple II ROM and they just kept burning it onto these dead chips until they got one that worked, where the stuck bits just happened to be right. Eventually he and a bunch of his coworkers managed to make homemade Apple IIs. I was playing on that from age two or so and watching him program.

Seibel: Was he a programmer or a hardware guy?

Fitzpatrick: He was an electrical engineer; he dabbled in programming. He taught me to program when I was five and jokes that I passed him up around six or seven. My mom says I was reading the Apple II programmers’ manual from the library at the same time as Clifford the Big Red Dog. Instead of “variables,” I would say, “valuables.” Some of my first memories are programming with my dad. Like he pulled me into the kitchen and he was writing down a program on paper. He asked, “What do you think it does?” I remember it was like, “10 PRINT HELLO, 20 GOTO 10.”

Seibel: So you started with BASIC?

Fitzpatrick: Yeah, that was BASIC. I couldn’t do stuff with the mouse, or stuff with higher graphics modes and colors, until a friend of our family introduced me to C and gave me Turbo C. This was maybe when I was eight or ten. My dad moved to Intel in ’84 and we moved to Portland. He helped design the 386 and 486. He’s still at Intel. We always had new, fun computers.

Seibel: Did you get into assembly programming at all?

Fitzpatrick: I did assembly a little on calculators. Like Z80 on the TI calculators, but that was about it.

Seibel: Do you remember what it was that drew you to programming?

Fitzpatrick: I don’t know. It was just always fun. My mom had to cut me off and give me computer coupons to make me go outside and play with friends. My friends would come over: “Brad’s on the computer again. He’s so boring.” My mom’s like, “Go outside and play.”

Seibel: Do you remember the first interesting program that you wrote?

Fitzpatrick: We had this Epson printer and it came with big, thick manuals with a programmers’ reference at the end. So I wrote something—this was back on an Apple—where I could draw something in the high graphics mode, and then, once my program finished drawing whatever it was drawing—lines or patterns or something—I’d hit control C and be typing in the background, in a frame buffer that’s not showing, and load my other program, which read the screen off and printed it.

Before that I remember writing something that every time I hit a key, it moved the head and I had wired backspace up to go backwards so as I typed it felt like a typewriter.

This was one of my first programs—it was something like K equals grab the next char. Then I said if K equals “a”, print “a”; if K is “b”, print “b”. I pretty much did every letter, number, and some punctuation. Then at one point I was like, “Wait, I could just say, ‘Print the variable!’” and I replaced 40 lines of code with one. I was like, “Holy shit, that was awesome!” That was some major abstraction for a six-year-old.

Those are the notable early ones. Then in middle school I would make games and I would make the graphics editors and the level editors for my friends, and my friends would make the graphics into levels, and then we would sell it to our classmates. I remember having to make games that detected EGA versus VGA. If one of ’em failed on VGA, it would fault back to EGA and use a different set of tiles that fit on the screen, so we’d have to have two sets of graphics for everything. People from school would buy it for like five bucks and they would go to install it and it wouldn’t work, and their parents would call my parents and yell, “Your son stole five dollars from my kid for this crap that doesn’t work.” My mom would drive me over there and sit in the cul-de-sac while I went in and debugged it and fixed it.

Seibel: During that time did you take any classes on programming?

Fitzpatrick: Not really. It was all one or two books from the library, and then just playing around. There weren’t really forums or the Internet. At one point I got on a BBS, but the BBS didn’t really have anything on it. It wasn’t connected to the Net, so it was people playing board games.

Seibel: Did your school have AP computer science or anything?

Fitzpatrick: Well, we didn’t have AP C.S., but we had a computer programming class. There was a guy teaching it but then I would teach sort of an advanced class in the back. They still use the graphics editor and the graphics library I wrote—their final project is to make a game. I still occasionally run into that C.S. teacher—he’s a friend of my family’s and I’ll see him at my brother’s soccer games—he’ll be like, “Yep, we still use your libraries.”

I did take the AP C.S. test. It was the last year it was in Pascal before they switched to C, which was one year before they switched to Java or something like that. I didn’t know Pascal so I went to a neighboring high school that had AP C.S. and I went to some night classes, like three or four of them. Then I found a book and learned the language, and I spent most of my time building asteroids in Pascal because I had just learned trig. I was like, “Oooh, sin and cosin; these are fun. I can get thrust and stuff like that.”

Seibel: How’d you do?

Fitzpatrick: Oh, I got a five. I had to write bigint classes. Now that’s one of the interview questions I give people. “Write a class to do arbitrary, bigint manipulation with multiplication and division.” If I did it in high school on an AP test, they should be able to do it here.

Seibel: Your freshman year in college you worked at Intel during the summer. Did you also work as a programmer during high school?

Fitzpatrick: Yeah, I worked at Tektronix for a while. Before I had any official job, I got some hosting account. I got kicked off of AOL for writing bots, flooding their chat rooms, and just being annoying. I was scripting the AOL client from another Windows program. I also wrote a bot to flood their online form to send you a CD. I used every variation of my name, because I didn’t want their duplicate suppression to only send me one CD, because they had those 100 free hours, or 5,000 free hours. I submitted this form a couple thousand times and for a week or so the postman would be coming with bundles of CDs wrapped up.

My mom was like, “Damn it, Brad, you’re going to get in trouble.” I was like, “Eh—their fucking fault, right?” Then one day I get a phone call and I actually picked up the phone, which I normally didn’t, and it was someone from AOL. They were just screaming at me. “Stop sending us all these form submissions!” I’m not normally this quick and clever, but I just yelled back, “Why are you sending me all this crap? Every day the postman comes! He’s dropping off all these CDs!” They’re like, “We’re so sorry, sir. It won’t happen again.” Then I used all those and I decorated my dorm room in college with them. I actually still have them in a box in the garage. I can’t get rid of them because I just remember them being such a good decoration at one point.

After I got kicked off of AOL, I got a shell account on some local ISP. That’s basically where I learned Unix. I couldn’t run CGI scripts, but I could FTP up, so I would run Perl stuff on my desktop at home to generate my whole website and then upload it. Then I got a job at Tektronix, like a summer intern job. I knew Perl really well and I knew web stuff really well, but I had never done dynamic web stuff. This was probably ’95, ’94—the web was pretty damn new.

Then I go to work at Tektronix and on my first day they’re introducing me to stuff, and they’re like, “Here’s your computer.” It’s this big SPARCstation or something running X and Motif. And, “Here’s your browser.” It’s Netscape 2 or something—I don’t remember. And, “If you have some CGIs, they go in this directory.” I remember I got a basic hello-world CGI, like three lines working that night and I was like, “Holy shit, this is so fun.” I was at work the next day at six in the morning and just going crazy with CGI stuff.

Then I started doing dynamic web-programming stuff on my own. Maybe at that point I had found a web server for Windows that supported CGI. I finally convinced my ISP—I’d made friends with them enough, or sent enough intelligent things that they trusted me—so they said, “OK, we’ll run your CGIs but we’re going to audit them all first.” They’d skim them and toss them in their directory. So I started running this Voting Booth script where you created a topic like, “What’s your favorite movie?” and you could add things to it and vote them up. That got more and more popular. That was going on in the background for a couple of years.

Seibel: That was FreeVote?

Fitzpatrick: Yeah, that turned into FreeVote after it flooded my host. Banner ads were really popular then, or they were just getting really popular, and I kept getting more and more money from that, better contracts, more cost per click. At the height I was getting 27 cents per click of banner ads, which I think is pretty ridiculous even by today’s standards. So at the height, I was making like $25, $27 grand per month on fucking clicks on banner ads.

This was all through high school—I did this in the background all of high school. And I worked at Intel two summers, and then started doing LiveJournal my last summer, right before college. So then my first year of college, I was just selling FreeVote, which I basically sold for nothing to a friend, for like $11 grand just because I wanted to get rid of it and get rid of legal responsibility for it.

Seibel: When you got on your ISP and got to use Unix, did that change your programming much?

Fitzpatrick: Yeah. It didn’t drive me crazy. I couldn’t understand what was going on with Windows. You’ve probably seen the Windows API—there are like twenty parameters to every function and they’re all flags and half of them are zero. No clue what’s going on. And you can’t go peek underneath the covers when something’s magically not working.

Seibel: Are there big differences you can identify between your early approach to programming or programming style to the way you think about programming now?

Fitzpatrick: I went through lots of styles, object-oriented stuff, and then functional stuff, and then this weird, hybrid mix of object-oriented and functional programming. This is why I really love Perl. As ugly as the syntax is and as much historical baggage and warts as it has, it never fucks with me and tells me what style to write in. Any style you want is fine. You can make your code pretty and consistent, but there’s no language-specified style. It’s only since I’ve been at Google that I’ve stopped writing much Perl.

I’ve also done a lot of testing since LiveJournal. Once I started working with other people especially. And once I realized that code I write never fucking goes away and I’m going to be a maintainer for life. I get comments about blog posts that are almost 10 years old. “Hey, I found this code. I found a bug,” and I’m suddenly maintaining code.

I now maintain so much code, and there’s other people working with it, if there’s anything halfway clever at all, I just assume that somebody else is going to not understand some invariants I have. So basically anytime I do something clever, I make sure I have a test in there to break really loudly and to tell them that they messed up. I had to force a lot of people to write tests, mostly people who were working for me. I would write tests to guard against my own code breaking, and then once they wrote code, I was like, “Are you even sure that works? Write a test. Prove it to me.” At a certain point, people realize, “Holy crap, it does pay off,” especially maintenance costs later.

Seibel: When did you start working with other people?

Fitzpatrick: It was pretty much towards the end of college when I started hiring other people, and especially once I moved back to Portland after college.

Then the early employees were customer support, so they didn’t write any code. Then slowly I started hiring programmers. The first person I hired was a friend of mine from online. His name is Brad Whitaker and we both had websites called BradleyLand or BradleyWorld, so we found each other’s websites. I was a couple of years ahead of him web programming–wise, or maybe a year, and he was asking me, “Hey, how do you that,” whether it was HTML, or frames, or CGI, or Perl stuff. So then I started getting a bunch of contract projects and I would give the ones I didn’t want to him. And then we had a project that was too big for either of us so we told the guy, “It’s going to take two people to do this project.” And he flew us out to Pennsylvania. Pittsburgh? I don’t know the East coast at all; I’m a Westcoast guy. Philadelphia? The cheesesteak place.

Seibel: Philadelphia.

Fitzpatrick: Yeah, and we met for the first time at some cheapo hotel and it felt like I knew him already. He was like, “Hey, what up?” He came in and took a piss in my hotel bathroom without even closing the door as I’m standing right there. I’m like, “Alright. You’re comfortable.” It was like we knew each other for four or five years, even though we had never met. We started working on this stuff together.

He moved up into my spare bedroom and we basically moved all the stuff out of my kitchen, set up a bunch of tables, and worked on computers. We would wake up around 10 or 11 and work until noon, and watch some TV—sit around in our boxers and watch TV, and hack, and stay up until 3 or 4 in the morning just working nonstop. Then another friend of mine moved down for the summer from UW. This was after my freshman year in college and then there were three of us working there. The third friend was living downtown. He would come on the light rail in the morning and skateboard over to my house. He would sit outside on Wi-Fi, just hacking until we woke up, opened the door, and let him in.

Once there were three of us, it was a little crowded in my house, so I was like, “Oh, OK, let’s get an office.” So we got an office and we were like, “Oh, we have all this space! Let’s hire more people.” We slowly got up to 12 over the next couple of years, and LiveJournal got more and more popular, and then more stressful too, because I was dealing with HR. Or my mom was dealing with HR and my mom was fighting with me because she worked for me. I had to make rules for my mom, like, “If you call me, it has to be personal or business; whatever one you start with, that’s how you end it. You can’t switch from work to personal or personal to work.” I just started hanging up on her if she switched. Then she’d call back and I’m like, “Nope, you lost.” So that was really stressful. She was really happy when I sold it, and she could stop working for me and we could stop fighting.

Seibel: Was your company still doing contract work or was it all LiveJournal at that point?

Fitzpatrick: It was pretty much all LiveJournal. We were also trying to start a photo hosting-service, which Flickr beat us to. Ours was probably overdesigned: beautifully abstract and plugged into everything. But each new infrastructure thing we did for LiveJournal, we were like, “How is this going to work on FotoBilder?” so we started building everything abstract. Memcached was abstract because there was no reason to tie it into LiveJournal. Then we built a file system like GFS, and we built a job queue. So we kept building all these infrastructure components for scaling that would work for either of our products, but also because the less intertwined spaghetti dependency-wise, the easier it is to maintain something. Even if it’s a little bit more work, if you can cut some dependencies, it was great, so we started building all that generic infrastructure.

Seibel: I’m curious about the process that you went through of scaling LiveJournal, in terms of where you started and how you learned the lessons you needed to learn along the way.

Fitzpatrick: So it started on one shared Unix box with other customers and pretty much killed that.

Seibel: Running as CGIs?

Fitzpatrick: Yes. Yeah. I think it was probably a literal CGI, fork up the whole world and die. There was a guy assigned to me at this ISP. I was having problems with my server dying all the time. I’m like, “I paid my $10.00 a month. Why isn’t it working?” So he would say, “Oh, do this.” Pretty soon I was learning Unix and learning what was actually going on.

Then I converted to FastGCI. Then I tuned Apache and turned off reverse DNS lookups. All these steps you go through. Finally, I was I/O-bound or CPU-bound. Then I got my own dedicated server, but it was still just one and it was dying and I was out of capacity. I had originally opened it up for my friends and I just left the signup page alive. Then they invited their friends who invited their friends—it was never really supposed to be a public site. It just had an open signup page on accident. So then I put something up on the LiveJournal news page and I said, “Help. We need to buy servers.”

I think that raised maybe six or seven thousand dollars or something to buy these two big Dells and put them in Speakeasy in downtown Seattle. Somebody recommend some servers, Dells, these huge 6U things, like ninety pounds each. The logical split was the database server and the web server. That was the only division I knew because I was running a MySQL process and an Apache process.

That worked well for a while. The web servers spoke directly to the world and had two network cards and had a little crossover cable to the database server. Then the web server got overloaded, but that was still fairly easy. At this point I got 1U servers. Then we had three web servers and one database server. At that point, I started playing with three or four HTTP load balancers—mod_backhand and mod_proxy and Squid and hated them all. That started my hate for HTTP load balancers.

The next thing to fall over was the database, and that’s when I was like, “Oh, shit.” The web servers scale out so nicely. They’re all stateless. You just throw more of them and spread load. So that was a long stressful time. “Well, I can optimize queries for a while,” but that only gives you another week until it’s loaded again. So at some point, I started thinking about what does an individual request need.

That’s when—I thought I was the first person in the world to think of this—I was like, we’ll shard it out—partition it. So I wrote up design doc with pictures saying how our code would work. “We’ll have our master database just for metadata about global things that are low traffic and all the per-blog and per-comment stuff will be partitioned onto a per-user database cluster. These user IDs are on this database partition.” Obvious in retrospect—it’s what everyone does. Then there was a big effort to port the code while the service was still running.

Seibel: Was there a red-flag day where you just flipped everything over?

Fitzpatrick: No. Every user had a flag basically saying what cluster number they were on. If it was zero, they were on the master; if it was nonzero, they were partitioned out. Then there was a “Your Account Is Locked” version number. So it would lock and try to migrate the data and then retry if you’d done some mutation in the meantime—basically, wait ’til we’ve done a migration where you hadn’t done any write on the master, and then pivot and say, “OK, now you’re over there.”

This migration took months to run in the background. We calculated that if we just did a straight data dump and wrote something to split out the SQL files and reload it, it would have taken a week or something. We could have a week of downtime or two months of slow migration. And as we migrated, say, 10 percent of the users, the site became bearable again for the other ones, so then we could turn up the rate of migration off the loaded cluster.

Seibel: That was all pre-memcached and pre-Perlbal.

Fitzpatrick: Yeah, pre-Perlbal for sure. Memcached might have come after that. I don’t think I did memcached until like right after college, right when I moved out. I remember coming up with the idea. I was in my shower one day. The site was melting down and I was showering and then I realized we had all this free memory all over the place. I whipped up a prototype that night, wrote the server in Perl and the client in Perl, and the server just fell over because it was just way too much CPU for a Perl server. So we started rewriting it in C.

Seibel: So that saved you from having to buy more database servers.

Fitzpatrick: Yeah, because they were expensive and slow to migrate. Web servers were cheap and we could add them and they would take effect immediately. You buy a new database and it’s like a week of setup and validation: test its disks, and set it all up and tune it.

Seibel: So all the pieces of infrastructure you built, like memcached and Perlbal, were written in response to the actual scaling needs of LiveJournal?

Fitzpatrick: Oh, yeah. Everything we built was because the site was falling over and we were working all night to build a new infrastructure thing. We bought one NetApp ever. We asked, “How much does it cost?” and they’re like, “Tell us about your business model.” “We have paid accounts.” “How many customers do you have? What do you charge?” You just see them multiplying. “The price is: all the disposable income you have without going broke.” We’re like, “Fuck you.” But we needed it, so we bought one. We weren’t too impressed with the I/O on it and it was way too expensive and there was still a single point of failure. They were trying to sell us a configuration that would be high availability and we were like, “Fuck it. We’re not buying any more of these things.”

So then we just started working on a file system. I’m not even sure the GFS paper had published at this point—I think I’d heard about it from somebody. At this point I was always spraying memory all over just by taking a hash of the key and picking the shard. Why can’t we do this with files? Well, files are permanent. So, we should record actually where it is because configuration will change over time as we add a more storage nodes. That’s not much I/O, just keeping track of where stuff is, but how do we make that high availability? So we figured that part out, and I came up with a scheme: “Here’s all the reads and writes we’ll do to find where stuff is.” And I wrote the MySQL schema first for the master and the tracker for where the files are. Then I was like, “Holy shit! Then this part could just be HTTP. This isn’t hard at all!”

I remember coming into work after I’d been up all night thinking about this. We had a conference room downstairs in the shared office building—a really dingy, gross conference room. “All right, everyone, stop. We’re going downstairs. We’re drawing.” Which is pretty much what I said every time we had a design—we’d go find the whiteboards to draw.

I explained the schema and who talks to who, and who does what with the request. Then we went upstairs and I think I first ordered all the hardware because it takes two weeks or something to get it. Then we started writing the code, hoping we’d have the code done by the time the machines arrived.

Everything was always under fire. Something was always breaking so we were always writing new infrastructure components.

Seibel: Are there things that if someone had just sat you down at the very beginning and told you, “You need to know X, Y, and Z,” that your life would have been much easier?

Fitzpatrick: It’s always easier to do something right the first time than to do a migration with a live service. That’s the biggest pain in the ass ever. Everything I’ve described, you could do on a single machine. Design it like this to begin with. You no longer make assumptions about being able to join this user data with this user data or something like that. Assume that you’re going to want to load these 20 assets—your implementation can be to load them all from the same table but your higher-level code that just says, “I want these 20 objects” can have an implementation that scatter-gathers over a whole bunch of machines. If I would have done that from the beginning, I’d have saved a lot of migration pain.

Seibel: So basically the lesson is, “You have to plan for the day when your data doesn’t all fit into one database.”

Fitzpatrick: Which I think is common knowledge nowadays in the web community. And people can go overkill on assuming that their site is going to be huge. But at the time, the common knowledge was, Apache is all you need and MySQL is all you need.

Seibel: It does seem that while you were writing all this stuff because you needed it, you also enjoyed doing it.

Fitzpatrick: Oh, yeah. I definitely try to find an excuse to use anything, to learn it. Because you never learn something until you have to write something in it, until you have to live and breathe it. It’s one thing to go learn a language for fun, but until you write some big, complex system in it, you don’t really learn it.

Seibel: So what languages would you say you’ve really lived and breathed with enough to claim as your own?

Fitzpatrick: Perl. C. Back in the day, BASIC, but I’m not even sure BASIC counts. I wrote a lot of Logo too. In our Logo class in elementary school, people were doing pen up, pen down and I would be not in graphics mode—there’s some key to get out of graphics mode—writing functions. My teacher would come over and say, “What are you doing? You’re doing the wrong thing. You’re supposed to be drawing houses.” “No, I’m writing Logo. Look,” “No, you’re not.” Then at the end of the class I’d do something—I had a library that drew every letter of the alphabet, but at arbitrary scales and rotations. So I could print entire messages on wavy banners going into the distance and stuff, and everyone was like, “What the fuck?” I don’t know if that one counts either.

But a lot of Perl and C, and then a lot of C++ in college for work and for Windows stuff. Then I forgot C++, or it atrophied, and now at Google, in the last year, it’s a lot of C++, Python, and Java. I also wrote a lot of Java back in the day when it first came out, but then I got sick of it. Now I’m writing a lot of Java again, and I’m kinda sick of it.

Seibel: Does it matter much to you what language you use?

Fitzpatrick: I’m still not happy with any of them. I don’t know what exactly would make me totally happy. I hate that for a given project you have to jump around all the time. I want something that lets me have static types and checks all that stuff at compile time, when I want. Perl gets me pretty close in that it lets me write in any style I want. It doesn’t let me do enough static checking at compile time but I can make it blow up pretty hard when I want to a runtime. But it’s still not good enough.

I want optional static typing. In Perlbal, there’s no reason for half the things to be performant except for the core, copying bytes around. I would like to give the runtime hints in certain parts of the code and declare types. But if I want to be lazy and mock something out, I want to write in that style.

Seibel: So you want types mostly so the compiler can optimize better?

Fitzpatrick: No. I also want it to blow up at compile time to tell me like, “You’re doing something stupid.” Then sometimes I don’t care and I want it to coerce for me at runtime and do whatever. I don’t want to be too optimistic about Perl 6, but they’re preaching a lot of things I want to see. But I don’t think it’ll ever come out.

Seibel: Do you like C++?

Fitzpatrick: I don’t mind it. The syntax is terrible and totally inconsistent and the error messages, at least from GCC, are ridiculous. You can get 40 pages of error spew because you forgot some semicolon. But—like anything else—you quickly memorize all the patterns. You don’t even read the words; you just see the structure and think, “Oh, yeah, I probably forgot to close the namespace in a header file.” I think the new C++ spec, even though it adds so much complexity, has a lot of stuff that’ll make it less painful to type—as far as number of keystrokes. The auto variables and the for loops. It’s more like Python style. And the lambdas. It’s enough that I could delude myself into thinking I’m writing in Python, even though it’s C++.

Seibel: And you use C++ for efficiency.

Fitzpatrick: Yeah, pretty much. I mostly use it at Google. Anything that’s halfway performant is in C++ there. I also write a ton of Java at Google.

Seibel: From what I understand, Google has a C++-centric culture because that’s what they used originally and they’ve built a whole bunch of software infrastructure around it. While you can’t undo all that history, there’s probably a lot of code written at Google in C++ where it’s not really necessary for performance.

Fitzpatrick: Especially because, over time, Java has gotten faster and the JVM has gotten a lot smarter. The thing that annoys me about Java is that everyone has such a strong aversion to JNI stuff. Sometimes a library is in C++. The Python people—in the outside community and inside Google—don’t care. They’re like, “Oh, we’ll, SWIG-wrap it.” They get on their way and they’re happy. Python gets support for something right away if it’s in C++ because they’re not religious about what the source language is.

Java people are like, “Must be pure Java. We cannot use JNI because then if the JVM crashes, we don’t know why.” The problem with that is you end up writing everything twice, once for C++ and Python and all the other languages, and then once for Java. So if they could come up with a good embedding story or get over this fear of JNI, then I wouldn’t mind it.

Seibel: What about explicit memory management versus garbage collection? People still argue about that. Do you have a strong opinion one way or the other?

Fitzpatrick: No, not really. I’m amused to watch other people’s strong opinions when generally they’re not backed up by anything. I personally don’t find it that annoying to manage memory, at least in C++ with like scoped pointers. I can write in C++ for days and never actually say “new” or “delete”. It seems to just all kind of work.

I rewrote memcached inside Google to work with Google infrastructure and to add it to App Engine. That was all written in C++ because I needed a very exclusive control of memory to reduce fragmentation. So I really appreciated having it there.

Seibel: The original memcached was in C. Did you redo it in C++ because C++ is more accepted within Google, or were there other advantages?

Fitzpatrick: I started to take the existing one and port it but it turned out to be more work. Memcached isn’t that much code to begin with, so it was a lot quicker to just rewrite in C++. It was like half as much code, rewriting it in C++.

Seibel: Do you think that was because of C++ or just because you were smarter this time around?

Fitzpatrick: It could be. Once, when I was 11 or 12, we were on a trip around the US and I wrote that game Mastermind on a TI-85 calculator. I’m writing this program—a couple hundred lines—on this tiny little screen trying to remember where I am. I ended up deleting the damn thing twice. So I wrote the thing three times. But then it got so easy. That’s a good point—the second time around it was a lot easier.

Seibel: You’ve done a lot of your work in Perl, which is a pretty high-level language. How low do you think programmers need to go—do programmers still need to know assembly and how chips work?

Fitzpatrick: I don’t know. I see people that are really smart—I would say they’re good programmers—but say they only know Java. The way they think about solving things is always within the space they know. They don’t think end-to-end as much. I think it’s really important to know the whole stack even if you don’t operate within the whole stack.

When I was doing stuff on LiveJournal, I was thinking about things from JavaScript to how things were interacting in the kernel. I was reading Linux kernel code about epoll and I was like, “Well, what if we have all these long TCP connections and JavaScript is polling with these open TCP connections that are going to this load balancer?” I was trying to think of how much memory is in each structure here. That’s still somewhat high-level, but then we were thinking about things like, we’re getting so many interrupts on the Ethernet card—do we switch to this NAPI thing in the kernel where rather than the NIC sending an interrupt on every incoming packet it coalesces them to boundaries that were equivalent to 100 megabits speed even though it was a gigabit NIC. We were collecting numbers to see at what point this made sense and freed up the processor.

We were getting a lot of wins for really low-level stuff. I had somebody recently tell me about something: “Java takes care of that; we don’t have to deal with that.” I was like, “No, Java can’t take care of this because I know what kernel version you’re using and the kernel doesn’t support it. Your virtual machine may be hiding that from you and giving you some abstraction that makes it look like that’s efficient, but it’s only efficient when you’re running it on this kernel.” I get frustrated if people don’t understand at least the surface of the whole stack.

In practice, nothing works. There are all these beautiful abstractions that are backed by shit. The implementation of libraries that look like they could be beautiful are shit. And so if you’re the one responsible for the cost of buying servers, or reliability—if you’re on call for pages—it helps to actually know what’s going on under the covers and not trust everyone else’s libraries, and code, and interfaces.

I almost don’t think I would be a programmer today if I was starting off. It’s just too ugly. This is why I’m so excited about things like App Engine. Someone described Google’s App Engine as this generation’s BASIC. Because this generation, everything is networked. When I was programming, it was one language, and it was on my own machine, and the deploy was up enter, or RUN enter. Kids today don’t want to write something stupid like a “bounce a ball” app on their own machine. They want a web site to interact with.

I still have people mailing me who are like, “Hey, I have this idea—I want to make Wikipedia meets YouTube, meets—” Everyone wants to do a web site where their favorite four web sites aren’t quite right and they want to make one that looks kind of like that.

The fact that App Engine gives you one button, “Put this on the Web,” and you write in one language, arguably a pretty easy-to-learn one, Python, is perfect. It’s a great intro to programming—there are so many layers and layers of bullshit that it gets rid of.

Seibel: How does that fit with your dismay at the Java guys who tell you, “Oh, Java takes care of that for you.” Isn’t that the same? “Well, App Engine will take care of that for you.”

Fitzpatrick: I don’t know. Maybe it’s because I know what’s going on. Actually the JVM isn’t that bad. I guess it’s when people have blind faith in their abstractions without understanding what’s going on.

Seibel: You had a lot of programming experience by the time you got to college and studied computer science. How did that work out?

Fitzpatrick: I skipped a lot of my early C.S. classes, because they were just really boring. I would go and take the tests. Then towards the end they got kind of fun, once you get to the 300- and 400-level classes. But right when it got interesting, I graduated. And they wouldn’t let me take the fun grad-level classes, because I wasn’t a grad student.

I remember in the compiler class, the final project was we had to take this existing language that we had been playing with and add a whole bunch of features, including one feature of our own choosing as the bonus part of the project. So I chose to implement run-time array bounds checking. Anyway, the professor took our compiled binary and ran his test suite against it, and it failed a couple of his tests. He was like, “Sorry, you get a C because you failed my unit test,” When I went to look at it, I was like, “You have off-by ones in your test suite.” So he gave me the grade back and I got an A, but I never got the bonus points for adding a feature to the language. I was angry at school at that point.

And I remember our database class was taught by someone who, it seemed, had no real-world experience with databases. At this point I’d worked with Oracle, Microsoft Server, and tons of MySQL. So I was asking all these realworld questions I actually wanted answers to—things that were melting right now—they would just give me some textbook answer. I’m like, “No, no. That doesn’t work.”

Seibel: You graduated in 2002. Do you have any greater appreciation now of what they were trying to teach you?

Fitzpatrick: Half the classes I totally loved, and either I learned something totally new that I wouldn’t have learned at the time, or I learned the proper background material and the proper terminology. Prior to that, I knew programming pretty well but I didn’t have the vocabulary to describe what it was I was doing. Or I would make up my own terminology for it and people would think I didn’t know what I was talking about. Formal C.S. education helped me be able to talk about it.

Seibel: Do you have any regrets about combining running a business with school? Would you rather have just done one or the other?

Fitzpatrick: No, I think that was the best way. I had friends who went to college and just did college, but I knew so much of it already, I would’ve been bored. I had one friend who also knew a whole bunch of it but he was of this school of thought that he’s at college to learn, not for grades, so he was, on the side, studying Arabic and Chinese and Japanese. And all the crazy programming languages. Every week it was like, “I have a new favorite language. This week I’m only going to write in OCaml.” So he kept himself busy that way. I kept myself busy and not bored other ways.

Then I had friends who dropped out after their freshman year just to do web stuff. A couple were doing a porn web site or something. They were like, “Oh, we’re making all this money.” But they just worked a whole bunch; they were always in their basement working. College was awesome for meeting people and partying. If I just did LiveJournal, I would’ve killed myself stresswise.

Seibel: Are you glad you studied computer science?

Fitzpatrick: I probably could have done without it. I did a lot of things I wouldn’t have done normally, so I guess it was good. I wish maybe I would have like done something else as well, maybe stayed another year and double-majored in something totally unrelated. Did linguistics more. I’m kind of sad I left college and I felt I only did half studying because so much of it I already knew. My early C.S. classes I barely attended and it was only towards the end where things just started to get interesting when it was like, “OK, you’re done.”

Seibel: Did you ever think about going to grad school?

Fitzpatrick: Yeah. It would have been fun, but I was busy.

Seibel: Do you try to keep up with the C.S. literature?

Fitzpatrick: Me and my friends still forward each other papers around, neat papers. I read something the other day about some new technique for resizing Bloom filters at runtime. It was pretty awesome. The papers that come out of the storage conferences, some out of industry and some from academics, about different cool systems—I try to read those. There are different reading groups at Google—systems reading groups or storage reading groups. I’ll see something on Reddit or a friend will forward a paper or something like that or link it on a blog.

Seibel: You just mentioned papers from the academy and from industry. Do you have any sense of whether those two meet in the right place these days?

Fitzpatrick: They kind of feel about the same to me. But it’s more interesting, a lot of times, to read the industry ones because you know they did it to solve a problem and their solution works as opposed to, “We’d think it would be cool if—” There’s a lot of crazier stuff that comes out of academia and it doesn’t actually work, so it’s just a crazy idea. Maybe they turn it into commercial things later.

Seibel: How do you design software?

Fitzpatrick: I start with interfaces between things. What are the common methods, or the common RPCs, or the common queries. If it’s storage, I try to think, what are the common queries? What indexes do we need? How are the data going to be laid out on disk? Then I write dummy mocks for different parts and flesh it out over time.

Seibel: Do you write mocks in the test-first sense so you can test it as you go?

Fitzpatrick: More and more. I always designed software this way, even before testing. I would just design interfaces and storage first, and then work up to an actual implementation later.

Seibel: What form would the design take? Pseudocode? Actual code? Whiteboard scribbles?

Fitzpatrick: Generally I would bring up an editor and just write notes with pseudocode for the schema. After it got good, I would make up a real schema and then I would copy-paste it in just to make sure that “create table” works. Once I got that all going, I’d actually go implement it. I always start with a spec.txt first.

Seibel: After you write a bunch of code do you ever discover that you really need to reconsider your original plan?

Fitzpatrick: Sometimes. But I’ve started with the hard bits or the parts I was unsure of, and tried to implement those parts first. I try not to put off anything hard or surprising to the end; I enjoy doing the hard things first. The projects that I never finish—my friends give me shit that it’s a whole bunch—it’s because I did the hard part and I learned what I wanted to learn and I never got around to doing the boring stuff.

Seibel: Do you have any advice for self-taught programmers?

Fitzpatrick: Always try to do something a little harder, that’s outside your reach. Read code. I heard this a lot, but it didn’t really sink in until later. There were a number of years when I wrote a lot of code and never read anyone else’s. Then I get on the Internet and there’s all this open source code I could contribute to but I was just scared shitless that if it wasn’t my code and the whole design wasn’t in my head, that I couldn’t dive in and understand it.

Then I was sending in patches to Gaim, the GTK instant-messenger thing, and I was digging around that code and I just saw the whole design. Just seeing parts of it, I understood. I realized, after looking at other people’s code, that it wasn’t that I memorized all my own code; I was starting to see patterns. I would see their code and I was like, “Oh, OK. I understand the structure that they’re going with.”

Then I really enjoyed reading code, because whenever I didn’t understand some pattern, I was like, “Wait, why the fuck did they do it like this?” and I would look around more, and I’d be like, “Wow, that is a really clever way to do this. I see how that pays off.” I would’ve done that earlier but I was afraid to do it because I was thinking that if it wasn’t my code, I wouldn’t understand it.

Seibel: And how do you tackle reading other people’s code? For starters, do you read code just to see how it works overall, or do you always go in with some change you want to make?

Fitzpatrick: Generally I wanted to change something. Or if you really respect some programmer, go read some of their code. Maybe that’ll make you realize that they’re mortal and they’re not really someone you should be idolizing. Or you learn something about their code.

Seibel: So say you’ve got a change you want to make; how do you tackle it?

Fitzpatrick: First step, take a virgin tarball or check out from svn, and try to get the damn thing to build. Get over that hurdle. That tends to be the hugest hurdle for most people—dependencies in the build system or they’re assuming this library is installed. I almost wish that these large projects just came with a virtual machine that was the build environment.

Seibel: You mean like a VMware virtual machine?

Fitzpatrick: Yeah, so if you just want to get into hacking on it really quickly, here’s all the dependencies. People’s connections are getting quick enough. That’s totally viable.

Anyway, once you have one clean, working build, kill it, and just make one damn change. Change the title bar to say, “Brad says, ‘Hello world.’” Change something. Even if everything’s ugly, just start making changes.

Then send out patches along the way. I find that’s the best way to start a conversation. If you get on a mailing list and you’re like, “Hey, I want to add a feature X,” the maintainer is probably going to be like, “Oh fuck, I’m so busy. Go away. I hate feature X.” But if you come to them and you’re like, “I want to add feature X. I was thinking something like the attached patch,” which is totally wrong but you say, “But I think it’s totally wrong. I’m thinking the right way might be to do X,” which is some more complex way, generally they’ll be like, “Holy crap, they tried, and look, they totally did it the wrong way.”

Maybe that pains the maintainer. They’re like, “Oh man, I can’t believe they went through all that effort to do it. It’s so easy to do it the right way.” Or, “Oh, wow, they did all this work in the wrong direction. I hope they don’t go in that direction any more.” And then they reply.

That’s always the best way to start a conversation. Even at Google, that’s the way I start a lot of conversations to a team I don’t know. When I fix a bug in their product the first thing I do is send them a patch in the mail and just say, “What do you guys think of this?” Or on the internal code-review tool I’d be like, “Here is a review. What do you think?” They could just say, “Fuck no, that’s totally the wrong fix.”

Seibel: Do you still read code for fun, as opposed to reading it because you need to work with it?

Fitzpatrick: Sometimes. I checked out Android source code for no real reason. The same with Chrome; when it went open source, I mirrored the repo and just looked around. I did the same thing with Firefox and Open Office. Some program you’ve been using and all of a sudden you have access and you might as well look.

Seibel: Programs like that, the code base is pretty huge. When you look at something like that for fun, how deeply do you get into it?

Fitzpatrick: Generally, I’ll just pipe find into less and try to understand the directory structure. Then either something grabs my eye or I don’t understand what something is. So I pick a random file and get a feel for it. Then I bounce around and wander aimlessly until I’m bored and then pick a new random spot to jump in.

A lot of times, I’ll work on building it in parallel with reading it because they’re very parallelizable tasks, especially if it’s hard to build. By the time it’s finally built, then I can start tweaking it if I want to.

Seibel: So when you read good code it either fits into patterns that you already understand, or you’d discover new patterns. But not all code is good. What are the first warning signs of bad code?

Fitzpatrick: Well, I’m particularly snooty now, having worked at Google with really strict style guidelines in all languages. On our top six or seven languages, there’s a really strict style guide that says, “This is how we lay out our code. This is how we name variables. This is how we do spacing and indentation, and these patterns and conventions you use, and this is how you declare a static field.”

We’ve started putting these online too, just as a reference for external contributors contributing to our projects. We wanted to have a documented policy so we don’t just say, “We don’t like your style.”

Now when I work on projects in C, the first thing I do is add a style guide. Once a project is mature and has a lot of people hacking on it, they’ll have a style guide. It’s not even always written, but the programmers just respect the style of code written already. Maybe they don’t like the brace style, but fuck it, it’s more important to have it consistent within a file, within a project, than to do it your favorite way.

Seibel: Do you ever do any pair programming?

Fitzpatrick: I think it’s pretty fun. It’s good for lots of things. Sometimes you just need to think and want to be left alone. I don’t subscribe to it all the time, but it’s definitely fun.

I start too many projects. I finish them because I have guilt if I don’t finish them, but I definitely context-switch way too often and I’m spread too thin. This is why I really need pair programming—it forces me to sit down for three solid hours, or even two or one solid hour, and work on one thing with somebody else, and they force me to not be bored. If I hit a bored patch, they’re like, “Come on. We’ve got to do it,” and we finish.

I like working alone but I just bounce all over the place when I do. On a plane I’ll bring extra laptop batteries and I have a whole development environment with local web servers and I’ll be in a web browser, testing stuff. But I’ll still be hitting new tabs, and typing “reddit” or “lwn”—sites I read. Autocomplete and hit Enter, and then—error message. I’ll do this multiple times within a minute. Holy fuck! Do I do this at work? Am I reading web sites this often that I don’t even think about it? It’s scary. I had a friend, who had some iptables rule, that on connection to certain IP addresses between certain hours of the day would redirect to a “You should be working,” page. I haven’t got around to doing that, but I need to do something like it, probably.

Seibel: What about code ownership? Is it important for people to own code individually or is it better for a team to share ownership?

Fitzpatrick: I don’t think code should be owned. I don’t think anyone really thinks that. The way it works within Google is that it’s one massive source tree, one root, and one unified build system across all of it. And so anyone can go and change anything. But there are code reviews, and directories have owners, always at least two people, just in case someone quits or is on vacation.

To check in you need three conditions met: You need someone to review it and say it looks good. You need to be certified in the language—basically you’ve proven you know the style of this language—called “readability.” And then you also need the approval above from somebody in the owner’s file in that directory. So in the case that you already are an owner of that directory and you have readability in that language, you just need someone to say, “Yeah, it looks good.” And it’s a pretty good system, because there tends to be a minimum of two, up to twenty, thirty owners. Once you work on a code base for a while, someone just adds you to owners. I think it’s a great system.

Seibel: So let’s go back in time a bit—how did LiveJournal start?

Fitzpatrick: It was just fucking around with my friends—what I wanted and what we thought would be funny. Commenting on LiveJournal was a practical joke. I was checking my LiveJournal right before I ran into class. We had just introduced friend pages and I saw something my friend wrote and it was really stupid and I wanted to make fun of him. “Oh, but I can’t reply.” So I went to class and all throughout class I was thinking, “How can I add a reply system?” I was thinking of the existing schema and how we could render it. I had a two-hour break between classes, so I add commenting and I reply something smartass and sarcastic and go to my other class. When I came back from my second class, he’s like, “What the fuck? We can comment now?”

Everything on LiveJournal was pretty much a joke. The whole security thing, like friends-only posts and private posts, was because a friend wrote that he went to a party and woke up drunk in a ditch the next day. His parents read it and were like, “What? You’re drinking?” He was like, “Brad, we need a way to lock this shit down!” I was like, “On it!” We already had friends, so we just made it so some posts are friends only and then your parents—just don’t be friends with them.

Seibel: In the early days of LiveJournal it seems your life was an endless series of late nights, sleeping late, and overall working long hours. How much of that is a necessary part of programming?

Fitzpatrick: I just thought it was the least stressful time. During the day, there’s always something coming up, like another meal is coming up, or a class, or maybe you get a phone call. There’s always some interruption. I can’t relax. If I go into work two hours before some meeting, that two hours is less productive than if I didn’t have that meeting that day or if the meeting was the first thing in the morning. Knowing that I have nothing coming up to bug me, I’m so much more relaxed.

At night I feel like this is my time and I’m stealing this time because everyone else is sleeping. There’s no noise and no interruptions, and I can do whatever. I still stay up late sometimes. I did it this weekend; I was up quite a bit working on different things. But that screws me up for days sleepwise. I did that mostly when I had to in college, because I had some project, and I was also doing LiveJournal on the side. The only time to do it was at night and also all our server maintenance had to be at night. And then in the summer, just because why not? There’s no reason to wake up early in the morning to go to a class or anything, so might as well work at night.

Seibel: What about the length and intensity? I’m sure you’ve done the 80-, 100-, 120-hour weeks. Is that necessary? Under what circumstances is that really necessary and when is it just a macho thing that we do?

Fitzpatrick: In my case, I’m not sure it was either necessary or a macho thing. I was having fun and it was what I wanted to be doing. Sometimes things were breaking, but even when they weren’t breaking, I was still doing it just because I was working on a new feature that I really wanted to see happen.

Seibel: Have you ever been in a situation where you really had to estimate how long things were going to take?

Fitzpatrick: Once I got to Six Apart. I guess that was my first experience, three and a half years ago. We had started doing migration—we’d have a customer and they’d say, “Can you move this data?” That requires adding this support for this code and testing, and pushing it out. I was terrible at it. I probably still am terrible at it, because I always forget a factor, like the bullshit multiplier of having to deal with interruptions and the fact that I’m never going to get away from maintaining a dozen projects on the side.

I think I’m getting better, but fortunately they don’t ask for that too often. And now when I actually do get a deadline for something, I’m like, “Yay! A deadline!” and I get so excited that the adrenaline kicks in, and I work, and I finish the damn thing. Nothing with Google is really a deadline. With Google it’s like, “What do you think about launching this? How does that feel?” It’s rare that there’s some real deadline. Most of them, we think it’d be nice to launch on this date and so everyone tries really hard. But you’re only letting down other people that want to see it launch by that day if you don’t finish something. And most of the things I work on are very “When it’s done, it’s done.”

Seibel: When you were hiring programmers at LiveJournal, did you manage them?

Fitzpatrick: Well, I kind of assumed that none of them would need managing; that they would just all be self-driven like me. That was a learning experience in HR, that some people just do what they’re told and don’t really have a passion for excellence. They’re just like, “Done. Next assignment,” Or they don’t tell you and just browse the Web. So I had a couple of painful experiences. But I think after a year or two of that, I learned that people are different.

Some are purists. They would just do abstraction on abstraction on abstractions. They would go really slowly and are very religious about their style. They’re like, “I’m an artisan programmer.” And I was like, “Your code doesn’t run. It’s not efficient and it doesn’t look like any of the other code that you’re interacting with.”

Seibel: Did you figure out how to make good use of people like that?

Fitzpatrick: One person, I tried dozens of different things. I think he might’ve been ten years older than me. I don’t know how much, because I never ask that—I was afraid of legal hiring questions. But I got the feeling that he didn’t want to work for some young punk. I was like 22. That one eventually didn’t work out. That was the only person I let go.

Other people I eventually figured out what motivated them. One guy was really good at tinkering and getting a prototype working. He wrote sysadmin Perl. He could wire stuff together, write shell scripts, and write really bad Perl and really bad C, but kind of get it working. Then we would be like, “Holy crap, you researched all this stuff, you got all these components talking to each other?”

We were setting up a voice bridge to LiveJournal so you record something and post it to LiveJournal. There were just so many moving parts involved. I thought it was painful as hell. He loved it. He figured it all out and got it working. Then we just rewrote it all. And we figured out that was the way it worked with him. He figured out the interface and we would fix it all up. Once I figured out that was his role, we got along great.

Seibel: So you’ve hired for your own company, and I assume you’ve been involved in hiring at Google. How do you recognize a great programmer?

Fitzpatrick: I often look for people that have done, like, a lot of stuff on their own that wasn’t asked of them. Not just their school project or just what their previous employer had them do. Somebody who was passionate about something and had some side project. How did they maintain it and how serious did they get with it? Or do they do a lot of quick hacks and abandon them?

Seibel: Do you have favorite interview questions?

Fitzpatrick: One of the ones I’ve given a few times because it was on my AP programming test is given two decimal numbers as strings of arbitrary length, multiply them. There are a lot of different ways that they could do it. If they’re really good at math—like I’m not—they can find some clever ways to do it really efficiently. Worst case, they can make a class that does just addition repeatedly.

I tell them from the beginning, “Don’t stress out. You don’t have to do it efficiently. Just get it done somehow.” Some people stress out and have no clue where to begin. That’s kind of a bad sign. The worst case, you just implement the algorithm you do in grade school.

I actually wrote a program in grade school to do my long division and multiplication and show the work. Including all the steps and where to cross out. So then we would get these problems, like ten per page or something, and I would type it into the computer and then just reproduce the problems in scribbles. I did the same thing in chemistry to find the orbitals of electrons. But the thing I find is by writing a program to cheat, you learn because you have to learn it really in depth to write that program.

Seibel: Do you think that would work for anyone? Instead of teaching kids long division, should we teach them how to program and then say, “OK, now your task is to write a program to do this long-division procedure”? By the time they’ve actually written that program, they’ll understand division. Or does that only work if you have some natural inclination that way?

Fitzpatrick: It worked for me. A lot of times, someone could teach you something and you’re like, “Yeah, yeah, sure. I understand.” You delude yourself but once you actually have to get down and do it at a real level and understand all the corner cases, it forces you to actually learn the thing. But I don’t know if that would work for everyone.

Seibel: Google has a bit of a reputation, as Microsoft also does, of interviewers using puzzle questions.

Fitzpatrick: I think those are kind of banned. Or strongly discouraged. Maybe some people still do them, but I think, in general, they’re discouraged.

Seibel: What did they ask you in your interview?

Fitzpatrick: One question was, imagine you have a bunch of computers on a switch and they turn on the whole rack; come up with an algorithm so every machine on the rack knows the status of all the other ones about whether they’re on or off. So basically a presence thing. That was pretty much the constraint. Basically, they described Ethernet: you could send a broadcast to everyone, or you could send it to a specific MAC address. So I just kind of walked through all the different strategies to minimize bandwidth and to minimize latency discovering when something’s dead. That was a fun one.

Seibel: What’s the worst bug you ever had to track down?

Fitzpatrick: I try not to remember them. I hate it when it’s something where your assumptions are so far off. The other day—this is definitely not an example of the worst one ever—I spent 90 minutes debugging something because I was writing to one output file and reading another file named the same thing but with one path component missing. I kept rerunning this huge MapReduce and seeing the output and putting it in GDB and stepping through it. “What the fuck? It’s not changing!” Finally I looked at the paths and I was like, “Holy crap.” I don’t know why I spent 90 minutes on it; I was so obsessed that I didn’t step back and check, is my command line correct?

There’s a lot of that. We always had some good stuff with Perl like the $_ isn’t lexically scoped. So if you fuck with $_ in a sort, you can mess with somebody else’s far away. So we had this bug that took us forever and we had a bunch of corruption going on. We finally figured that out. Then I audited all our code we had a new policy of “never do this.”

Seibel: What are your debugging tools? Debuggers? Printlns? Something else?

Fitzpatrick: Println if I’m in an environment where I can do that. Debugger, if I’m in an environment that has good debuggers. GDB is really well maintained at Google and is kind of irreplaceable when you need it. I try not to need it too often. I’m not that great at it, but I can look around and kind of figure things out generally. If I have to go in there, I generally can find my way out. I love strace. Strace, I don’t think I could live without. If I don’t know what some program is doing, or what my program is doing, I run it under strace and see exactly what’s happening. If I could only have one tool, it would probably be that. All the Valgrind tools, Callgrind and all that, those are good.

But a lot of times lately, if there’s something weird going on, I’m like, “OK, that function is too big; let’s break that up into smaller parts and unit-test each one of them separately to figure out where my assumptions are wrong, rather than just sticking in random printlns.”

Then maybe in the process of refactoring, I have to think about the code more, and then it becomes obvious. I could, at that point, go back to the big, ugly state where it was one big function and fix it but I’m already halfway there; I might as well continue making it simpler for the next maintainer.

Seibel: How do you use invariants in your code? Some people throw in ad hoc asserts and some people put in invariants at every step so they can prove formal properties of their programs, and there’s a big range in the middle.

Fitzpatrick: I don’t go all the way to formal. My basic rule is, if it could possibly come from the end user, it’s not a run-time crash. But if it is my code to my code, I crash it as hard as possible—fail as early as possible.

I try to think mostly in terms of preconditions, and checking things in the constructor and the beginning of a function. Debug checks, if possible, so it compiles away. There are probably a lot of schools of thought and I’m probably not educated about what the proper way to do it is. There are languages where all this stuff is actually a formal part of the language. Pretty much all the languages I write in, it’s up to you.

Seibel: You wrote once that optimization is your favorite part of programming. Is that still true?

Fitzpatrick: Optimization is fun because it’s not necessary. If you’re doing that, you’ve got your thing working and nothing else is more important and you’re either saving money or doing it because it’s like a Perl golf contest—how short can I make this or how much faster. We would identify hotspots in LiveJournal, and I would send out some contests. “Here’s some code. Here’s the benchmark. Make it fast.” I sent our load balancer’s header parsing. We were all writing crazy regexps that didn’t backtrack and tried to capture things with the most efficient capture groups. And we were all competing, getting faster and faster and faster. Then one guy comes over the next day. He had written it all in C++ with XS, and so he was like, “I win.”

Seibel: The flip side of that these days is…

Fitzpatrick: Programmers’ time is worth more and all that crap? Which can be true. This is true for a small number of machines. Once you get to a lot of machines, all of a sudden the programmers’ time is worth less than the number of machines that this will be deployed against, so now write it in C and profile the hell out of it, and fix the compiler, and pay people to work on GCC to make this compile faster.

Seibel: But even Google uses C++ rather than assembly, so there’s some point at which trying to squeeze the maximum performance isn’t worth it. Or is the theory that a good C++ compiler generates better code than all but the most freakishly rare assembly coders?

Fitzpatrick: We still have some stuff in assembly, but it’s very rare. We have profiling for lots and lots of stuff and it has to really be justified to rewrite it even from Perl to C, and then from C to assembly. Even if it’s all x86, there are all different variations of x86. Do you really want to write assembly for every variation of x86? This one uses SSE 2 and that one uses SSE 3.1. Let the compiler deal with it.

Seibel: You learned to program from programming manuals when you were a little kid. Are there any books that you strongly recommend to new programmers now or think that everyone should read?

Fitzpatrick: Back when I was doing Perl—even for people that knew Perl really well—I would recommend MJD’s Higher-Order Perl. The book is really fun in that it starts somewhat simple and you’re like, “Yeah, yeah, I know what a closure is.” And then it just continues to fuck with your head. By the end of the book, you’re just blown away. Even though I knew all that stuff, in theory, just watching it taken to the extreme really changed how I thought. I recommended that to a bunch of my friends and it blew their minds. In general, any book that gives people a different style to think in. That’s the most recent example I can think of.

Seibel: I see you’ve got The Art of Computer Programming up there; it doesn’t look too worn. How much of it have you read?

Fitzpatrick: Oh, I didn’t get it until less than five years ago, maybe five years ago. I bounce around and read parts of it for fun. But at the time I got it, I had already learned a lot of it through C.S.. So it probably would have been more valuable early on, but I didn’t really know about it prior to the Internet.

Seibel: How much math do you think is necessary to be a programmer? To read Knuth and really understand it, you’ve got to be pretty mathematically sophisticated, but do you actually need that to be a programmer?

Fitzpatrick: You don’t need that much math. For most programmers, day to day, statistics is a lot more important. If you’re doing graphics stuff, math is a lot more important but most people doing Java enterprise stuff or web stuff, it’s not. Logic helps and statistics comes up a lot.

Seibel: You obviously still enjoy programming. But reading some of your LiveJournal entries from when you were in college, it seems like there were times when you were pretty stressed out and hating computers.

Fitzpatrick: Oh, well, I always hate computers. I don’t think we’ve really made any progress in quite a long time. Computers seem slower, and crashier, and buggier than ever. But I’m such an optimist, I keep thinking that they’ll get better. It seems like my computing experience was happier ten years ago than it is today. It seems like my computer was faster ten years ago; like my computer worked better ten years ago. Things have gotten faster but the software has gotten slower and buggier in the meantime.

Seibel: Why do you think that is?

Fitzpatrick: I don’t know. Has the bar been lowered? Or are computers faster so you don’t need to be efficient or you don’t need to know what you’re doing? I have no clue. Some combination of all of the above, or maybe there are so many layers of abstraction that people don’t know what the hell is going on underneath because the computers are so damn fast that it hides your stupidity.

Seibel: So maybe things are not as fast as they ought to be given the speed of computers. But ten years ago there was no way to do what people, as users, can do today with Google.

Fitzpatrick: Yeah. So some people are writing efficient code and making use of it. I don’t play any games, but occasionally I’ll see someone playing something and I’m like, “Holy shit, that’s possible?” It just blows me away. Obviously, some people are doing it right.

I guess I’m mostly dissatisfied with the state of my desktop. It seems like there is a lot of good interesting stuff going on in the back end. But as I’m using my computer I’m more and more frustrated with it. It seems like my Mac shouldn’t have a beach ball spinning all the time.

Seibel: Do you have any interest in writing better desktop software?

Fitzpatrick: The problem is, no one uses it. You want to write stuff people use, which comes down to web apps. I lost my laptop the other day and people were like, “Oh, my God, did you lose stuff?” I had no files on there. It was an Internet terminal. And it was an encrypted disk so I’m not worried about my password or cookies or anything like that. People won’t download programs, I don’t think.

Seibel: Are you more motivated by having users or just by the fun of programming?

Fitzpatrick: There’s definitely some stuff I write for me and I write it explicitly for me as the only user and I could care less if I get patches and stuff. But a lot of times I want to work with other people. Having users is a key to getting contributors. More users find more bugs and find more use cases. It’s more fun to work with other people, especially on open source stuff.

It always feels good when someone writes in to tell you, “Hey, we’re now using your software for x.” That’s pretty cool. When I see the number of web sites that use memcached or the load balancer or something, I’m like, “Ah, that’s cool.” I remember all these porn sites started telling me they’re using my file system. Well, that says something. I’m helping serve up porn. On Craigslist, every request goes through a web server that is basically a front end to memcached. OK. That’s cool.

Seibel: Do you think programmers are overenamored of new things? New languages, new tools, new whatever?

Fitzpatrick: They might be. I don’t know if that’s desperation in hoping the new thing doesn’t suck, like the new programming language does what we all want. But users are the same way. Users always like to get the one with the higher version number even if it sucks more.

I don’t know if programmers are statistically different than humans in general. New must be better. Which is not always the case, but people hope it is. They want it to be.

I remember talking to my dentist a while back and she was going on and on about, like, the advances in dentistry over the last, like, five years, and she was really excited about them.

Seibel: A lot of being a modern programmer requires finding the right pieces that you need to use and understanding them just well enough to use them. How do you deal with that?

Fitzpatrick: CPAN has everything. There are 14 ID3 parsers. Pick one.

Seibel: So that, in a way, is the problem facing the modern programmer—there are 14 of them. How do you pick?

Fitzpatrick: Google search—which one’s highest? Which one do people tend to like? And knowing people. I got so much more involved in the open source community once I started going to all these conferences because then I would meet people and see who was respected, and who was cool.

Then I would see their code: I remember that guy. He was awesome. He was fun, friendly, and attentive, and he really cared about his code. He was really passionate when people complained about it. I’m going to use that one because if I find any bugs, I know he’ll be crazy about fixing them. As opposed to the grumpy guy who maybe writes great code, but he’s grumpy and not fun to interact with if you have a question or a bug. So you pick maintainers you trust or respect.

Seibel: Then is there any trick to quickly figuring out whether something’s going to suit your needs?

Fitzpatrick: I just start. I don’t plug it right into my code—first I write a test program that uses the couple functions I know I’m going to need, make sure they work. Or write a unit test for just that library on just the data I plan to use with it. A lot of libraries out there don’t even have their own tests. Even if it does, maybe you read the doc and you don’t really trust that it does what it says it does, or the doc wasn’t clear about how it behaves. So I write my own tests for the shit I care about. I figure since I’m going to have to write something to learn the library anyway, my first Hello, World program might as well be a unit test against it.

Seibel: What about the actual tools you use—you’re still an Emacs user, right?

Fitzpatrick: I’m still an Emacs user. I wish I were better at Emacs. But I know all the keystrokes but I don’t really customize it much. I steal other people’s customizations and I can kind of read it. But I find myself getting annoyed by something and saying, “I should go write some Elisp to bind it to a key.” And then I don’t.

Steve Yegge is working on project to basically replace all of Elisp with JavaScript. So I keep saying, I’ll wait for him to do that so I don’t have to learn another language. I’ll just write it in JavaScript. I don’t mind JavaScript as a language. It’s browsers that suck. At Google I write a lot of stuff in JavaScript that I then embed in Java and C++. I figure JavaScript is a good embedding language.

Seibel: Are there any tools that you are forced to use regularly that you just hate? Other than your whole desktop?

Fitzpatrick: Yeah, the whole desktop. There are a whole lot of things on my desktop. All these browsers are always hanging and crashing and using tons of memory. My whole operating system hanging. My coworkers try to tell me—if they see me doing something in Emacs—that Eclipse or IntelliJ does it for them automatically. So every six months I try out one of them, Eclipse or IntelliJ. And the damn thing just sits there spinning forever, consuming memory and maybe crashes in the middle of me typing or can’t keep up with me typing. Come on—syntax-highlight in the background or compile in a different thread. Why are you blocking my typing to do this? OK, I’ll try it again in six months, guys. So I’m glad I’m not forced to use that. I should really get better at Emacs, though.

My learning curve is, I learn something pretty rapidly until I get to this point where I’m pretty productive and good enough. Then I kind of plateau at like maybe 90 percent or 80 percent, where I’m productive and I don’t have to look things up and I’m happy. And then it slowly gets better after that. It’s only after I’m super comfortable with something that I’m like, “I’m going to go dig around the docs for this language—the man pages—and learn every nook and cranny.”

Seibel: Is that maybe wise these days? There are so many things you could learn. You could spend forever just learning how to use your editor, and how much software would you write then?

Fitzpatrick: Yeah, but I’ve always found—at least for your editor—it always pays off. Whenever I learn something, it pays off within, I don’t know, a week or two. Whenever I write a stupid little shell script in my bin directory, or a little Perl script, or something to automate my life, it always pays off.

Seibel: So you’ve never gotten trapped in the pit of endless toolsmithing?

Fitzpatrick: Nah. I tend to do it for a purpose. I definitely know people who are just always working on their personal tools and never get anything done. I can go a little bit more in that direction and be safe, though.

Seibel: What do you think is the most important skill for a programmer to have?

Fitzpatrick: Thinking like a scientist; changing one thing at a time. Patience and trying to understand the root cause of things. Especially when you’re debugging something or designing something that’s not quite working. I’ve seen young programmers say, “Oh, shit, it doesn’t work,” and then rewrite it all. Stop. Try to figure out what’s going on. Learn how to write things incrementally so that at each stage you could verify it.

Seibel: Is there anything that you did specifically to improve your skill as a programmer?

Fitzpatrick: Sometimes I’ll go out of my way to write something in a language that I would rather not write it in—and I know it’ll take me longer to write in that language—because I know I’ll be better in the end. Like when I got to Google, there were a lot of times where I was writing one-off things and I’d go to write it in Perl. Then I’d be like, “Ah, no, I should write this in Python.” Now I write tons of Python and it doesn’t bother me—I barely have to look things up. Perlbal was originally written in C# just to learn that.

Seibel: And are there skills apart from programming itself that you think would-be programmers should develop?

Fitzpatrick: There’s communication, but I’m not sure that’s something you can really practice. Deal with people on mailing lists a lot. Written communication style goes a long way. But that’s a general life thing, right? There was some study about who was successful after high school. Was it the smart kids or the social kids? It turned out that it was the social kids who ended up making all the money in life, not the people with the good grades. I thought that was interesting.

Seibel: That seems to be a bit of a change from the past. It used to be programmers could be gnomes hiding in an office. These days it’s all mailing lists and collaboration.

Fitzpatrick: Well, at the places I’ve worked, either on open source or at companies, everyone depends on each other. The motivating factor is, “I’m going to write this code because I know you’re going to be needing it in two weeks or I’m going to be needing yours in two weeks.” There’s always a human level.

Seibel: People have claimed that there are orders of magnitude differences in productivity between the best and worst programmers. Has that been your experience?

Fitzpatrick: Yeah, but it’s probably like that in every field. It’s how much experience you have. It’s not often the case where I know of two people who do the same sort of programming and have been doing it the same amount of time, but differ by a factor of ten times. It seems like if you’re not getting better all the time, you’re probably getting frustrated and you drop out.

I guess there are the people who just do it for a job but don’t really enjoy it. Which is OK. But it’s kind of weird to compare those people with people who are hardcore programmers. What’s ten times more productive when one person works ten times the hours and thinks about it nonstop and the other person just does it at his job?

Seibel: You just mentioned taking a scientist’s approach to debugging. Do you consider yourself a scientist, an engineer, an artist, or a craftsman?

Fitzpatrick: Either scientist or engineer. Probably more engineer. I would say scientist was second, but only in the sense of the scientific method of changing one thing at a time and how you diagnose problems. Engineer for the design aspect of things. I definitely have friends who call themselves artists or craftsmen. I’ve never thought of myself as that.

Seibel: On the other hand, there’s a lot of engineering envy in software. You hear the jokes about, “If people built skyscrapers the way we build software, the first woodpecker would destroy civilization.” Do you think building software is a well-understood engineering discipline?

Fitzpatrick: No. I don’t think it’s there yet. You don’t need a license, right, to write code. Not that I want tons of regulation, but it would be nice to know that some of these PHP programmers with these XSS exploits aren’t the same people writing the air-traffic-control system. I’d like there to be some official line between those people.

I have a friend who’s a structural engineer and he went to school forever and took all this engineering certification stuff. It’s kind of comforting to know that the people who build the bridges I’m on studied this shit forever and took tons of tests and stayed up all the time studying.

Seibel: But what test could you give a programmer that would give you confidence that they can write software that will work?

Fitzpatrick: I don’t know. It’s kind of scary.

Seibel: Even without licensing, do you think programmers have any sort of special ethical responsibilities to society? We’re arguably a profession, and professions have codes of conduct.

Fitzpatrick: You shouldn’t kill anyone. Like with flight-control software. But that’s kind of a rare case. I would like to ask that everyone is consistent on their credit-card forms to like let me put in fucking spaces or hyphens. Computers are good at removing that shit. Don’t tell me how to format my numbers. But there’s no ethics there. Just stupidity.

Seibel: You’re 28 now. Do you have any worries that programming is sort of a young person’s game, that you’re going to lose a step as you get older?

Fitzpatrick: No. The worst case, I could always just stop and work on fun things on my own. I don’t feel like I’m competing with anyone right now and I don’t really care if other people are better because I feel like there are tons of people who are better already. I figure we are always in the middle anyway, so I’m happy to stay in the middle.

Seibel: So programming is what you’d do for fun, even if you quit working?

Fitzpatrick: Oh, yeah. I’d still do stupid shit. I have this silly board game on my phone. I was kind of tired—I couldn’t work on anything serious, so I wrote a solver for that game. Tried to do some dynamic programming and did different board sizes and did a bunch of random boards and made a histogram of how many moves it takes to solve the board for different board sizes. I sent it to the author because the game has a really bad estimate of par. Basically, to advance in the game you have to do better than par. And everyone on the mailing list was noticing that the game got easier as you went on because his estimate of par he kind of pulled out of nowhere. So I sent him the histograms at every board size. I think in the new version of the game he adjusted par. That was a fun hack, just on the shuttle home. I could retire and just do dumb shit like that all day.

Оглавление книги