: Coders at Work: Reflections on the craft of programming

Joe Armstrong

Joe Armstrong

Joe Armstrong is best known as the creator of the programming language Erlang and the Open Telecom Platform (OTP), a framework for building Erlang applications.

In the modern language landscape, Erlang is a bit of an odd duck. It is both older and younger than many popular languages: Armstrong started work on it in 1986a year before Perl appearedbut it was available only as a commercial product and used primarily within Ericsson until it was released as open source in 1998, three years after Java and Ruby appeared. Its roots are in the logic programming language Prolog rather than some member of the Algol family. And it was designed for a fairly specific kind of software: highly available, highly reliable systems like telephone switches.

But the characteristics that made it good for building telephone switches alsoand almost inadvertentlymade it quite well suited to writing concurrent software, something which has drawn notice as programmers have started wrestling with the consequences of the multicore future.

Armstrong, too, is a bit of an odd duck. Originally a physicist, he switched to computer science when he ran out of money in the middle of his physics PhD and landed a job as a researcher working for Donald Michieone of the founders of the field of artificial intelligence in Britain. At Michies lab, Armstrong was exposed to the full range of AI goodies, becoming a founding member of the British Robotics Association and writing papers about robotic vision.

When funding for AI dried up as a result of the famous Lighthill report, it was back to physics-related programming for more than half a decade, first at the EISCAT scientific association and later the Swedish Space Corporation, before finally joining the Ericsson Computer Science Lab, where he invented Erlang.

In our several days of conversation over his kitchen table in Stockholm, we talked about, among other things, the Erlang approach to concurrency, the need for better and simpler ways of connecting programs, and the importance of opening up black boxes.

Seibel: How did you learn to program? When did it all start?

Armstrong: When I was at school. I was born in 1950 so there werent many computers around then. The final year of school, I suppose I must have been 17, the local council had a mainframe computerprobably an IBM. We could write Fortran on it. It was the usual thingyou wrote your programs on coding sheets and you sent them off. A week later the coding sheets and the punch cards came back and you had to approve them. But the people who made the punch cards would make mistakes. So it might go backwards and forwards one or two times. And then it would finally go to the computer center.

Then it went to the computer center and came back and the Fortran compiler had stopped at the first syntactic error in the program. It didnt even process the remainder of the program. It was something like three months to run your first program. I learned then, instead of sending one program you had to develop every single subroutine in parallel and send the lot. I think I wrote a little program to display a chess boardit would plot a chess board on the printer. But I had to write all the subroutines as parallel tasks because the turnaround time was so appallingly bad.

Seibel: So you would write a subroutine with, basically, a unit test so you would see that it had, in fact, run?

Armstrong: Yes. And then youd put it all together. I dont know if that counts as learning programming. When I went to university I was in the physics department at University College of London. I think we probably had programming from the first year. Then you had this turnaround of three hours or something. But again it was best to run about four or five programs at the same time so you got them back fairly quickly.

Seibel: In high school, was it an actual school course?

Armstrong: It was an after-hours coursecomputer club or something. We went to see the computer, I remember. Lots of serious-looking older men wearing white coats with pens stuck in their pockets wandering around, like, a church. It was a very expensive computer.

Seibel: You were studying physics; when did you shift to programming?

Armstrong: Well, as an undergraduate some of the courses involved writing programs and I really enjoyed that. And I got to be very good at debugging. If all else failed, I would debug peoples programs. The standard debugging was one beer. Then it would go upa two-beer problem or a three-beer problem or something like that.

Seibel: That was in terms of how many beers they had to buy you when you debugged their program?

Armstrong: Yeah, when I fixed their program. I used to read programs and think, Why are they writing it this way; this is very complicated, and Id just rewrite them to simplify them. It used to strike me as strange that people wrote complicated programs. I could see how to do things in a few lines and theyd written tens of lines and Id sort of wonder why they didnt see the simple way. I got quite good at that.

When I really got to programming was after I finished my first degree and I decided I wanted to do a PhD. So I started to do a PhD in high-energy physics and joined the bubble chamber group there and they had a computer. A DDP-516, a Honeywell DDP-516. And I could use it all by myself. It was punched cards, but I could run the programs thereI could put them into the thing and press a button and whoomp, out came the answer immediately. I had great fun with that. I wrote a little chess program for it.

This was when real core memory was knitted by little old ladies and you could see the coresyou could see these little magnets and the wires went in and out. Frightfully expensiveit had something like a 10MB disk drive that had 20 platters and weighed 15 kilos or something. It had a teletext interfaceyou could type your programs in on that.

And then came this glass TTY which was one of the first visual display units and you could type your programs in and edit them. I thought this was fantastic. No more punched cards. I remember talking to the computer manager and saying, You know, one day everybody will have these. And he said, Youre mad, Joe. Completely mad! Why not? Well, theyre far too expensive.

That was really when I learned to program. And my supervisor at the time, he said, You shouldnt be doing a PhD in physics. You should stop and do computers because you love computers. And I said, No, no, no. Ive to finish this stuff that I was doing. But he was right, actually.

Seibel: Did you finish your PhD?

Armstrong: No, I didnt because I ran out of money. Then I went to Edinburgh. When I was reading physics we used to go and study in the physics library. And in the corner of the physics library there was this section of computer science books. And there were these brown-backed volumes called Machine Intelligence, Volumes 1, 2, 3, and 4, which came from Edinburgh, from the Department of Machine Intelligence there. I was supposed to be studying physics but I was eagerly reading these things and thought, Oh, thats jolly good fun. So I wrote to Donald Michie, who was the director of the Department of Machine Intelligence at Edinburgh, and said I was very interested in this kind of stuff and did he have any jobs. And I got back a letter that said, well, they didnt at the moment but he would like to meet me anyway, see what sort of person I was.

Months later I got a phone call, or letter, from Michie, saying, Ill be in London next Tuesday; can we meet? Im getting the train to Edinburgh; can you come to the station? I went to the station, met Michie, and he said, Hmmm! Well, we cant have an interview herewell, well find a pub. So we went to a pub and I chatted to Michie and then a bit later I got another letter from him, he says, Theres a research job at Edinburgh, why dont you apply for it. So I became Donald Michies research assistant and went to Edinburgh. That was my transition between physics and computer science.

Michie had worked with Turing at Bletchley Park during the second World War and got all of Turings papers. I had a desk in Turings library, so all around me were Turings papers. So I was a year at Edinburgh. After that Edinburgh kind of collapsed because James Lighthill, a mathematician, was hired by the government to go and investigate artificial intelligence at Edinburgh. And he came back and said, Nothing of commercial value will ever come out of this place.

It was like one gigantic playpen kind of place. I was a founding member of the British Robotics Association and we all thought this was really going to have enormous relevance. But the funding agenciesRobotics! Whats this stuff? Were not going to fund this! And so there was a period around 72, I guess, when all the funding dried up and everybody said, Well, we had fun while we were here; better go and do something else.

Then its back to being a physicist. I came to Sweden and I got a job as a physicist programmer for the EISCAT scientific association. My boss had come from IBM and he was older than me and he wanted a specification and he would go and implement it. We used to argue over this. He said, Whats bad about the job is we dont have a job description and we dont have a detailed specification. And I said, Well, a job with no job description is a really good job. Because then you can form it how you like. Anyway, he left after about a year and I got the bosss job, the chief designer.

I designed a system for them and that was what I suppose youd call an application operating systemits something that runs on top of the regular operating system. By now computers were becoming quite reasonable. We had NORD-10 computers which were NorwegianI think they were an attempt to get into the PDP-11 market.

I worked there for almost four years. Then I got a job for the Swedish Space Corporation and built yet another application operating system to control Swedens first satellite, which was called Viking. That was a fun projectIve forgotten the name of the computer but it was a clone of the Amdahl computer. It still only had line editors. It didnt have full-screen editors. And all your programs had to be in one directory. Ten letters for the file name and three letters for the extension. And a Fortran compiler or assembler and thats it.

The funny thing is, thinking back, I dont think all these modern gizmos actually make you any more productive. Hierarchical file systemshow do they make you more productive? Most of software development goes on in your head anyway. I think having worked with that simpler system imposes a kind of disciplined way of thinking. If you havent got a directory system and you have to put all the files in one directory, you have to be fairly disciplined. If you havent got a revision control system, you have to be fairly disciplined. Given that you apply that discipline to what youre doing it doesnt seem to me to be any better to have hierarchical file systems and revision control. They dont solve the fundamental problem of solving your problem. They probably make it easier for groups of people to work together. For individuals I dont see any difference.

Also, I think today were kind of overburdened by choice. I mean, I just had Fortran. I dont think we even had shell scripts. We just had batch files so you could run things, a compiler, and Fortran. And assembler possibly, if you really needed it. So there wasnt this agony of choice. Being a young programmer today must be awfulyou can choose 20 different programming languages, dozens of framework and operating systemsand youre paralyzed by choice. There was no paralysis of choice then. You just start doing it because the decision as to which language and things is just madetheres no thinking about what you should do, you just go and do it.

Seibel: Another difference these days is that you can no longer understand the whole system from top to bottom. So not only do you have lots of choices to make, theyre all about which black boxes you want to use without necessarily fully understanding how they work.

Armstrong: Yeahif these big black boxes dont work properly, and you have to modify them, I reckon its easier just to start from scratch and just write everything yourself. The thing that really hasnt worked is software reuse. Its appallingly bad.

Seibel: Yet youre the architect not only of Erlang but of an application framework, the Open Telecom Platform. Is it reusable?

Armstrong: To an extent its reusable. But the same problem will occur. If that framework exactly solves your problemif some programmer who doesnt know anything about the design criteria for OTP looks at it in a few years time and says, Oh, thats great; thats exactly what I want to do, then its fine and you get this measure of reusability. If its not, then you have a problem.

Fairly recently Ive seen people say, This is really kind of artificial, were twisting the code to fit into this OTP framework. So I say, Well, rewrite the OTP framework. They dont feel they can change the framework. But the frameworks just another program. Its really rather easy. And I go into it and then it does what they want. They look at it and they say, Yeah, well, thats easy. They accept that its easy. But they say, Well, our project management doesnt want us messing around with the framework. Well, give it a different name then or something.

Seibel: But do you think its really feasible to really open up all those black boxes, look inside, see how they work, and decide how to tweak them to ones own needs?

Armstrong: Over the years Ive kind of made a generic mistake and the generic mistake is to not open the black box. To mentally think, this black box is so impenetrable and so difficult that I wont open it. Ive opened up one or two black boxes: I wanted to do a windowing system, a graphics system for Erlang, and I thought, Well, lets run this on X Windows. What is X Windows? Its a socket with a protocol on top of it. So you just open the socket and squirt these messages down it. Why do you need libraries? Erlang is message based. The whole idea is you send messages to things and they do things. Well, thats the idea in X Windowsyouve got a window, send it a message, it does something. If you do something in the window it sends you a message back. So thats very much like Erlang. The way of programming X Windows, however, is through callback librariesthis happens and call this. Thats not the Erlang way of thinking. The Erlang way of thinking is, send a message to something and do something. So, hang on, lets get rid of all these libraries in betweenlets talk directly to the socket.

And guess what? Its really easy. The X protocols got, I dont know, 100 messages, 80 messages or something. Turns out you only need about 20 of them to do anything useful. And these 20 messages you just map onto Erlang terms and do a little bit of magic and then you can start sending messages to windows directly and they do things. And its efficient as well. Its not very pretty because I havent put much effort into graphics and artistic criteriatheres a lot of work there to make it look beautiful. But its not actually difficult.

Another one is this typesetting system I did where the abstraction boundary I opened up is Postscript. As you get to that boundary you think, I dont want to go through the boundary, because whats underneath isyou imagineenormously complicated. But again, it turns out to be very easy. Its a programming language. Its a good programming language. The abstraction boundary is easy to go through and once youve gone through, theres a lot of benefit.

For my Erlang book, my publisher said, Weve got tools to make diagrams. But the thing I dont like about diagramming tools is its really difficult to get an arrow to meet exactly. And your hand hurts. I thought, The amount of time to write a program that spits out Postscript and then say, I want a circle there and the arrow goes exactly there, and get the program right, isnt long. It takes a few hours. Doing diagrams with programs takes about the same time as doing them in a WYSIWYG thing. Only there are two benefits. Your hand doesnt hurt at the end and even when you blow the thing up to a magnification of 10,000, the arrow points exactly right.

I cant say beginner programmers should open up all these abstractions. But what I am saying is you should certainly consider the possibility of opening them. Not completely reject the idea. Its worthwhile seeing if the direct route is quicker than the packaged route. In general I think if you buy software, or if you use other peoples software, you have to reckon with an extremely long time to tailor itit doesnt do exactly what you want, it does something subtly different. And that difference can take a very long time to solve.

Seibel: So you started out saying software reuse is appallingly bad, but opening up every black box and fiddling with it all hardly seems like movement toward reusing software.

Armstrong: I think the lack of reusability comes in object-oriented languages, not in functional languages. Because the problem with object-oriented languages is theyve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.

If you have referentially transparent code, if you have pure functionsall the data comes in its input arguments and everything goes out and leaves no state behindits incredibly reusable. You can just reuse it here, there, and everywhere. When you want to use it in a different project, you just cut and paste this code into your new project.

Programmers have been conned into using all these different programming languages and theyve been conned into not using easy ways to connect programs together. The Unix pipe mechanismA pipe B pipe Cis trivially easy to connect things together. Is that how programmers connect things together? No. They use APIs and they link them into the same memory space, which is appallingly difficult and isnt cross-language. If the language is in the same family its OKif theyre imperative languages, thats fine. But suppose one is Prolog and the other is C. They have a completely different view of the world, how you handle memory. So you cant just link them together like that. You cant reuse things. There must be big commercial interests for whom it is very desirable that stuff wont work together. It creates thousands of jobs for consultants. And thousands of tools to solve problems that shouldnt exist. Problems that were solved years ago.

I think its really weird that we have very few programming languages that describe the interaction between things. I keep coming back to ways of gluing things together and ways of describing protocols. We dont have ways of describing this protocol in between things: if I send you one of them then you send me one of these. We have ways of describing packets and their types but we have very restricted ways of describing the protocols.

Programming is fundamentally different to the way we construct things in the real world. Imagine youre a car manufacturer. You buy components from subcontractors. You buy a battery from Lucas and you buy a generator from somewhere. And you bolt things togetheryou construct things by placing things next to each other. You build a house by putting the bricks on top of each other and putting the door there. Thats how we make chips. You get a printed circuit board that basically just provides this connection. But you can think of making electronic things as you buy all these chips and you connect the legs of some to others with wires. And thats how you make hardware. But we dont make software like that. We should make software like that and we dont.

The reason we dont, has to do with concurrency. You see, the chips, when you put them next to each other, they all execute in parallel. And they send messages. They are based on this message-passing paradigm of programming, which is what I believe in. And thats not how we write software together. So I think one direction Erlang might take, or I would like it to take, is this component direction. I havent done it yet, but Id like to make some graphic front ends that make components and Id like to make software by just connecting them together. Dataflow programming is very declarative. Theres no notion of sequential state. Theres no program counter flipping through this thing. It just is. Its a declarative model and its very easy to understand. And I miss that in most programming languages.

Thats not to say that whats inside an individual black box isnt very complicated. Take grep, for example. Seen from the outsideimagine a little square. The input is a stream of data, a file. You say cat foo span grep and grep has got some arguments, its got a regular expression its got to match. OK. And out of grep come all the lines that match that regular expression. Now, at a perceptual level, understanding what grep does is extremely simple. It has an input which is a file. It has an input which is a regular expression. It has an output which is a set of lines or a stream of lines that match the regular expression. But that is not to say that the algorithm inside the black box is simpleit could be exceedingly complicated.

Whats going on inside the black boxes can be exceedingly complicated. But gluing things together from these complicated components does not itself have to be complicated. The use of grep is not complicated in the slightest. And what I dont see in system architectures is this clear distinction between the gluing things together and the complexity of the things inside the boxes.

When we connect things together through programming language APIs were not getting this black box abstraction. Were putting them in the same memory space. If grep is a module that exposes routines in its API and you give it a char* pointer to this and youve got to malloc that and did you deep copy this stringcan I create a parallel process thats doing this? Then it becomes appallingly complicated to understand. I dont understand why people connect things together in such complicated ways. They should connect things together in simple ways.

Seibel: Comparing how you think about programming now with how you thought when you were starting out, whats the biggest change in your thinking?

Armstrong: The big changes in how I think about programming have nothing to do with the hardware. Obviously its a lot faster and lot more powerful but your brain is a million times more powerful than the best software tools. I can write programs and then suddenly, days later, say, Theres a mistake in that programif this happens and that happens and that happens and this happens, then it will crash. And then I go and look in the codeyup, I was right. There has never been a symptom. Now you tell me a development system that can do that kind of stuff. So the changes that have happened as a programmer, theyre mental changes within me.

There are two changes and I think theyre to do with the number of years you program. One is, when I was younger quite often I would write a program and work at it until its finished. When it was finished I would stop working on it. It was done, finished. Then Id get an insightAh! Wrong! Idiot! Id rewrite it. Again: Yeah, its wrongrewrite it.

I remember thinking to myself, Wouldnt it be nice if I could think all of this stuff instead of writing it? Wouldnt it be nice if I could get this insight without writing it. I think I can do that now. So I would characterize that period, which took 20 years, as learning how to program. Now I know how to program. I was doing experiments to learn how to program. I think I know how to program now and therefore I dont have to do the experiments anymore.

Occasionally I have to do very small experimentswrite extremely small programs just to answer some question. And then I think through things and they more or less work as I expect when I program them because Ive thought through them. That also means it takes a long time. A program that you write, you get the insight, you rewriteit might take you a year to write. So I might think about it for a year instead. Im just not doing all this typing.

Thats the first thing. The second thing thats happened is intuition. When I was younger, I would do the all-night hacks, programming to four in the morning and you get really tired and its macho programmingyou hack the code in hour after hour. And its not going well and you persevere and you get it working. And I would program when the intuition wasnt there.

And what Ive learned is, programming when youre tired, you write crap and you throw it all away the next day. And 20 years ago I would program although I was getting a strong feeling that this isnt righttheres something wrong with this code. I have noticed over the years, the really good code I would write was when Im in complete flowjust totally unaware of time: not even really thinking about the program, just sitting there in a relaxed state just typing this stuff and watching it come out on the screen as I type it in. That codes going to be OK. The stuff where you cant concentrate and somethings saying, No, no, no, this is wrong, wrong, wrongI was ignoring that years ago. And Id throw it all away. Now I cant program anymore if it says, No. I just know from experience, stopdont write code. Stop with the problem. Do something else.

Because I was good at math and that sort of stuff at school, I thought, Oh, Im a logical person. But I took these psychology tests and got way high scores on intuition. And quite low scores on logical thinking. Not lowI can do math and stuff; Im quite good at them. But because I was good at math I thought science was about logic and math. But I wouldnt say that now. Id say its an awful lot of intuition, just knowing whats right.

Seibel: So now that you spend more time thinking before you code, what are you actually doing in that stage?

Armstrong: Oh, Im writing notesIm not just thinking. Doodling on paper. Im probably not committing much to code. If you were to monitor my activity itd be mostly thinking, a bit of doodling. And another thing, very important for problem solving, is asking my colleagues, How would you solve this? It happens so many times that you go to them and you say, Ive been wondering about whether I should do it this way or that way. Ive got to choose between A and B, and you describe A and B to them and then halfway through that you go, Yeah, B. Thank you, thank you very much.

You need this intelligent white boardif you just did it yourself on a white board theres no feedback. But a human being, youre explaining to them on the white board the alternative solutions and they join in the conversation and suggest the odd thing. And then suddenly you see the answer. To me that doesnt extend to writing code. But the dialog with your colleagues who are in the same problem space is very valuable.

Seibel: Do you think its those little bits of feedback or questions? Or is it just the fact of explaining it?

Armstrong: I think it is because you are forcing it to move it from the part of your brain that has solved it to the part of your brain that has verbalized it and they are different parts of the brain. I think its because youre forcing that to happen. Ive never done the experiment of just speaking out loud to an empty room.

Seibel: I heard about a computer science department where in the tutors office they had a stuffed animal and the rule was you had to explain your problem to the stuffed animal before you could bother the tutor. OK, Mr. Bear, heres the thing Im working on and heres my approachaha! There it is.

Armstrong: Really? I must try that.

Seibel: Talk to your cats.

Armstrong: The catsabsolutely! I worked with this guy who was slightly older than me and very clever. And every time Id go into his office and ask him a question, every single question, he would say, A program is a black box. It has inputs and it has outputs. And there is a functional relationship between the inputs and the outputs. What are the inputs to your problem? What are the outputs to your problem? What is the functional relationship between the two? And then somewhere in this dialog, you would say, Youre a genius! And youd run out of the room and he would shake his head in amazementI wonder what the problem was, he never said. So hes your bear which you explain the problem to.

Seibel: The doodlingis that writing little snippets of code or is it literally graphical doodles?

Armstrong: Its more bubbles with arrows. You know when you explain things to people on a white boardyou draw bubbles and arrows and equations and notations. Not code. Code fragmentspiddly bits of code sometimes because thats a compact way to express something. This is in the thinking period. Very occasional code experiments because I dont know how long it takes to do something. So Ill write ten lines of code and time something.

Seibel: You mean how long it takes for the computer to do it?

Armstrong: Yeah. Does that take a millisecond or a microsecondI dont know. I can guess but I want to confirm that guess. And so Im only looking at the bits I dont really know. But I have a great stock of experience programming Erlang so I know pretty much what things are going to do. Problem solving was the same years ago. It was, identify the difficult bits, write the small prototypes, identify the areas of uncertainty, writing very small bits of code. Essentially I do the same thing now but I have less reason to do these small experiments. If its Erlang. If Im doing Ruby or Java then I have to go back and do a lot of experiments because I dont know whats going to happen.

Seibel: Then somewhere in this thinking process you get to the point where you know how to write the code?

Armstrong: Yeah, then all the bits fit together. But maybe I cant explain it to anybody. I just get a very strong feeling that if I start writing the program now itll work. I dont really know what the solution is. Its like an egg. The chickens ready to lay the egg. Now Im ready to lay the egg.

Seibel: And thats the point at which you need to go into flow and not be interrupted.

Armstrong: Yes, yes.

Seibel: So there are still presumably a lot of details to be sorted out at the code level which requires your concentration.

Armstrong: Oh yes. But then there are two types of those things. The stuff that really needs the concentration is the stuff that is not automaticyouve got to think about it. Youve got this really tricky garbage collectionexactly what needs to be marked and exactly whereyouve got to think hard about that. You know youll find a solution because youve kind of bounded it in. And you know its in the right little black box.

Michelangelo is doing the roof of the Sistine Chapel or something and hes got a whole team of painters helping him. So he would sketch the big picture first. These huge areas have got to be done in blue and green. So thats rather like writing a program. The first sketch is this broad sketch where everythings in the right place. Some of these places are going to be filled with uniform color and just can be filled in fairly rapidlyyou dont have to think.

And then you get to the details of the eyesthats tricky stuff. You know you can do it. And the eye is in the right place because the picture is OK. So you go and do the eye and the detail. Thats not to say thats easythats the difficult bit, actually. Youve got to really concentrate while youre doing the eye. You dont have to really concentrate while youre doing the forehead or the cheeks because theyre fairly uniform. A bit of stubble here so you pay a sort of half concentration.

Then type it all in and get the syntax errors out and run a few little tests to make sure it works. And thats all rather relaxing. See a little compiler error there and you fix it. Once youre experienced at a language you dont even bother to read the diagnostic. It just says the line numberyou dont read what it says. That lineoh, yeah. Thats wrong, you retype it.

I gave a course in Erlang in Chicago. I was wandering around the class and Id notice, theres something wrong. Oh, theres a comma missing there or thatll crash before that happens and youre not linked. My wifes very good at proofreading and she says errors spring out of the page at you. A missing comma or a spelling mistakethey literally spring out of the page at her.

And programming errors just spring out of the page if I look at other peoples code, wandering around. It doesnt feel like conscious thought is involvedits holistic. You see everything on the screen and theres the error, bumpf. So its just a matter of correcting those surface errors.

One thats tricky is slight spelling errors in variable names. So I choose variable names that are very dissimilar, deliberately, so that error wont occur. If youve got a long variable like personName and youve got personNames with an s on the end, thats a list of person names, that will be something that my eye will tend to read what I thought it should have been. And so Id have personName and then listOfPeople. And I do that deliberately because I know that my eye will see what I thought Id written. But punctuation, I do see thatI do see the commas and the brackets as being wrong. And of course Emacs colors everything and auto-indents and the brackets are different colors. So this is really easy.

Seibel: At the point that you start typing code, do you code top-down or bottom-up or middle-out?

Armstrong: Bottom up. I write a little bit and test it, write a little bit and test it. Ive gone over to this writing test cases first, now. Unit testing. Just write the test cases and then write the code. I feel fairly confident that it works.

Seibel: Back to a bit of your history, it was after the Swedish Space Corporation that you went to Ericssons research lab?

Armstrong: Yes. And it was a very, very fortunate time to come, it must have been 84. I think I had come to the lab something like two years after it had started. So we were very optimistic. Our view of the world was, yes well solve problems and then well push them into projects and we will improve Ericssons productivity. This view of the world wasnt yet tinged by any contact with reality. So we thought it would be easy to discover new and useful stuff and we thought that once we had discovered new and useful stuff then the world would welcome us with open arms. What we learned later was, it wasnt all that easy to discover new stuff. And its incredibly difficult to get people to use new and better stuff.

Seibel: And Erlang was one of those new and useful things you expected them to use?

Armstrong: Yes. Absolutely. So what happened was, first of all it was just Prolog. I sort of made a little language and people started using it. And then Robert Virding came along and said, Hey, this looks like fun. And hed been reading my Prolog and he said, Can I modify it a bit? Thats pretty dangerous because Robert says that and you end up with one comment at the top of the program that says, Joe thought of this stuff and Ive changed a bit, and then its completely changed. So Robert and I just rewrote this stuff back and forth and we had great argumentsAhhh, I cant read your code, its got blanks after all the commas.

Then we found somebody inside Ericsson who wanted a new programming language or wanted a better way of programming telephony. We met up with them once a week for about, I cant remember, six months, nine months. And the general idea was we would teach them how to program and they would teach us about telephonywhat the problem was. I remember it was both frustrating and very stimulating. That changed the language because we had real people using it and that resulted in a study where they thought, Yeah, this would be OK but its far too slowthey measure the performance of it and said, Its gotta be 70 times faster. So then we said, This phase is now over. Well make it go 70 times faster and theyll carry on programming it and we have to do this in two years or something.

We had several false starts. And we had several really embarrassing moments. Big mistake: dont tell people how fast something is going to be before youve implemented it. But ultimately we figured out how to do it. I wrote a compiler in Prolog. And Rob was doing the libraries and things. Were now kind of two years in. Then I thought I could implement this abstract machine in C so I started writing my first-ever C. And Mike Williams came along and looked at my C and said, This is the worst C Ive ever seen in my entire life. This is appallingly bad. I didnt think it was that bad but Mike didnt like it. So then Mike did the virtual machine in C and I did the compiler in Prolog. Then the compiler compiled itself and produced byte-code and you put it in the machine and then we changed the grammar and the syntax and compiled the compiler in itself and came out with an image that would bootstrap and then were flying. Weve lost our Prolog roots and were now a language.

Seibel: Has there ever been anything that youve found difficult to work into the Erlang model?

Armstrong: Yeah. We abstract away from memory, completely. If you were turning a JPEG image into a bitmap data, which depends on the placement of the data in a very exact sense, that doesnt work very well. Algorithms that depend on destructively upgrading statethey dont work well.

Seibel: So if you were writing a big image processing work-flow system, then would you write the actual image transformations in some other language?

Armstrong: Id write them in C or assembler or something. Or I might actually write them in a dialect of Erlang and then cross-compile the Erlang to C. Make a dialectthis kind of domain-specific language kind of idea. Or I might write Erlang programs which generate C programs rather than writing the C programs by hand. But the target language would be C or assembler or something. Whether I wrote them by hand or generated them would be the interesting question. Im tending toward automatically generating C rather than writing it by hand because its just easier.

But Id use an Erlang structure. Ive got some stuff that does my family images and things. So I use ImageMagik with some shell scripts. But I control it all from Erlang. So I just write wrappers around it and call os:command and then the ImageMagik command. So its quite nice to wrap up things in. Wouldnt want to do the actual image processing in Erlang. Itd be foolish to write that in Erlang. Cs just going to be a lot better.

Seibel: Plus, ImageMagik is already written.

Armstrong: That doesnt worry me in the slightest. I think if I was doing it in OCaml then I would go down and do it because OCaml can do that kind of efficiency. But Erlang cant. So if I was an OCaml programmer: OK, what do I have to do? Reimplement ImageMagik? Right, off we go.

Seibel: Just because its fun?

Armstrong: I like programming. Why not? You know, Ive always been saying that Erlang is bad for image processingIve never actually tried. I feel it would be bad but that might be false. I should try. Hmmm, interesting. You shouldnt tempt me.

The really good programmers spend a lot of time programming. I havent seen very good programmers who dont spend a lot of time programming. If I dont program for two or three days, I need to do it. And you get better at ityou get quicker at it. The side effect of writing all this other stuff is that when you get to doing ordinary problems, you can do them very quickly.

Seibel: Is there anything that you have done specifically to improve your skill as a programmer?

Armstrong: No, I dont think so. I learned new programming languages but not with the goal of becoming a better programmer. With the goal of being a better language designer, maybe.

I like to figure out how things work. And a good test of that is to implement it yourself. To me programming isnt about typing code into a machine. Programming is about understanding. I like understanding things. So why would I implement a JPEG thing like we talked about earlier? Its because Id like to understand wavelet transforms. So the programming is a vehicle to understand wavelet transformations. Or why do I try to do an interface to X Windows? Because I wanted to understand how the X protocol worked.

Its a motivating force to implement something; I really recommend it. If you want to understand C, write a C compiler. If you want to understand Lisp, write a Lisp compiler or a Lisp interpreter. Ive had people say, Oh, wow, its really difficult writing a compiler. Its not. Its quite easy. There are a lot of little things you have to learn about, none of which is difficult. You have to know about data structures. You need to know about hash tables, you need to know about parsing. You need to know about code generation. You need to know about interpretation techniques. Each one of these is not particularly difficult. I think if youre a beginner you think its big and complicated so you dont do it. Things you dont do are difficult and things youve done are easy. So you dont even try. And I think thats a mistake.

Seibel: Several of the folks Ive talked to have recommended learning different programming languages because it gives you different perspectives on how to solve problems.

Armstrong: Languages that do different things. Theres no point learning lots of languages that all do the same thing. Certainly Ive written quite a lot of JavaScript and quite a lot of Tcl and quite a lot of C and quite a lot of Prologwell, an enormous amount of Prolog and an enormous amount of Fortran and an enormous amount of Erlang. And a bit of Ruby. A bit of Haskell. I sort of read all languages and Im not fluent at programming them all. Certainly I can program in quite a lot of languages.

Seibel: No C++?

Armstrong: No, C++, I can hardly read or write it. I dont like C++; it doesnt feel right. Its just complicated. I like small simple languages. It didnt feel small and simple.

Seibel: What languages influenced the design of Erlang?

Armstrong: Prolog. Well, it grew out of Prolog, obviously.

Seibel: Theres not a lot of Prolog discernible in it today.

Armstrong: Well, unificationpattern matching, that comes directly from Prolog. And the kind of data structures. Tuples and lists have slightly different syntax in Prolog but theyre there. Then there was Tony Hoares CSP, Communicating Sequential Processes. Also Id read about Dijkstras guarded commandsthats why I require that some pattern should always match, there shouldnt be a default caseyou should explicitly require that some branch always match. I think those are the main influences.

Seibel: And where did you get the functional aspect?

Armstrong: Once youve added concurrency to Prolog you really just had to make sure it didnt backtrack after youd done something. In Prolog you could call something and then backtrack over the solution to basically undo the effect of calling it. So you had to realize if this statement says, Fire the missiles, and whoom, off they go, you cant backtrack over it and reverse that. Pure Prolog programs are reversible. But when youre interacting with the real world, all the things you do are one way. Having said, fire the missiles, the missiles fire. Having said, Change the traffic lights from red to green, they change from red to green and you cant say, Oh, that was a bad decision; undo it.

Now weve got a concurrent language and parallel processes and inside these processes were doing full Prolog with backtracking and all that kind of stuff. So the Prolog became very deterministic with cuts everywhere to stop it from backtracking.

Seibel: Where the irreversible things would be sending messages to other processes?

Armstrong: Yes. But its just a function call and maybe not of the function that fires the rockets but one that calls something else that calls something else that calls it so its just a pain kind of trying to keep these two worlds separate. So the code you wrote inside a process became more and more functional, sort of a dialect of Prolog which was a functional subset. And so if its a functional subset, might as well make it completely functional.

Seibel: Yet Erlang is pretty different from most functional languages these days in being dynamically typed. Do you feel like part of the functional language community?

Armstrong: Oh yes. When we go to functional programming conferences, I suppose we argue about our differences. We argue about eager evaluation and lazy evaluation. We argue about dynamic type systems and static type systems. But despite everything the central core of functional programming is the idea of nonmutable statethat x isnt the name of a location in memory; its a value. So it cant change. We say x equals three and you cant change it thereafter. All these different communities say that has enormous benefits for understanding your program and for parallelizing your program and for debugging your program. Then there are functional languages with dynamic type systems like Erlang and functional languages with static type systems and theyve both got their good and bad points.

Itd be really nice to have the benefits of a static type system in Erlang. Maybe in certain places we could annotate programs to make the types more explicit so the compiler can derive the types and generate much better code.

Then the static type people say, Well, we really rather like the benefits of dynamic types when were marshaling data structures. We cant send an arbitrary program down a wire and reconstruct it at the other end because we need to know the type. And we haveCardelli called it a system thats permanently inconsistent. We have systems that are growing and changing all the time, where the parts may be temporarily inconsistent. And as I change the code in a system, its not atomic. Some of the nodes change, others dont. They talk to each otherat certain times theyre consistent. At other timeswhen we go over a communication boundarydo we trust that the boundary is correct? They might fib. So we need to check certain stuff.

Seibel: So early on you earned your beer by debugging other peoples programs. Why do you think you were such a good debugger?

Armstrong: Well, I enjoyed debugging. At this point in the program you print out a few variables and things to see whats going on and theyre all according to what you expect. And at this point in the program its right. And somewhere later its wrong. So you look halfway in betweenits either right or wrong and you just do this interval halving. Provided you can reproduce an error. Errors that are nonreproducible, thats pretty difficult to debug. But they werent giving me that. They were giving me reproducible errors. So just carry on halving until you find it. You must ultimately find it.

Seibel: So do you think you just had a more systematic view?

Armstrong: Yeah, they gave up. I dont know whyI couldnt really understand why they couldnt debug programs. I mean, do you think debugging is difficult? I dont. You just stop it and slow it down. I mean, Im just talking about batch Fortran.

OK, debugging real-time systems or garbage collectorsI remember once Erlang crashedit was early daysand it crashed just after Id started it. I was just typing something. It had built in sort of Emacsy commands into the shell. And I typed erl to start it and you get into a read-eval-print loop. And Id typed about four or five characters and made a spelling mistake. And then I backed the cursor a couple of times and corrected it and it crashed with a garbage collection error. And I knew thats a deep, deep, error. And I thought, Can I remember exactly what did I type in? Because it was only about 12 characters or something. I restarted and typed and it didnt crash. And I sat there for like an hour and a half trying probably a hundred different things. Then it crashed again! Then I wrote it down. Then I could debug it.

Seibel: What are the techniques that you use there? Print statements?

Armstrong: Print statements. The great gods of programming said, Thou shalt put printf statements in your program at the point where you think its gone wrong, recompile, and run it.

Then theresI dont know if I read it somewhere or if I invented it myselfJoes Law of Debugging, which is that all errors will be plus/minus three statements of the place you last changed the program. When I worked at the Swedish Space Corporation my boss was a hardware guy. We were up at Esrange, the rocket-launching site and satellite-tracking station in the north. And one time he was banging his head, debugging some bug in the hardware, plugging in oscilloscopes, and changing things. And I said, Oh, can I help? And he said, No Joe, you cant help herethis is hardware. And I said, Yeah, but it must be like softwarethe bug will be pretty near to the last change you made to the hardware. And he went, I changed a capacitor. Youre a genius! Hed replaced one capacitor with a bigger capacitor and he unsoldered it and put the original one back and it worked. Its the same everywhere. You fix your car and it goes wrongits the last thing you did. You changed somethingyou just have to remember what it was. Its true with everything.

Seibel: So have you ever proved any of your programs correct? Has that kind of formalism ever appealed to you?

Armstrong: Yes and no. Ive manipulated programs algebraically to just show that they were equivalent. Havent really gone into theorem proving as such. I did a course in denotational semantics and things like that. I remember giving up. The exercise was given: let x = 3 in let y = 4 in xplus y show that the eager evaluation scheme given by the equations foo and the lazy evaluation scheme given by the equations bar, both evaluate to seven.

Fourteen pages of lemmas and things later I thought, Hang onx is three, y is four, x plus y; yeah seven. At the time I was writing the Erlang compiler. If it took lots of pages to prove that three plus four is seven then the proof that my compiler was in any sense correct would have been thousands and thousands of pages.

Seibel: Do you prefer to work alone or on a team?

Armstrong: I like a workplace of teams, if you see what I mean. Im not antisocial. But I just like programming by myself. Certainly I like collaborating with people in the sense of discussing problems with them. I always thought the coffee break that you have when you got to work and out came all the ideas that youd had on your walk to work was very valuable. You get a lot of insights then. Good to thrash your ideas out in front of the crowd. Youre put in a position of explaining your ideas which, for me, moves them from one part of my brain to another part. Often when you explain things then you understand them better.

Seibel: Have you ever pair programmedsat down at a computer and produced code with another person?

Armstrong: Yeah. With Robert, Robert Virding. We would tend to do that when both of us were kind of struggling in the dark. We didnt really know what we were doing. So if you dont know what youre doing then I think it can be very helpful with someone who also doesnt know what theyre doing. If you have one programmer whos better than the other one, then theres probably benefit for the weaker programmer or the less experienced programmer to observe the other one. Theyre going to learn something from that. But if the gaps too great then they wont learn, theyll just sit there feeling stupid. When I have done pair programming with programmers about the same ability as me but neither of us knew what we were doing, then its been quite fun.

Then there are what I might call special problems. I wouldnt attempt them if Ive got a cold or Im not on good physical form. I know its going to take three days to write and Ill plan a day and not read email and start and its gonna be four hours solid. Ill do it at home so I know I wont be interrupted. I just want to do it and get into this complete concentrated state where I can do it. I dont think pair programming would help there. It would be very disruptive.

Seibel: Whats an example of that kind of problem?

Armstrong: Figuring out bits of a garbage collectorits the imperative codingwhere youve got to remember to mark all those registers. Or doing some lambda lifting in the compiler, which is pretty toughyou relabel all the variables and then youve got four or five layers of abstract data types all messing around and frames with different stuff in them and you think, Ive got to really understand this, really think deeply about it. You want to concentrate.

I vary the tasks I do according to mood. Sometimes Im very uninspired so I think to myself, Ah, who shall I go and disturb now. Or Ill read some emails. Other times I feel, right now Im going to do some hard coding because Im in the mood for it. Youve got to be sort of right to do the coding. So hows that going to work with two people? One of them is just not in a concentrating mode and wants to read his emails and things.

Seibel: You did do a kind of serial pair programming with Robert Virding, when you passed the code back and forth rewriting it each time.

Armstrong: Yeah. One at a time. I would work on the program, typically two or three weeks, and then Id say, Well, Ive had enough, here you are, Robert. And hed take it. Every time we did this, it would come back sort of unrecognizable. He would make a large number of changes and itd come back to me and Id make a large number of changes.

Seibel: And they were productive changes?

Armstrong: Oh, absolutely. I was delighted if he found better ways of doing things. We both got on very well. He used to generalize. I remember once I found a variableI followed it round and round through about 45 routines and then, out it came, at the end, never even used. He just passed this variable in and out of 45 different functions. I said, Whats that for? You dont use it. He said, I know. Reserved for future expansion. So I removed that.

I would write a specific algorithm removing all things that were not necessary for this program. Whenever I got the program, it became shorter as it became more specific. And whenever Robert took my program it became longer, adding generality. I believe this Unix philosophya program should do what its supposed to do and nothing else. And Roberts philosophy is it should be a general program and then the program itself should be a specific case of the general program. So he would add generality and then specialize it.

Seibel: That seems like a pretty deep philosophical divide. Was there any benefit to having the program go through those two extremes?

Armstrong: Oh yes. Every cycle it improved. I think it was a lot better because of that. And probably better than either of us could have done on our own.

Seibel: Can you talk about how you design software? Maybe take example of something like OTP.

Armstrong: OTP was designed by me and Martin Bj?rklund and Magnus Fr?berg. There were just the three of us did the original design. We met every morning at coffee and had a long conversationabout an hour to two hoursand we covered the white board in stuff. Id take loads of notesI wrote all the documentation immediately and they wrote all the code. Sometimes Id write a bit of code as well. And when I was writing the documentation Id discover, I cant describe this, we have to change it. Or they would run into me and say, Nah, it doesnt work; this idea we had this morning, because of this, this, this, and this it doesnt work. At the end of the day we either got to the point where we got all the documentation and all the code or enough of the code and enough of the documentation that we knew it was going to work. And then we called it a day.

Some days it didnt work so we said, OK, well do it again tomorrow. There wasnt enough time to do a second pass in a day. But about one pass in a day worked fine. Because it gives us about two hours to discuss it in the morning, about two hours to write the documentation or code it up. And if you spent four hours really thinking hard, thats a good days work. So that worked very, very well. I dont know how long we worked like that for. Ten weeks, twelve weeks, something like that. And then we got the basic framework and then we had more people. Wed specified the architecturenow we could start growing it. Wed get three or four more programmers in.

Seibel: And then how did you divvy up the work for those new folks?

Armstrong: Well, we knew what were prototypes and what were final versions. Ive always taken the view of system design, you solve the hard problems first. Identify the hard problems and then solve them. And then the easy problems, you know theyll just come out in the wash. So theres a bit of experience there in classifying them as easy and hard. I know IP failover or something like that is going to be fairly hard. But I know that parsing a configuration file is going to be easy. In the prototype you might just have a configuration file that you read. You dont syntax check ityou dont have a grammar. In the production version you might do it in XML and have a complete grammar and validate it. But you know that thats a mechanical step to do that. It will take a competent programmer several weeks, or whatever time it takes. But its doable, its predictable in time, and there shouldnt be any nasty surprises on the way. But getting the communication protocols right, and getting them working properly when things fail, that I would do in a small group.

Seibel: So in this case you wrote the documentation before, or at least while, the code was being written. Is that how you usually do it?

Armstrong: It depends on the difficulty of the problem. I think with very difficult problems I quite often start right by writing the documentation. The more difficult it is, the more likely I am to document it first.

I like documentation. I dont think a program is finished until youve written some reasonable documentation. And I quite like a specification. I think its unprofessional these people who say, What does it do? Read the code. The code shows me what it does. It doesnt show me what its supposed to do. I think the code is the answer to a problem. If you dont have the spec or you dont have any documentation, you have to guess what the problem is from the answer. You might guess wrong. I want to be told what the problem is.

Seibel: Is the documentation you write at this stage internal documentation that another programmer would read or documentation for the user?

Armstrong: Its for user guides. It sort of switches me into a different mode of thinking. I just start, in order to do this, create a directory called that, put this file in there, rename this as that and that is guiding the structure. Ive sort of pondered the question. I bet Knuth would say, Well, all programs are literate programs. You dont write the code and then write the documentation. You write both at the same time, so its a literate program. Im not there. I dont think that. I dont know if his view is because he publishes his programs.

I dont know if its a left-brain/right-brain shift, or what it is, but when you write the documentation you think about the program differently to when you write the code. So I guess writing literate programs forces that shift as youre doing it. Which might be very productive. I did do some literate Erlang though I havent actually used it for a very long time. So thats an interesting ideaperhaps I should wake it up again and write some stuff using literate Erlang. Im not against the idea but Im sort of impatient and wanted to write the code and not the documentation. But if you really want to understand it then I think writing the documentation is an essential step.

If I were programming Haskell, I would be forced to think about the types pretty early and document them and write them down. If youre programming in Lisp or Erlang you can start writing the code and you havent really thought about the types. And in a way, writing the documentation is thinking about the types in a way. I suppose you start off with is a. You say, A melody is a sequence of notes. Right. OK. A melody is a sequence of chords where each chord is a parallel composition of notes of the same duration. Just by defining terms in your documentationa something is a somethingyoure doing a sort of type analysis and youre thinking declaratively about what the data structures are.

Seibel: Do you think overall programming languages are getting better? Are we on a trajectory where we learn enough lessons from the past and come up with enough new ideas?

Armstrong: Yes. The new languages are good. Haskell and things like that. Erlang. Then there are some funny languages that should really be used. Prolog is a beautiful language but not widely used. It sort of peaked; Kowalski called it a solution looking for a problem.

Seibel: Dan Ingalls mentioned Prolog as an example of the kind of idea that we should really revisit now that weve had a couple decades of Moores Law.

Armstrong: Prolog is so different to all the other programming languages. Its just this amazing way of thinking. And its not appropriate to all problems. But it is appropriate to an extremely large set of problems. Its not widely used. And its a great shame because programs are incredibly short. I think I went into shock when I wrote my first Prolog program. Its a kind of shocking experience. You just walk around going, wheres the programI havent written a program. You just told it a few facts about the system, about your problem. Here it is figuring out what to do. Its wonderful. I should go back to Prologdrop Erlang.

Seibel: Are there other skills that are not directly related to programming that you feel have improved your programming or that are valuable to have as a programmer?

Armstrong: Writing is. Theres some computer scientist that said, Oh, if youre no good at English youll never be a very good programmer.

Seibel: I think Dijkstra had something about that.

Armstrong: Ive occasionally been asked to advise people at universities on choice of syllabus subjects for computer science courses, being as how I work for industrywhat does industry want? And I say, Well, turn em out being able to write and argue cogently. Most graduates who come out, and theyve got degrees in computer science, writings not their strong point.

I think its actually very difficult to teach because its very individual. Somebodys got to take your text and a red pen and explain to you what you did wrong. And thats very time consuming. Have you ever read Hammings advice to young researchers?

Seibel: You and Your Research?

Armstrong: He says things like, Do good stuff. He says, If you dont do good stuff, in good areas, it doesnt matter what you do. And Hamming said, I always spend a day a week learning new stuff. That means I spend 20 percent more of my time than my colleagues learning new stuff. Now 20 percent at compound interest means that after four and a half years I will know twice as much as them. And because of compound interest, this 20 percent extra, one day a week, after five years I will know three times as much, or whatever the figures are. And I think thats very true. Because I do research I dont spend 20 percent of my time thinking about new stuff, I spend 40 percent of my time thinking about new stuff. And Ive done it for 30 years. So Ive noticed that I know a lot of stuff. When I get pulled in as a troubleshooter, boom, do it that way, do it that way. You were asking earlier what should one do to become a better programmer? Spend 20 percent of your time learning stuffbecause its compounded. Read Hammings paper. Its good. Very good.

Seibel: Do you find some code beautiful?

Armstrong: Yes. Why this is I dont know. The funny thing is, if you give two programmers the same problemit depends on the problem, but problems of a more mathematical nature, they can often end up writing the same code. Subject to just formatting issues and relabeling the variables and the function names, its isomorphicits exactly the same algorithms. Are we creating these things or are we just pulling the cobwebs off? Its like a statue thats there and were pulling the cobwebs off and revealing the algorithm thats always been there. So are we inventing a new algorithm or are we inventing a structure that already exists? Some algorithms feel like that. I think its more the mathematical algorithms. I dont get that feeling when Im implementing a telephony protocol or something. Thats not a statue that Im pulling the cobwebs off.

Seibel: So thats similar to the beauty of math, because its part of nature. Then there are other levels at which code sort of has an aesthetic.

Armstrong: Yeah. Its kind of feng shui. I like minimalistic code, very beautifully poised, structured code. If you start removing things, if you get to the point where if you were to remove anything more it would not work any moreat this point it is beautiful. Where every change that you could conceivably make, makes it a worse algorithm, at that point it becomes beautiful.

Seibel: You mentioned that when you and Robert Virding were passing the code back and forth how each of you changed the low-level details of formatting, stuff that programmers argue endlessly about.

Armstrong: Thats not affecting the beauty of the algorithm.

Seibel: But its part of the aesthetic. Its peoples taste.

Armstrong: Yeah. But I wouldnt say, This is ugly code because theres a blank after the comma. Ugly is when its done with a linear search and it could have been done with a binary interval halving. Or it could have done logarithmically and its done linearly. For the wrong reasons. Sure do it linearly if we know were searching through a list of ten elements, who cares? But if its a big data structure then it should have been done with a binary search. And so its really not very pretty to do it in a linear form. The mathematical algorithmsthats like Platonic beauty. This is more like architecture. You admire a fine buildingits not a mathematical object. Not a solid or a sphere or a prismits a skyscraper. It looks nice.

Seibel: What makes a good programmer? If you are hiring programmerswhat do you look for?

Armstrong: Choice of problem, I think. Are you driven by the problems or by the solutions? I tend to favor the people who say, Ive got this really interesting problem. Then you ask, What was the most fun project you ever wrote; show me the code for this stuff. How would you solve this problem? Im not so hung up on what they know about language X or Y. From what Ive seen of programmers, theyre either good at all languages or good at none. The guy whos a good C programmer will be good at Erlangits an incredibly good predictor. I have seen exceptions to that but the mental skills necessary to be good at one language seem to convert to other languages.

Seibel: Some companies are famous for using logic puzzles during interviews. Do you ask people that kind of question in interviews?

Armstrong: No. Some very good programmers are kind of slow at that kind of stuff. One of the guys who worked on Erlang, he got a PhD in math, and the only analogy I have of him, its like a diamond drill drilling a hole through granite. I remember he had the flu so he took the Erlang listings home. And then he came in and he wrote an atom in an Erlang program and he said, This will put the emulator into an infinite loop. He found the initial hash value of this atom was exactly zero and we took something mod something to get the next value which also turned out to be zero. So he reverse engineered the hash algorithm for a pathological case. He didnt even execute the programs to see if they were going to work; he read the programs. But he didnt do it quickly. He read them rather slowly. I dont know how good he would have been at these quick mental things.

Seibel: Are there any other characteristics of good programmers?

Armstrong: I read somewhere, that you have to have a good memory to be a reasonable programmer. I believe that to be true.

Seibel: Bill Gates once claimed that he could still go to a blackboard and write out big chunks of the code to the BASIC that he written for the Altair, a decade or so after he had originally written it. Do you think you can remember your old code that way?

Armstrong: Yeah. Well, I could reconstruct something. Sometimes Ive just completely lost some old code and it doesnt worry me in the slightest. I havent got a listing or anything; just type it in again. It would be logically equivalent. Some of the variable names would change and the ordering of the functions in the file would change and the names of the functions would change. But it would be almost isomorphic. Or what I would type in would be an improved version because my brain had worked at it.

Take the pattern matching in the compiler which I wrote ten years ago. I could sit down and type that in. It would be different to the original version but itd be an improved version if I did it from memory. Because it sort of improves itself while youre not doing anything. But itd probably have a pretty similar structure.

Im not worried about losing code or anything like that. Its these patterns in your head that you remember. Well, I cant even say you remember them. You can do it again. Its not so much remembering. When I say you can remember a program exactly, I dont think that its actually remembering. But you can do it again. If Bill could remember the actual text, I cant do that. But I can certainly remember the structure for quite a long time.

Seibel: Is Erlang-style message passing a silver bullet for slaying the problem of concurrent programming?

Armstrong: Oh, its not. Its an improvement. Its a lot better than shared memory programming. I think thats the one thing Erlang has doneit has actually demonstrated that. When we first did Erlang and we went to conferences and said, You should copy all your data. And I think they accepted the arguments over fault tolerancethe reason you copy all your data is to make the system fault tolerant. They said, Itll be terribly inefficient if you do that, and we said, Yeah, it will but itll be fault tolerant.

The thing that is surprising is that its more efficient in certain circumstances. What we did for the reasons of fault tolerance, turned out to be, in many circumstances, just as efficient or even more efficient than sharing.

Then we asked the question, Why is that? Because it increased the concurrency. When youre sharing, youve got to lock your data when you access it. And youve forgotten about the cost of the locks. And maybe the amount of data youre copying isnt that big. If the amount of data youre copying is pretty small and if youre doing lots of updates and accesses and lots of locks, suddenly its not so bad to copy everything. And then on the multicores, if youve got the old sharing model, the locks can stop all the cores. Youve got a thousand-core CPU and one program does a global lockall the thousand cores have got to stop.

Im also very skeptical about implicit parallelism. Your programming language can have parallel constructs but if it doesnt map into hardware thats parallel, if its just being emulated by your programming system, its not a benefit. So there are three types of hardware parallelism.

Theres pipeline parallelismso you make a deeper pipeline in the chip so you can do things in parallel. Well, thats once and for all when you design the chip. A normal programmer cant do anything about the instructionlevel parallelism.

Theres data parallelism, which is not really parallelism but it has to do with cache behavior. If you want to make a C program go efficiently, if *p is on a 16-byte boundary, if you access *p, then the access to *(p + 1) is free, basically, because the cache line pulls it in. Then you need to worry about how wide the cache lines arehow many bytes do you pull in in one cache transfer? Thats data parallelism, which the programmer can use by being very careful about their structures and knowing exactly how its laid out in memory. Messy stuffyou dont really want to do that.

The other source of real concurrency in the chip are multicores. Therell be 32 cores by the end of the decade and a million cores by 2019 or whatever. So you have to take the granules of concurrency in your program and map them onto the cores of the computer. Of course thats quite a heavyweight operation. Starting a computation on a different core and getting the answer back is itself something that takes time. So if youre just adding two numbers together, its just not worth the effortyoure spending more effort in moving it to another core and doing it and getting the answer back than you are in doing it in place.

Erlangs quite well suited there because the programmer has said, I want a process, I want another process, I want another process. Then we just put them on the cores. And maybe we should be thinking about actually physically placing them on cores. Probably a process that spawns another process talks to that process. So if we put it on a core thats physically near, thats a good place to put it, not on one thats a long way away. And maybe if we know its not going to talk to it a lot maybe we can put it a long way away. And maybe processes that do I/O should be near the edge of the chipthe ones that talk to the I/O processes. As the chips get bigger were going to have to think about how getting data to the middle of the chip is going to cost more than getting it to the edge of the chip. Maybe youve got two or three servers and a database and maybe youre going to map this onto the cores so well put the database in the middle of the chip and these ones talk to the client so well put them near the edge of the chip. I dont knowthis is research.

Seibel: You care a lot about the idea of Erlangs way of doing concurrency. Do you care more about that ideathe message-passing shared-nothing concurrencyor Erlang the language?

Armstrong: The ideaabsolutely. People keep on asking me, What will happen to Erlang? Will it be a popular language? I dont know. I think its already been influential. It might end up like Smalltalk. I think Smalltalks very, very influential and loved by an enthusiastic band of people but never really very widely adopted. And I think Erlang might be like that. It might need Microsoft to take some of its ideas and put some curly braces here and there and shove it out in the Common Language Runtime to hit a mass market.


: 0.040. /Cache: 0 / 0