{'EXIT', joe, goodbye}

April 2019 ยท 7 minute read

Jesper, I have this idea in which we’ll connect all of the worlds Erlang systems to each other, imagine if every process could talk to every other process, world-wide!

— Joe Armstrong

Joe was never short on ideas when you spoke to him. At conferences, he would engage people with his newest idea, or he would find people across the room and connect them. This often resulted in new acquaintances, interesting conversations, and new insights. Joe acted as the fountain from which insights sprang. He always had a new project going, and was keen to tell about it.

Joe would speak to everyone. It didn’t matter if they were new to the world of Erlang, or computers, or if you had 20 years of experience. He would quickly find your level, and then discuss at that level.

In my mind, three rules embodied the ideas of Joe:

  • They were always practical and wanted to solve a grand problem with computers. Most often, the limitations of physics played a role.

  • They always felt a little bit crazy, mainly due to the novelty of the idea.

  • They were going to restructure your brain, and how you thought about stuff.

When I was writing a BitTorrent client for Erlang, Joe was quick to comment on the code “This part uses defensive code style, you can probably just let it crash.” It took some years for that lesson to fully sink in, as it did with most of Joes stuff. He would gently nudge you in a direction, and by following his trajectory, you landed perfectly.

Most people reading this will know Joe for his work on Erlang. But he had lots of ideas, and not all of them pertained directly to Erlang itself, though it was often the vehicle. He had a keen interest in music and did collaboration on the Sonic Pi with Sam Aaron. Up until recently, Joe was working on improving wiki tooling. In particular, he’d realized how he needed a quine,[1] the wiki should contain its own source code so it could reproduce itself. This would ensure the longevity of the wiki. This work was in April 2019, so he was working on stuff up until his untimely death.

Joe liked to find minimal solutions to problems. He would strip layer upon layer off a problem until he had the core. He created a stack of universal binary formats, UBF. The ingenious part of this was that to transfer a term, one would transfer a program which when executed on a small stack-based virtual machine would yield said term. And the commands were chosen from the printable ASCII alphabet.

The higher layers in the stack was the basis for a lot of discussions I had with Joe. Both of us agreed that what happens “inside” an Erlang process wasn’t that interesting in the grand scale of things. It was the communication which was important, and it should have a contract system. Joe had certainly invented most of this before I even started looking at Erlang as a language. And I still—​to this day—​believe that this is future of protocol communication.

To see how visionary Joe was, his idea was to use such a contract system to get any Erlang process on any node to talk to another Erlang process, somewhere out in the universe. Essentially, this gives you “Serverless” operation, so he was a couple of years ahead of the pack on that.

Joe also wanted to mix this with a content addressable storage spanning the internet. He would speak fondly of IPFS which to a large extent was this vision. The main idea is to generate the reference key for data from the data itself, usually a cryptographic hash over the data. This in turn provides integrity: if the data changes, so does its key. Now, if you refer to data by its key, then you can verify you got the right data. Joe wanted to use this as a basis for software: “I can send you the hash, and you can fetch the library if you don’t have it.” he told me. He also wanted to use this idea to protect old software so “It could still run after many years.”--another reason Joe preferred minimalistic approaches to software. “You see, a package version is the hash of its source code, not a version number, that would be silly.”

He was adamant on making computers useful.

The greatest brain restructure, however, was always in the idea that your abstract logic of a program has to execute in the physical environment. This meant coping with failure when it happened as the only way to build robust software. Programming without this profound insight—​software cannot proactively remove all error—​is as silly as it is dangerous in hindsight.

Once you embrace fault tolerance, then you have the feeling of a burden released, and programs gets far easier to write. Of course, the more seasoned programmer knows when you can be proactive and when you have to be reactive. But if there one thought which has shaped the way I think about programming the most, it is this.

The second greatest brain restructure was Joe’s dismissal of performance in software. In a post to the Erlang mailing list, Joe would come up with the following table:

This is what we do: (We = ///)

    1) hack it in erlang
    2) fast enough - ship it
    3) tweak the erlang
    4) fast enough? - ship it
    5) hack it in C?
    6) fast enough - ship it
    7) tweak the C
    8) fast enough? - ship it
    9) make an FPGA
    10) fast enough - ship it
    11) make an ASIC
    12) fast enough - ship it

As you go down this list things cost more and more - step 11 costs
1M$/try - to pass step 9
you need to have a high volume product (10's to hundreds of thousands of units)

The core idea is that if you want to go really fast, you need to implement hardware specific to the problem, and you can only hope to tweak the software so far in performance. And he would make a point about setting up a specified target before starting the tweakery, so you’d know when to stop. Over the years, I’ve veered in the same general direction as Joe.

One thing I liked about Joe was his fearless approach to computing and to life. He’d never bow to authority. He’d would never be afraid to provoke if it was necessary. He never ever stopped R&D. He would tinker with things until he understood them, then find something new to look at. But only after he told you about his findings. Sharing was his modus operandi.

At conferences—​in which Joe was an attendant—​there would be this hallway track going on where people would talk with Joe. A slot would be skipped here and there. But some interesting conversation would be had, and people would convene around him in a circle that grew ever larger. I’ve watched more than once when a “innocent bystander” was drawn into his web of stories, findings, and insights.

Being a speaker with Joe in the audience was always a blast. Once he sensed he was allowed to interact, you could be sure he would “heckle” your talk in the most awesome way. It only took a smirk and you saying {hello, joe} and that would be his cue. We should have mic’ed him up at times.

When he did speak up, he would often frame a question such that it put a speaker in a position where they could really show their work. Never have I heard a bad question by Joe.

This was the Joe I knew. I think we owe him a lot with regard to pushing software and computer science ahead. The best we can do is to make sure his grand ideas are not lost upon us, and that they are implemented in the future to come.

And then perhaps, we will reach a point where Joe’s prediction from this Month will ring true:

One day computers might become useful.

— Joe Armstrong

1. A quine is a self-reproducing program