jlouis' Ramblings

Musings on tech, software, and other things

Two Technologies which are Bad for You

For some odd reason, we end up in situations in this industry which looks even more grave than earlier. In this post, I am going to rant. I am going to rant on two things, which I think contributes negatively. I wanted to write a nice clean post, but rants sometimes fits ones mood better, I guess.

JSON

How did we get JSON? Well, we had ASN.1 which is a fine format for many things. But the complexity of that format had us rethink the way we do computing. XML was born. XML is arguably the worst format ever. It is expensive to parse. It carries little structure. There are 2–3 ways of encoding the same data. And it has a myriad of complicated layers for which most people can only comprehend a small fraction. At once I knew these intricately — but today I have luckily forgotten most of the details.

Enter JSON. It is a simple format. Everyone can write a somewhat decent parser/unparser for it. So it won. But JSON is utterly miserable:

First off, we would have been better off if we locked down the encoding to UTF-8. That is, the media type[0] for JSON documents mandates UTF-8 and then we have separate media types for insane encodings. This would allow us to come with a big hammer and reject any JSON document encoded in other ways.

Second, JSON inherits all the miserable bad traits from everyones favorite language to hate: javascript.

In JSON, a number is a crippled IEEE 754 double floating point value (binary64). In 754, we use 52 bits for the significand with an extra bit implicitly present. This gives 53 bits of precision. Thus, up until the value 9007199254740992 we have a precise integer representation, but once we go beyond this, we lose precision. Try storing the above number +1 in a piece of code. It is not representable directly in JSON, which means you have to resort to other solutions. The most common one being to write the string “9007199254740993” into the JSON document. But this only works well for languages with implicit conversions of types. Any other language will not work well for this construction.

There is no distinction between binary data and string data. Since the encoding is UTF-8 this is important, since there are byte-sequences which are not valid UTF-8. The consequence is that whenever you want to send binary data, you have to encode it, typically in base64 or base85. This leads to each subsystem having to implement the same encoding and agree. Whenever your code has to work with a new kind of JSON data, you have to adhere to its rules which requires more code.

JSON has a dictionary/object type. This type only allows keys which are strings. That is, the type is (using Erlang spec’s)

-type json() :: … | [{string(), json()]}.

Storing anything indexed by a non-string family is impossible. No using the values ‘true’, ‘false’ or a number value as the index. This seems oddly limiting. Again, you have to resort to a predetermined encoding as above for integers to make this work out.

The type Zoo of JSON is extremely weak. There are almost no constructions. Since there is no well-defined way of tagging subtrees in JSON, you can’t really encode things. You need to define encodings that looks like

["~#Date", DateRep]

where the notion “~#Date” says that second element of the array is a representation of dates in some predetermined format.

Another problem is that JSON has no built-in hyperlink which utterly crippled HATEOAS (see the next section).

Finally, JSON is expensive to parse. It has a self-describing nature so keys in objects repeats in the data. This leads to having to traverse more bytes leading to slower parses. Furthermore, getting parsing right slows down the parser rather than speeds it up; encouraging hacks of incorrectness.

The Solution

Luckily, the guys at Cognitect thought about this and more. They defined the format Transit[2, 3] in which all of the limitations have been cleared away. While the format still has some way to go, it is a far better solution than what we have now.

To be viable, Transit needs a more formal specification of its implicit caching. I have spent some time with an OCaml and Erlang implementation (the Erlang implementation together with Isaiah Peng). And we are far from something that is stable enough for daily use. But given enough time, it will eventually improve to the point where it can replace JSON.

Transit is clever because it reuses JSON (or MsgPack) as the underlying transport. This enables systems to interoperate with Transit, as long as they have a proper Transit library. It also means any new self-describing format can probably used as a transport for Transit. The de-coupling of format from transport is a nice welcome change. The architecture layers nicely.

REST HTTP APIs

Here is a recent announcement by Google[4]:

Designed to let you easily deliver Gmail-enabled features, this new API is a standard Google API, which gives RESTful access to a user’s mailbox under OAuth 2.0 authorization. It supports CRUD operations on true Gmail datatypes such as messages, threads, labels and drafts.

Stop right there! This is not a protocol. It is an endpoint in the Google infrastructure which requires the Google infrastructure to operate. If you build anything on this, you tie yourself into that infrastructure, with no way to get out again.

It is true of any “API” by any company: Micro-soft, Facebook, Twitter. API is originally “Application Programming Interface”. But in this new-fangled context, it is more akin to “Automatic Prison Installation” in which your system becomes part of a walled garden.

The internet is built on open protocols. But “these are not the protocols you are looking for”. An API is not an open, decentralized protocol. It is a closed environment which is closed for extension. Worse, the success of Javascript and HTTP have forced everyone to speak HTTP to solve their problems. To interact in the modern world is to speak HTTP, preferably with JSON or XML data (oh, the horror!).

REST is a magnificent idea[5]. But we aren’t doing it. We are doing some subset of what REST entails in many situations. We ignore HATEOAS, mostly because JSON can’t represent links in unique ways. We can’t even agree on a standardized de-facto way of doing pagination in REST environments because so many systems only understands a subset of what it entails. Interaction with old systems has you breaking REST again and again, making it a sorry replacement for the original ideas of Fielding[6].

The end result is more walled gardens with closed implementations of software. We are tightly coupling our work to the work of the big centralized companies which are only keen on selling information about our behavior to 3rd parties, or doing a business by acting as middlemen, pairing our behavior with 3rd parties.

What made a protocol like HTTP or TCP beautiful is that they specify a minimal set of requirements for interoperation. TCP is very wide in the sense you can implement it on an Arduino shield with one buffer around 1 kilobyte. And also in hardware on a 40 gigabit Ethernet card. And those two implementations can talk to each other, albeit slowly. TCP is simple enough you can build a conforming implementation on a Commodore 64 or less. Likewise, a minimal HTTP implementation is easily possible.

The reason this works is because the protocol specify minimal requirements while being open for extension. None of these new APIs do that. Ugh!

The solution

We need to go back to our roots and start building protocols again. This change will never come from a large company. It has to rely on an open tinkerer culture. We need well-defined protocols and multiple implementations of these. Protocol design is quickly becoming a lost art. Rather, people are satiated with “APIs” to the point where they can’t recover.

We need de-centralization. The internet is built on the idea of end-to-end interaction between machines. The current state is more of a client/server infrastructure in which few central entities drive the status quo.

We need distribution. If we rely on big players, we tie ourselves to their solutions. This will be a bane going forward as we build infrastructure around vendors only interesting in lock-in.

And finally, we need education. A lot of the current “protocol design” is so bad compared to what can be found in old RFCs. If you want to implement something new, you need to study the past a lot before you build it. If you reject an old idea, you need to explain why. If you reinvent an old idea, you need to know you reinvented it and what happened historically for that idea not to catch on.

If we don’t do anything, the internet as we know it will be taken from us.

[0] Some call this MIME-type, but the name was changed a long time ago. [1] See http://en.wikipedia.org/wiki/Double-precision_floating-point_format [2] See http://blog.cognitect.com/blog/2014/7/22/transit [3] https://github.com/cognitect/transit-format [4] http://googleappsdeveloper.blogspot.dk/2014/06/introducing-new-gmail-api.html [5] http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http [6] Roy T. Fielding: http://en.wikipedia.org/wiki/Roy_Fielding  —  funnily enough, as of this writing, someone stuck a link to Sinatra on Roy’s wikipedia page. Sinatra is a Ruby-framework for doing “REST”. It is funny because Sinatra is everything — except for being a nice HTTP citizen. If you want a better approach, look at Webmachine or Cowboy-REST (both Erlang projects).