[HTTP APIs & REST] Advantages and Disadvantages of HTTP APIs
With this post, I’m continuing publishing the v2 of my book dedicated to APIs. If you like this book, please rate it on GitHub, Amazon, or Goodreads and
Chapter 36. Advantages and Disadvantages of HTTP APIs
After the three introductory chapters dedicated to clarifying terms and concepts (that's the price of the popularity of the technology) the reader might wonder why this dichotomy exists in the first place, i.e., why do some HTTP APIs rely on HTTP semantics while others reject it in favor of custom arrangements? For example, if we consider the JSON-RPC response format, we will quickly notice that it might be replaced with standard HTTP protocol functionality. Instead of this:
HTTP/1.1 200 OK
{
"jsonrpc": "2.0",
"id",
"error": {
"code": -32600,
"message": "Invalid request"
}
}
the server could have simply responded with 400 Bad Request
, passing the request identifier as a custom header like X-OurCoffeeAPI-Request-Id
. Nevertheless, protocol designers decided to introduce their own custom format.
This situation (not only with JSON-RPC but with essentially every high-level protocol built on top of HTTP) has developed due to various reasons. Some of them are historical (such as the inability to use many HTTP protocol features in early implementations of the XMLHttpRequest
functionality in web browsers). However, new RPC protocols relying on the bare minimum of HTTP capabilities continue to emerge today.
To answer this question, we must emphasize a very important distinction between application-level protocols (such as JSON-RPC in our case) and pure HTTP. A 400 BadRequest
is a transparent status for every intermediary network agent but a JSON-RPC custom error is not. Firstly, only a JSON-RPC-enabled client can read it. Secondly, and more importantly, in JSON-RPC, the request status is not metadata. In pure HTTP, the details of the operation, such as the method, requested URL, execution status, and request / response headers are readable without the necessity to parse the entire body. In most higher-level protocols, including JSON-RPC, this is not the case: even a protocol-enabled client must read a body to retrieve that information.
How does an API developer benefit from the capability of reading request and response metadata? The modern client-server communication stack, as envisioned by Fielding, is multi-layered. We can enumerate a number of intermediary agents that process network requests and responses:
Frameworks that developers use to write code
Programming language APIs that frameworks are built on, and operating system APIs that compilers / interpreters of these languages rely on
Intermediary proxy servers between a client and a server
Various abstractions used in server programming, including server frameworks, programming languages, and operating systems
Web server software that is typically placed in front of backend handlers
Additional modern microservice-oriented tools such as API gateways and proxies
The main advantage that following the letter of the HTTP standard offers is the possibility to rely on intermediary agents, from client frameworks to API gateways, to read the request metadata and perform actions based on it. This includes regulating timeouts and retry policies, logging, proxying, and sharding requests, among other things, without the necessity to write additional code to achieve these functionalities.
If we try to formulate the main principle of designing HTTP APIs, it will be: you would rather design an API in a way that intermediary agents can read and interpret request and response metadata.
Unlike many alternative technologies, which are usually driven by a single large IT company (such as Facebook, Google, or Apache Software Foundation), tools for working with HTTP APIs are developed by many different companies and collaborations. As a result, instead of a single homogeneous but limited toolkit provided by a single vendor, we have a plethora of instruments that work with HTTP APIs. The most notable examples include:
Proxies and gateways (nginx, Envoy, etc.)
Different IDLs (first of all, OpenAPI) and related tools for working with specifications (Redoc, Swagger UI, etc.) and auto-generating code
Programmer-oriented software that allows for convenient development and debugging of API clients (Postman, Insomnia), etc.
Of course, most of these instruments will work with APIs that utilize other paradigms. However, the ability to read HTTP metadata makes it possible to easily design complex pipelines such as exporting nginx access logs to Prometheus and generating response status code monitoring dashboards in Grafana that work out of the box.
Additionally, let's emphasize that the HTTP API paradigm is currently the default choice for public APIs. Because of the aforementioned reasons, partners can integrate an HTTP API without significant obstacles, regardless of their technological stack. Moreover, the prevalence of the technology lowers the entry barrier and the requirements for the qualification of partners' engineers.
The main disadvantage of HTTP APIs is that you have to rely on intermediary agents, from client frameworks to API gateways, to read the request metadata and perform actions based on it without your consent. This includes regulating timeouts and retry policies, logging, proxying, and sharding requests, among other things. Since HTTP-related specifications are complex and the concepts of REST can be challenging to comprehend, and software engineers do not always write perfect code, these intermediary agents (including partners' developers!) will sometimes interpret HTTP metadata incorrectly, especially when dealing with exotic and hard-to-implement standards. Usually, one of the stated reasons for developing new RPC frameworks is the desire to make working with the protocol simple and consistent, thereby reducing the likelihood of errors when writing integration code.
The Question of Performance
When discussing the advantages of alternative technologies such as GraphQL, gRPC, Apache Thrift, etc., the argument of lower performance of JSON-over-HTTP APIs is often presented. Specifically, the following issues with the technology are commonly mentioned:
The verbosity of the JSON format:
Mandatory field names in every object, even for an array of similar entities
The large proportion of technical symbols (quotes, braces, commas, etc.) and the necessity to escape them in string values
The common approach of returning a full resource representation on resource retrieval requests, even if the client only needs a subset of the fields
The lower performance of data serializing and deserializing operations
The need to introduce additional encoding, such as Base64, to handle binary data
Performance quirks of the HTTP protocol itself, particularly the inability to serve multiple simultaneous requests through one connection.
Let's be honest: HTTP APIs do suffer from the listed problems. However, we can confidently say that the impact of these factors is often overestimated. The reason API vendors care little about HTTP API performance is that the actual overhead is not as significant as perceived. Specifically:
Regarding the verbosity of the format, it is important to note that these issues are mainly relevant when compressing algorithms are not utilized. Comparisons have shown that enabling compression algorithms such as gzip largely reduces the difference in sizes between JSON documents and alternative binary formats (and there are compression algorithms specifically designed for processing text data, such as brotli).
If necessary, API designers can customize the list of returned fields in HTTP APIs. It aligns well with both the letter and the spirit of the standard. However, as we already explained to the reader in the “Partial Updates” chapter, trying to minimize traffic by returning only subsets of data is rarely justified in well-designed APIs.
If standard JSON deserializers are used, the overhead compared to binary standards might indeed be significant. However, if this overhead is a real problem, it makes sense to consider alternative JSON serializers such as simdjson. Due to their low-level and highly optimized code, simdjson demonstrates impressive throughput which would be suitable for all APIs except some corner cases.
Generally speaking, the HTTP API paradigm implies that binary data (such as images or video files) is served through separate endpoints. Returning binary data in JSON is only necessary when a separate request for the data is a problem from the performance perspective. These situations are virtually non-existent in server-to-server interactions and/or if HTTP/2 or a higher protocol version is used.
The HTTP/1.1 protocol version is indeed a suboptimal solution if request multiplexing is needed. However, alternate approaches to tackling the problem usually rely on… HTTP/2. Of course, an HTTP API can also be served over HTTP/2.
Let us reiterate once more: JSON-over-HTTP APIs are indeed less performative than binary protocols. Nevertheless, we take the liberty to say that for a well-designed API in common subject areas switching to alternative protocols will generate quite a modest profit.
Advantages and Disadvantages of the JSON Format
It's not hard to notice that most of the claims regarding HTTP API performance are actually not about the HTTP protocol but the JSON format. There is no problem in developing an HTTP API that will utilize any binary format (including, for instance, Protocol Buffers). Then the difference between a Protobuf-over-HTTP API and a gRPC API would be just about using granular URLs, status codes, request/response headers, and the ensuing (in)ability to use integrated software tools out of the box.
However, on many occasions (including this book) developers prefer the textual JSON over binary Protobuf (Flatbuffers, Thrift, Avro, etc.) for a very simple reason: JSON is easy to read. First, it's a text format and doesn't require additional decoding. Second, it's self-descriptive, meaning that property names are included. Unlike Protobuf-encoded messages which are basically impossible to read without a .proto
file, one can make a very good guess what a JSON document is about at a glance. Provided that request metadata in HTTP APIs is readable as well, we ultimately get a communication format that is easy to parse and understand with just our eyes.
Apart from being human-readable, JSON features another important advantage: it is strictly formal meaning it does not contain any constructs that can be interpreted differently in different architectures (with a possible exception of the sizes of numbers and strings), and the deserialization result is aligned very well with native data structures (i.e., indexed and associative arrays) of almost every programming language. From this point of view, we actually had no other choice when selecting a format for code samples in this book.
If you happen to design a less general API for a specific subject area, we still recommend the same approach for choosing a format:
Estimate the overhead of preparing and introducing tools to decipher binary protocols versus the overhead of using not the most optimal data transfer protocols.
Make an assessment of what is more important to you: having a quality but restricted in its capabilities set of bundled software or having the possibility of using a broad range of tools that work with HTTP APIs, even though their quality is not that high.
Evaluate the cost of finding developers proficient with the format.
This is Chapters 36 of “The API” book being written by Sergey Konstantinov. I also have a book on the history of beer and historical beer styles, a Telegram channel on interesting classical music recordings, a travel photo blog on Unsplash, and a website with ranking fantasy & science fiction novels based on awards they received.