When should I choose one over the other?
The way I see it API wrappers are so much simpler to use but I feel like there's something I'm not seeing, so can you enlighten me?
REST is a software design to decouple clients from APIs though it is often misunderstood as simple URI design thingy due to the fact that it is often based on HTTP.
Advantages of using APIs and clients that support REST is clearly, that clients are not coupled to any API in particular and are therefore tolerant to changes done on the server side like moving resources to different endpoints. Like a browser is able to present content of a sheer infinite number of web pages, true RESTful clients should behave identical and be able to communicate with any API that supports REST. It may learn on the fly how to deal with new content type by looking up some processing routines dynamically (similar to plugins of certain applications) or fallback to a default handling (reporting errors or presenting unknown data as plain text).
An API wrapper is often used to create clients that are limited to a certain API only. This simplifies development as the client can contain certain logic needed for interaction with the API like en/de-coding messages sent to or received from the service, list all available operations and similar stuff. Often URI endpoints are also either injected through properties or hardcoded into the application. Also, the content type is often limited to XML or JSON and the rules on how to treat responses are hardcoded into the client directly. All these steps however tightly couple the client to the API. In case the API changes (or is enriched by further endpoints) the API wrapper has to be updated and shipped to each consumer otherwise users either won't be able to use the API or make use of the latest features.
API wrapper are often tailor made for the usage of the API and are also often much simpler to implement. They, however, also require constant updating in cases the API itself is changing as the wrapper is incapable of handling these changes itself. REST clients on the other hand are far more complicate to develop as the client somehow has to know (or learn) the semantical meaning of a certain response and has to infer somehow how to act upon received responses. Some parts of it are yet an active field of research (at least in automated processing).
As you asked on when to use which: In cases where you have to create an all-purpose client, a REST client is for sure the right thing to do. However, identifying the correct semantical treatment will be the true chanllenge for these clients IMO. In cases where you only need to provide a client frontend for customers (or users) and not much change is expected on the API itself, a wrapper may be much easier to implement. However, please don't call such a client RESTful!
Related
We are building set of new REST APIs.
Let's say we have a resource /users with the following fields:
{
id: 1
email: "test#user.com"
}
Clients implement this API and can then update this resource by sending a new resource representation to PUT /users/1.
Now let's say we add a new property name to the model like so:
{
id: 1
email: "test#user.com"
name: "test user"
}
If the models the existing clients are using are to call our API not updated, then calls to PUT /users/1 will remove the new name property since PUT is supposed to replace the resource. I know that the clients could work straight with the raw json to ensure they always receive any new properties that are added in the API, but that is a lot of extra work, and under normal circumstances clients are going to create their own model representations of the API resources on their side. This means that any time any new property is added, all clients need to update the code/models on their side to make sure they aren't accidentally removing properties. This creates unneeded coupling between systems.
As a way to solve this problem, we are considering not implementing PUT operations at all and switching updates to PATCH where properties that aren't passed in are simply not changed. That seems technically correct, but might not be in the spirit of REST. I am also slightly concerned about client support for the PATCH verb.
How are others solving this problem? Was is the best practice here?
You are in a situation where you need some form of API versioning. The most appropriate way is probably using a new media-type every time you make a change.
This way you can support older versions and a PUT would be perfectly legal.
If you don't want this and just stick to PATCH, PATCH is supported everywhere except if you use ancient browsers. Not something to worry about.
Switching from PUT to PATCH will not fix your problem, IMO. The root cause, IMO, is that clients already consider the data being returned for a representation to follow a certain type. According to Fielding
A REST API should never have “typed” resources that are significant to the client.
(Source)
Instead of using typed resources clients should use content-type negotiation to exchange data. Here, media-type formats that are generic enough to gain widespread adoption are for sure beneficial, certain domains may however require a more specific representation format.
Think of a car-vendor Web page where you can retrieve the data from your preferred car. You, as a human, can easily identify that the data depicts a typical car. However, the media-type you most likely received the data in (HTML) does not state by its syntax or the semantics of its elements that the data describes a car, unless some semantic annotation attributes or elements are present, though you might be able to update the data or use the data elsewhere.
This is possible as HTML ships with a rich specification of its elements and attributes, such as Web forms that not only describe the supported or expected input parameters but also the URI where to send the data to, the representation format to use upon sending (implicitly given by application/x-www-form-urlencoded; may be overwritten by the enctype attribute though) or the HTTP method to use, which is fixed to either GET or POST in HTML. Through this, a server is able to teach a client on how a request needs to be built. As a consequence the client does not need to know anything else besides having to understand the HTTP, URI and HTML specifications.
As Web pages are usually filled with all kinds of unrelated stuff, such as adds, styling information or scripts, and the XML(-like) syntax, which is not every ones favourite, as it may increase the size of the actual payload slightly, most so-called "REST" APIs do want to exchange JSON-based documents. While plain JSON is not an ideal representation format, as it does not ship with link-support at all, it is though very popular. Certain additions such as JSON Hyper-Schema (application/schema+json hyper-schema) or JSON Hypertext Application-Language (HAL) (application/hal+json) add support for links and link-relation. These can be used to render data received from the server as-is. However, if you want a response to automatically drive your application state (i.e. to dynamically draw the GUI with the processed data) a more specific representation format is needed, that can be parsed by your client and act accordingly as it understands what the server wants it to do with it (= affordance). If you like to instruct a client on how to build a request support for other media-types such as hal-forms or ion need to be supported. Certain media-types furthermore allow you to use a concept called profiles, that allow you to annotate a resource with a semantic type. HAL JSON i.e. does support something like that where the Content-Type header may now contain a value such as application/hal+json;profile=http://schema.org/Car that hints the media-type processor that the payload follows the definition of the given profile and may thus apply further validity checks.
As the representation format should be generic enough to gain widespread usage, and URIs itself shouldn't hint a client as well what kind of data to expect, an other mechanism needs to be used. Link relation names are basically an annotation for URIs that tell a client about the purpose of a certain link. A pageable collection might return links annotated with first, prev, next and last which are pretty obvious what they do. Other links might be hinted with prefetch, that hint a client that a resource can be loaded right after loading the current resource finished as it is very likely that the client will retrieve this resource next. Such media-types, however, should be either standardized (defined in a proposal or RFC and registerd with IANA) or follow the schema proposed by Web linking, (i.e. as used by Dublin Core). A client that just uses the URI for an invoked link-relation name will still work in case the server changes its URI scheme instead of attempting to parse some parameters from the URI itself.
In regards to de/coupling in a distributed system a certain amount of coupling has to exist otherwise parties wont be able to communicate at all. Though the point here is, the coupling should be based on well-defined and standardized formats that plenty of clients may support instead of exchanging specific representation formats only a very limited number of clients support (in worst case only the own client). Instead of directly coupling to the API and using an undefined JSON-based syntax (maybe with external documentation of the semantics of the respective fields) the coupling should now occur on the media-types parties can use to exchange the format. Here, not the question of which media-type to support should be asked but how many you want to support. The more media-types your client or server supports, the more likely it is to interact with other peers in the distributed system. On the grand-scheme of things, you want a server to be able to server a plethora of clients while a single client should be able to interact with (in best case) every server without the need for constant adoptions.
So, if you really want to decouple clients from servers, you should take a closer look at how the Web actually works and try to mimic its interaction model onto your application layer. As "Uncle Bob" Robert C. Martin mentioned
An architecture is about intent! (Source)
and the intention behind the REST architecture is the decoupling of clients from servers/services. As such, supporting multiple media-types (or defining your own-one that is generic enough to reach widespread adoption), looking up URIs just via their accompanying link-relation names and relying on content-type negotiation as well as relying only on the provided data may help you to achieve the degree of decoupling you are looking for.
All nice and well in theory, but so far every rest api I encountered in my career had predefined contracts that changed over time.
The problem here is, that almost all of those so called "REST APIs" are RPC services at its heart which should not be termed "REST" to start with - this is though a community issue. Usually such APIs ship with external documentation (i.e. Swagger) that just re-introduce the same problems classical RPC solutions, such as CORBA, RMI or SOAP, suffer from. The documentation may be seen as IDL in that process without the strict need for skeleton classes, though most "frameworks" use some kind of typed data classes that will either ignore the recently introduced field (in best case) or totally blow up on invocation.
One of the problems REST suffers from is, that most people haven't read Fieldings thesis and therefore don't see the big picture REST tries to establish but claim to know what REST is and therefore mix up things and call their services RESTful which lead to a situation where REST != REST. The ones pointing out what a REST architecture is and how one might achieve it are called out as dreamers and unworldly when the ones proclaiming the wrong term (RPC over HTTP = REST) continue to do so adding to the confusion of especially the ones just learning the whole matter.
I admit that developing a true REST architecture is really, really hard as it is just too easy to introduce some form of coupling. Hence, a very careful design needs to be done that needs time and also costs money. Money plenty of companies can't or don't want to spend, especially in a domain where new technologies evolve on a regular basis and the ones responsible for developing such solutions often leave the company before the whole process had finished.
Just saying it shouldn’t be ‘typed’ is not really a viable solution
Well, how often did you need to change your browser as it couldn't interact with a Web page? I don't talk about CSS-stuff or browser-specific CSS or JS stuff. How often needed the Web to change in the last 2-3 decades? Similar to the Web, the REST architecture is intended for long-lasting applications for years to come, that supports natural evolution by design. For simple frontend-2-backend systems it is for sure overkill. It starts to shine especially in cases where there are multiple peers not under your control you can interact with.
My understanding of REST is simply that a resource needs some means of self-describing itself. My understanding is that this isn't specifically tied to any one protocol (i.e. HTTP) and that there are theoretically numerous ways of achieving this. This is based on an answer to a SO question here: SOAP vs REST (differences) (and unlike the terrible answer to this question: Are Relay and Graphql RESTful?)
Since a GraphQL API is self-describing via introspection, doesn't that mean that GraphQL is RESTful by default since a client can use introspection to figure out how to query it?
While GraphQL is often mentioned as the replacement for REST, both tackle different problems actually.
REST, to start with, is not a protocol but just a style, which, if applied correctly and fully, just decouples clients from servers. A server following the REST principals will therefore provide the client with any information needed to take further steps. A client initially starts without any a-priori knowledge and learns on the fly through issuing requests and processing responses. HATEOAS describes the interaction model a REST architectue should be build upon. It thereby states that a link should be used to request new information which drives its internal flow. On utilizing similar representation to Web forms (HTML) a server can teach a client on needed inputs. Through the affordance of the respective elements a client knows, without any need for external documentation, what to do. I.e. It might find a couple of options to chose one or multiple options from, enter or update some freetext or push some buttons. In HTML forms usually trigger a POST request and send the entered data as application/x-www-form-urlenceded to the server though the form element itself may define something different.
While REST is protocol agnostic, meaning it can be build up ontop of many protocols, HTTP is probably the most prominent one. A common sample for a RESTful client is the Web browser we are all to familiar with. It will start by invoking either a bookmarked URI or invoke one entered in the address bar and progress from there on.
HTTP doesn't specify the representation the request or response has to be sent in but leaves that to clients and servers negotiating them. This helps in decoupling as both client and servers can rely on the common interface (HTTP) and only bind strongly onto the known media types used to exchange data in. A peer not being able to process a document in a certain representation (due to the lack of the respective mime type support) will indicate his other peer via a respective HTTP status code that it does not understand, and therefore can't serve, the requested media-type format. The media type, which is just a human readable documentation of the syntax and the semantics of the data payload, is therefore the most important part in a REST architecture. Even Fielding claimed:
A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]
A media type teaches a peer how to parse and interpret the received payload and to actually make sense out of it, though plenty of people still confuse REST for a JSON based HTTP API with over-engineered URIs they put to much effort in to give the URI some kind of logical sense when actually neither client nor server will interpret it anyway as they will probably use the link relation name given for the URI.
GraphQL on the other hand is a basically just a query language which gives the client the power to request specific fields and elements it wants to retrieve from the server. It is, loosely speaking, some kind of SQL for the Web, or as Fielding termed it just a Remote Data Access (RDA). It therefore has to have some knowledge of the available data beforehand which couples clients somehow to the server. If the server will rename some of the fields, the client might not be able to retrieve that kind of information further, though I'm not a GraphQL expert.
As stated above, REST is often confused for a JSON based HTTP API that allows to perform queries on directly mapped DB entries/entities. Keep in mind that REST doesn't prohibit this, though its focus is on the decoupling of peers not the retrieval aspect of some Web exposed database entries. As Jim Webber pointed out in a great talk back in 2011 in REST you don't simply expose database tables, you create a domain application protocol which clients will follow along like in a text-based computer game or in a typical Webshop system on the internet.
Especially the linked introspection documentation of GraphQL reminds me of reflection in Java, which couples to the actual class model available. If something along the datamodel changes, how does the GraphQL interaction behave? Is it able to change and adapt? Is a client built for one API able to work with an other API out of the box? All these are basically requirements for a true RESTful client. It basically has to adept to changes in future as the server is free to evolve anytime. It further shouldn't assume certain endpoints returning certain types but use content type negotiation to request a representation it can work upon.
These should give you enough insights to determine for yourself whether GraphQL can be RESTful or not. In my opinion it isn't, but my insights into GraphQL are rather limited, TBH.
Because graphql publishes Metadata about its types, it's entirely plausible (I think) to build a graphql client that could consume any graphql endpoint ...
SOAP did the exact same thing, though it was still an RPC protocol. A client could look up the ...?wsdl information at run-time and then generate a request according to the schema defined in the WSDL dynamically, though what usually happened was that some pre-generated stub-classes were generated based on the WSDL data that got compiled into a specific client. A client dynamically generating a request still needed a routine that defines what message-type to create and what data the message required as input.
While SOAP could potentially define multiple endpoints within a WSDL, in most cases only one was defined though. This endpoint usually only operates on POST requests even when later on (SOAP 1.2) GET would have been possible also.
According to Fielding's thesis
REST uses a resource identifier to identify the particular resource involved in an interaction between components.
, what would be the resource identifier in GraphQL? GraphQL's documentation states that
... In contrast, GraphQL's conceptual model is an entity graph. As a result, entities in GraphQL are not identified by URLs. Instead, a GraphQL server operates on a single URL/endpoint, usually /graphql, and all GraphQL requests for a given service should be directed at this endpoint.
Similar to SOAP, all the request are targeted towards a single endpoint. This has some impact if you consider caching, which is a further constraint REST implies. How are responses cacheable if the URI is the key used to store the response in the cache?
While all of the aggregation stuff and the flexibility may be nice from a consumer perspective, they are, probably, not in line with the constraints of REST, though Fielding himself claimed that REST is not applicable in all situations and that designers should select a style that fits their needs as not every style is the "silver bullet" to each problem. Even Mike Amundsen stated that GraphQL violates at least 3 constraints imposed by the REST architecture, even though GraphQL seems to have changed the default retrieval method from POST to GET since.
Usually, if you aim for long-living APIs that should be free to evolve in future and that has to deal with lots of clients, especially ones not under your direct control, this is when REST starts to shine. Fielding admits that most developers have problems when thinking long-term. For a single frontend-to-backend system or for a tailor-made client interacting with the own API, REST is not the architecture one should probably follow.
Last but not least, in a later tweet Fielding stated
There is no such thing as a REST endpoint. There are resources. A countably infinite set of resources bound only by restrictions on URL length. A client can POST to a REST service to create a resource that is a GraphQL query, and then GET that resource with all benefits of REST…
which I interpret as, don't focus to much on justifying whether GraphQL is REST or not, but think about how you can integrate its benefits into the overall design.
My understanding of SOAP vs REST:
REST = JSON, simple consistent interface, gives you CRUD access to 'entities' (Abstractions of things which are not necessarily single DB rows), simpler protocol, no formally enforced 'contract' (e.g. the values an endpoint returns could change, though it shouldn't)
SOAP = XML, more complex interface, gives you access to 'services' (specific operations you can apply to entities, rather than allowing you to CRUD entities directly), formally enforced, pre-stated 'contract' (like a WSDL, where e.g. the return types are predefined and formalized)
Is that a broadly correct assessment?
What about a mixture?
If so, what do I call an API that is a mixture?
For example, If we have what at surface level looks like a REST API (returns JSON, no WSDL or formalized contract defined - but instead of giving you access to the 'entities' that the system manages (User, product, comment, etc) it instead gives you specific access to services and complex operations (/sendUserAnUpdate/1111, /makeCommentTextPurple/3333, /getAllCommentsByUserThisYear/2222) without having full coverage?
The 'services' already exist internally, and the team simply publishes access to them on a request by request basis, through what would otherwise look like a REST API.
Question:
What is the 'mixture' typically referred to as (besides, maybe, a bad API). Is there a word for it? or a concept I can refer to that'll make most developers understand what I'm referring to, without having to say the entire paragraph I did above?
Is it just "JSON SOAP API?", "A Service-based REST API?" - what would you call it?
Thanks!
Thanks!
If you take a look at all those so-called REST-APIs your observation might seem true, though REST actually is something completely different. It describes an architecture or a philosophy whose intent it is to decouple clients from servers, allowing the latter one to evolve in future without breaking clients. It is quite similar to the typical Web page interaction in that a server will teach a client on what it needs and only reacts on client-triggered requests. One has to be pretty careful and pendant when designing REST services as it is too easy to include a coupling that may affect clients when a change is introduced, especially with all the pragmatism around in (commercial) software engineering. Stefan Tilkov gave a great talk on REST back in 2014 that, alongside with Jim Webber or Asbjørn Ulsberg, can be used as introduction lectures to what REST is at its core.
The general premise in REST should always be that a server teaches clients what they need and what a server expects and offers choices to the client via links. If the server expects to receive data from the client it will send a form-esque representation to inform the client about the respective fields it supports and based on the affordance of the respective elements contained in the form a client knows whether to select one or multiple options, enter some free text or enter a date value and such. Unfortunately, most of the media-type formats that attempt to mimic HTML's forms are still in draft versions.
If you take a look at HTML forms in particular you might sense what I'm refering to. Each of the elements that may occur inside a form are well defined to avoid abmiguity and improve interoperability. This is defacto the ultimate goal in REST, having one client that is able to interact with a sheer amount of other services without having to be adapted to each single API explicitely.
The beauty of REST is, it isn't limited to a single representation form, i.e. JSON, in fact there is almost an infinite number of possible representation formats that could be exchanged in a REST environment. Plain application/json is a terrible media-type for REST applications IMO as it doesn't include any defintions in regards to links and forms and doesn't describe the semantics of certain fields that may be shipped in requests and responses. The lack of semantical description usually leads to typed resources where a recipient expects that receiving data from i.e. /api/users returns some specific user data, that may differ from host to host. If you skim through IANA's media type registry you will find a couple of media-type formats you could have used to transfer user-related data and any client supporting these representation formats whold be able to interact with this enpoint without any issues. Fielding himself claimed that
A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). (Source)
Through content-type negotiation client and server will negotiate about a representation format both support and understand. The question therefore shouldn't be which one to support but how many you want to support. The more media-type your API or client is able to exchange payloads for, the more likely it will be to interact with other participants.
Most of those so-called REST APIs are in reality just RPC services exposed via HTTP that may or may not respect and support certain HTTP operations. HTTP thereby is just a transport layer whose domain is the transfer of files or data over the Web. Plenty of people still believe that you shouldn't put verbs in URIs when in reality a script or process usually doesn't (and shouldn't) care whether a URI contains a verb or not. The URI itself is just a pointer a client will follow and invoke when it is interested in receiving the payload. We humans are also not that much interested in the URI itself in regards to the content it may return after invoking that URI. The same holds true for arbitrary clients. It is more important what you ship along with that URI. On the Web a link can be annotated with certain text and/or link relation names that set the links content in relation to the current page. It may hint a client that certain content may be invoked before the whole response was parsed as it is quite likely that the client will also want to know about that. preload i.e. is such a link-relation name that hints the client about that. If certain domain-specific terms exist one might use an extension scheme as defined by Web linking or reuse common knowlege or special microformats.
The whole interaction in a REST environment is similar to playing a text-based computer game or following a certain process flow (i.e. ordering and paying produts) defined by an application domain protocol, that can be designed as a state machine. The client is therefore guided through the whole process. It basically just follows the orders the server gave it, with some choices to break out of the process (i.e. cancel the order before paying).
SOAP on the otherhand is, as you've stated, an XML-based RPC protocol reusing a subset of HTTP to exchange requests and responses. The likelihood that when you change something within your WSDL plenty of clients have to be adapted and recompiled are quite high. SOAP even defines its own security mechanism instead of reusing TLS, which requires explicit support by the clients therefore. As you have a one-to-one communication model due to the state that may be kept in process, scaling SOAP services isn't that easy. In a REST environment this is just a matter of adding a load-balancer before the server and then mirroring the server n-times. The load-balancer can send the request to any of the servers due to the stateless constraint
What is the 'mixture' typically referred to as (besides, maybe, a bad API). Is there a word for it? or a concept I can refer to that'll make most developers understand what I'm referring to, without having to say the entire paragraph I did above?
Is it just "JSON SOAP API?", "A Service-based REST API?" - what would you call it?
The general term for an API that communicates on top of HTTP would be Web API or HTTP API IMO. This article also uses this term. It also lists XML-RPC and JSON-RPC besides SOAP. I do agree with Voice though that you'll receive 5 answers on asking 4 people about the right term to use. While it would be convenient to have a respective term available everyone would agree upon, the reality shows that people are not that interested in a clear separation. Just look here at SO on the questions taged with rest. There is nothing wrong with not being "RESTful", though one should avoid the term REST for truly RPC services. Though I think we are already in a situation where the term REST can't be rescued from misusage and marketing purposes.
For something that requires external documentation to use and that ships with its own custom, non-standardized representation format or that just exposes CRUD for domain objects I'd add -RPC to it, as this is more or less what it is at its heart. So if the API sends JSON and the representation to expect is documented via Swagger or some other external documentationJSON-RPC would probably the most fitting name IMO.
To sum up this post, I hope I could shed some light on what REST truly is and how your observation is flawed by all those pragmatic attempts that unfortunately are RPC through and through. If you change something within their implementation, how many clients will break? In addition to that you can't reuse the client that you've implemented for API A to interact with API B (of a different company or vendor) out of the box and therefore have to either adapt your client or create a new one solely for that API. This is true RPC and therfore should be reflected in the name somehow to hint developers about future expectations. Unfortunately, the process of naming things propperly, especially in regards to REST, seems already lost. There is a fine but tiny group who attempt to spread the true meaning, like Voice, Cassio and some others, though it is like fighting windmills. The best advice here would be to first discuss the naming conventions and what each participant understand on which term and then agree on a naming scheme everyone agrees on to avoid future confusion.
My understanding of SOAP vs REST
...
Is that a broadly correct assessment?
No.
REST is an "architectural style", which is to say a coordinated collection of architectural constraints. The World Wide Web is an example of an application built using the REST architectural style.
SOAP is a transport agnostic message protocol specification, based on XML Information Set
If so, what do I call an API that is a mixture?
I don't think you are going to find an authoritative terminology here. Colloquially, you are likely to hear the broad umbrella term "web api" to describe an HTTP API that isn't "RESTful".
The whole space is rather polluted by semantic diffusion.
We have a lot of different Microservices and we test amongst other things the REST APIs of these different Microservices. These automated REST API tests are not included in the Microservice projects/Repos. Instead we have a testautomation project containing all different tests, like API test and end2end tests, because we want to test everything as a black box.
Now the problem is that there are infinitely combinations of different Microservice versions are possible to test against on a test environment (example: Executing the tests today against Microservice A with version 1.0 and Microservice B with version 2.0 is different to execute the same tests tomorrow against Microservice A with version 1.1 and Microservice B with version 2.1). So, we will need some kind of versioning or tagging our testautomation project or the executed tests, that we are able to identify which combinations of different Microservice versions are valid and which combinations are not valid/working, because e.g. some tests will fail.
Are there any recommendations or experiences to implement and integrate such a versioning/tagging mechanism?
To me the actual problem already lays grounded in your actual design. You state that you maintain some micro-services based on a REST architecture, though in such an environment you don't need to version any endpoint as such to start with. Fielding himself answered how a API in a REST environment should be versioned by simply responding with: Don't.
But why is that? One of the few constraints REST has is HATEOAS (or Hate-Us as I tend to pronounce it) which stands for Hypertext-As-The-Engine-Of-Application-State. This acronym basically just describes the interaction model used in the Web, which will have no preasumption on the content to receive and will only render to the user what it received, including any URIs returned by the server. A browser will trigger a state change upon calling an endpoint targeted by a URI invoked by the user. This might be a link, an image or a form-button (or what not). The core idea here is that the client will be served by the API or server with all the information it needs to take further actions and just present the results to the user.
While browsing the Web page of your preferred manufacturer or vendor you might notice that you'll most likely receive a HTML page containing images, links and further content. The browser itself isn't aware of the product offered on that site though it is sill able to render the result to the user as it knows how to render HTML. If you visit an other page your browser will still be able to render HTML regardless of the content that page offers. Yet if one of these pages is changing in some way your browser will still be able to render the result to you, unless the server responds with a media type that your browser isn't yet aware of, which is though very unlikely on the Web.
What most self-claimed "REST" APIs however return is some arbitrary content specific to a certain API, even though most of these use application/json as representation format. A tailor-made client, that has some knowledge about the API built in, is usually interacting with such an API that however is very unlikely to be able to interact with any other API out there. If something on the API level changes the likelihood of breaking that client without any additional updates of it are therefore high. This is very common to RPC like systems such as SOAP, RMI and CORBA.
Such clients often assume certain endpoints such as /api/users/12345 to return data about a particular user in a most likely JSON representation. The payload is marshalled later on to a object of the underlying programming language probably ignoring any unknown fields and nulling out specified fields that aren't available within the response. Though the fundamental problem here is that clients assume that certain endpoints have a certain types. "Smart" developers will now introduce versioning to the endpoints so that the above mentioned URI will change to /api/v1/users/12345 for a JSON representation containing the old fields while /api/v2/users/12345 will return the new fields. Both versions, however, still describe the same user. Having two different URIs for the same user is already a bad design per se, though usually the versioning of an endpoint does not come alone. Usually the whole API itself is versioned itself so that if you encounter a breaking change you are forced to introduce a whole new API version either copying the other unmodified resources or reusing the same models internally further just exposed under multiple URIs again.
Instead of assuming endpoints to return a certain type with a predefined representation format clients and server should negotiate about the content. HTTP here in particular supports content type negotiation where a client informs a server about its capabilities and the server should respond in a representation format understood by the client. This could be something like application/vnd.acmee-users+json or application/vcard+xml or the like. A client understanding application/vnd.acmee-users.v2+json i.e. might get served by the server the new representation while older clients will still inform the server that they only understand application/vnd.acmee-users+json and get served by such a representation. How the server handles the change internally is not of interest to the client. It is just interested in a representation format it can handle.
Though versioning media-types is also not the preferred way of versioning changes by some architects out there as you fundamentally still describe the same thing just with a bit different syntax or a slightly different semantics. HTML i.e. still ships with application/html (rarely with text/html) but not with application/html_5 or the like. It is designed explicitly in a way to stay backwards compatible. A server generating HTML 5 output will still be rendered on a browser that only supports HTML 4.01 or 2. Maybe not all elements will be rendered the same way as on a HTML 5 compatible browser, but the client won't stop to work unexpectedly.
Mark Nottingham, who is the co-chair in the IETF HTTP working group, stated that the underlying principle of versioning is to not break existing clients. Therefore according to him
This implies that API versioning absolutely cannot be tied to software versioning in any way; doing so will needlessly limit (and often break) your clients, and generally upset people. (Source)
Nottingham even states that the product-token used in User-Agent or Server headers should be taken in preference to any URI or media-type versioning to produce responses specific for certain software versions. With the sheer number of client software out there I'm not the biggest fan of that approach however, as this would require the server to have certain knowledge of the capabilities of HTTP clients and their versions used. For APIs, however, that only have a limited number of clients, probably most of them also under the same control as the API/server, this could be a viable approach though.
As you might see for yourself, in a REST architecture there is no real need to version endpoints itself as client will only handle what they are served with by the API/server. Whether a product-token approach is preferable over a media-type based one might be rather opinionated. The latter one, however, should be based on standardized media-types, registered with IANA. In best case the media-type itself is designed in a way that is backward compatible like HTML, which might avoid introducing new media-types for the same things over and over again.
As Phil Sturgeon mentioned in one of his blog posts
If people are going to design their APIs as RPC with a RESTish facade, they should just commit to being an RPC API and build endpoint for specific clients like they’re literally already doing.
Just be honest about it. Hide the false intention, RPC the lot, document as such, and maybe just use gRPC.
Is there a proof of concept client(i.e. web application) that represents a real-world application implemented using and taking advantage of the RESTful principles?
All I could find are API browsers but the development of a real world application(i.e. a social network or ecommerce website) is quite different.
I've read Roy's work and related papers but I still can't gasp how to make the most of Restful in the client development. I always end-up storing state on the client or specialise the media/type rendering. For example the same resource(i.e. profile resource) is rendered differently based on context(i.e. on the homepage, on the product page or on the dedicated profile page) so farewell media-type -> code on demand rendering.
I really can't see any advantage(in the way I work) of HATEOAS over an API with well defined/auto-generated IDL(i.e. json hyper-schema).
My current conclusion is that only generic clients(i.e. google) can benefit from HATEOS not real-world/specialised applications. The specialised client development doesn't seem to take any benefit if your API is HATEOS-enabled instead of being IDL described.
While it's true that HATEOAS gives you URI flexibility, and human discovery of flows, the real benefit is using it as an encoding of resource state.
If you have a state machine associated with a resource, you will have some states that permit certain state transitions and not others.
The opportunity to effect a possible state transition is offered to REST clients via operations against resource URIs - using HATEAOS hypermedia, you can define the transitions by a known rel link name, and then include or exclude the rel links, depending on which transitions are permitted by the current state.
This means the logic of determining which transitions are valid is kept server side - the client can choose to hide or disable UI options depending on if the associated rel link is present.
Another reason to include or exclude a particular rel link may be related to the access control permissions offered to the current user. Simply exclude them if the current user isn't permitted to carry out the transition.
If you are not dynamically including or excluding rel links based on resource state and/or state of the authorized user, then your analysis of the pros cons is pretty spot on, because you are not using them for the real reason they were included. After all, the S in REST stands for state! :)
HATEOS is a design philosophy / style / flavor and this is largely a matter of taste or a tradeoff between full-blown code gen and a hand-written API.
The key differentiating aspect of HATEOS is the way references are constructed to other resources in the API (namely, by a full URL). This removes a lot of the documentation burden that you might otherwise encounter if the API response only includes an ID (and not the full URL to the resource).
However, when you use HATEOS with JSON instead of XML you lose some of the other context (e.g. should I PUT or GET or POST to this endpoint?) and so you must supplement this with some other kind of metadata if you want to generate a client, or documentation for humans.
In my experience HATEOS APIs are much easier for humans to consume with simple REST clients (e.g. cURL) compared to a WSDL or IDL which assumes the client is using generated code and will never touch the API directly.
Tradeoffs
So why would you choose HATEOS vs WSDL or some other generated option?
The basic assumption for APIs (which is not always true) is that they will have many flavors of clients / consumers, possibly implemented in different languages. This means that over time, writing and updating clients is more work than writing the service.
If you or your business are going to maintain the API clients yourself then there is a cost tradeoff between generating code for all of the clients (WSDL, SWIG, etc.) or hiring a language-specific developer to maintain one.
Chances are a generated API client is not going to follow the idiomatic style for any given language, and the code is generally ugly. If these things matter to you then you will probably want a human to write the client code. If you don't care about this, then you can stop reading about HATEOS and use a WSDL or similar approach instead.
In case you do want to optimize for a human to consume the API, though, HATEOS succeeds because it conveys contextual information to a human, and this makes it easier to write clients without extensive API documentation.
Example
For an example of a HATEOS-like API take a look at the GitHub API. It is quite easy to browse with a REST client and once you learn how to authenticate you can find most of the things you want by following referenced data URLs. You will still need to reference the documentation for specific details and advanced use-cases (like POSTing data) but it is very easy to write a simple client for GitHub without pulling in a GitHub client library or reading the docs end-to-end.