Why document RESTful APIs if they are supposed to use HATEOAS? - rest

In an effort to grasp the underlying purpose of a RESTful API, I've begun delving into HATEOAS.
According to that wikipedia page,
A REST client needs no prior knowledge about how to interact with any particular application or server beyond a generic understanding of hypermedia. By contrast, in a service-oriented architecture (SOA), clients and servers interact through a fixed interface shared through documentation or an interface description language (IDL).
Now, I don't really understand how that's supposed to work, unless there exists prior knowledge of what is available in an API, which decries the stated purpose of HATEOAS. In fact, tools such as Swagger exist for the express purpose of documenting RESTful APIs.
So while I understand that HATEOAS can allow a webservice to indicate the state of a resource, I am missing the link (haha) demonstrating how a client application could possibly figure out what to do with the returned follow-up links in the absence of some sort of "fixed interface".
How is HATEOAS supposed to accomplish this?

You're confusing things. Tools like Swagger don't exist for the express purpose of documenting RESTful APIs. They exist for the express purpose of documenting HTTP APIs that aren't RESTful! You need tools like that for APIs that are not hypertext driven and focus documentation on URI semantics and methods instead of media-types. All those fancy tools that generate lists of URIs and HTTP methods are exactly the opposite of what you are supposed to do in REST. To quote Roy Fielding on this matter:
A REST API should spend almost all of its descriptive effort in
defining the media type(s) used for representing resources and driving
application state, or in defining extended relation names and/or
hypertext-enabled mark-up for existing standard media types. Any
effort spent describing what methods to use on what URIs of interest
should be entirely defined within the scope of the processing rules
for a media type (and, in most cases, already defined by existing
media types). [Failure here implies that out-of-band information is
driving interaction instead of hypertext.]
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
HATEOAS doesn't preclude the need for all documentation. HATEOAS allows you to focus your documentation on your media-types. Your application is supposed to conform to the underlying protocol -- HTTP, most of the time -- and the authoritative documentation on that protocol is all your clients should need for driving the interaction. But they still need to know what they are interacting with.
For instance, when you enter Stack Overflow, you know there are users, questions and answers. You need documentation on what exactly are those things, but you don't need documentation detailing what a click on a link or a button does. Your browser already knows how to drive that, and that's how it works everywhere else.
To quote another answer, REST is the architectural style of the web itself. When you enter Stack Overflow, you know what an User, a Question and an Answer are, you know the media types, and the website provides you with the links to them. A REST API has to do the same. If we designed the web the way people think REST should be done, instead of having a home page with links to Questions and Answers, we'd have a static documentation explaining that in order to view a question, you have to take the URI stackoverflow.com/questions/, replace id with the Question.id and paste that on your browser. That's nonsense, but that's what many people think REST is.

There are various answers to this question depending on the level of autonomy of the client interacting with the service ...
Lets first look at a human driven client - the browser. The browser knows nothing about banks, concerts, cats and whatever else you find on the net - but it certainly knows how to render HTML. HTML is a media type with support for hypermedia (links and forms). In this case you have a perfectly working application with a client that only understands generic hypermedia. The "fixed interface" here is HTML.
Then we have the autonomous clients or "scripted" clients that are supposed to interact with a service without human interaction. This is probably the kind of client you are thinking of when comparing REST to SOA(P). You could find such clients in integration scenarios where two independent computer system exchange data in some predefined way.
Such autonomous clients must certainly agree on something in order to interact with each other. The question is what this "something" is or is not.
In a service oriented architecture the clients agree on specific URLs/endpoints and specific "methods" to invoke on those endpoints (RPC) - this adds coupling on the URL structure used. It also forces the client to know which method to call on what service - the server cannot change URLs and it cannot move a "method" from one service to another without breaking clients.
REST/hypermedia based systems behaves differently. In such a system the client and server agrees on one common entry URL where the client can lookup (GET) a service document, at runtime, describing all the possible interactions with the server using hypermedia controls such as links or forms. These hypermedia controls informs the client about how to interact with the service (the HTTP method and payload encoding) and where to interact with the service. Which in essence means we do not have "a service" anymore but possibly many different services as the client will be told, at runtime, where and how to interact with them.
So how does the client know which hypermedia controls it should look for? It does so by agreeing on a set of identifiers the server will use to identify the relevant controls. For links this is often referred to as "link relation types".
This leads us to what kind of "something" it is that servers and clients agree on - it is 1) a hypermedia enabled media type, 2) the root service index URL, 3) the hypermedia control identifiers and 4) the payload expected for each of the controls. At runtime the client then discovers the remaining URLs, HTTP methods and payload encoding (such as JSON, XML or URL-encoded key/value pairs).
Currently there are a small set of general purpose media types for hypermedia APIs - Mason, HAL, Sirene, Collection JSON, Hydra for JSON-LD and probably a few more.
If your are interested then I have covered this topic in various blog postings:
The role of media types in RESTful web services
Media types for APIs
Selling the benefits of hypermedia in APIs
Hypermedia API documentation example
RESTful resources are not typed

Related

Terminology question: API somewhere between SOAP and REST - what is the name for them?

My understanding of SOAP vs REST:
REST = JSON, simple consistent interface, gives you CRUD access to 'entities' (Abstractions of things which are not necessarily single DB rows), simpler protocol, no formally enforced 'contract' (e.g. the values an endpoint returns could change, though it shouldn't)
SOAP = XML, more complex interface, gives you access to 'services' (specific operations you can apply to entities, rather than allowing you to CRUD entities directly), formally enforced, pre-stated 'contract' (like a WSDL, where e.g. the return types are predefined and formalized)
Is that a broadly correct assessment?
What about a mixture?
If so, what do I call an API that is a mixture?
For example, If we have what at surface level looks like a REST API (returns JSON, no WSDL or formalized contract defined - but instead of giving you access to the 'entities' that the system manages (User, product, comment, etc) it instead gives you specific access to services and complex operations (/sendUserAnUpdate/1111, /makeCommentTextPurple/3333, /getAllCommentsByUserThisYear/2222) without having full coverage?
The 'services' already exist internally, and the team simply publishes access to them on a request by request basis, through what would otherwise look like a REST API.
Question:
What is the 'mixture' typically referred to as (besides, maybe, a bad API). Is there a word for it? or a concept I can refer to that'll make most developers understand what I'm referring to, without having to say the entire paragraph I did above?
Is it just "JSON SOAP API?", "A Service-based REST API?" - what would you call it?
Thanks!
Thanks!
If you take a look at all those so-called REST-APIs your observation might seem true, though REST actually is something completely different. It describes an architecture or a philosophy whose intent it is to decouple clients from servers, allowing the latter one to evolve in future without breaking clients. It is quite similar to the typical Web page interaction in that a server will teach a client on what it needs and only reacts on client-triggered requests. One has to be pretty careful and pendant when designing REST services as it is too easy to include a coupling that may affect clients when a change is introduced, especially with all the pragmatism around in (commercial) software engineering. Stefan Tilkov gave a great talk on REST back in 2014 that, alongside with Jim Webber or Asbjørn Ulsberg, can be used as introduction lectures to what REST is at its core.
The general premise in REST should always be that a server teaches clients what they need and what a server expects and offers choices to the client via links. If the server expects to receive data from the client it will send a form-esque representation to inform the client about the respective fields it supports and based on the affordance of the respective elements contained in the form a client knows whether to select one or multiple options, enter some free text or enter a date value and such. Unfortunately, most of the media-type formats that attempt to mimic HTML's forms are still in draft versions.
If you take a look at HTML forms in particular you might sense what I'm refering to. Each of the elements that may occur inside a form are well defined to avoid abmiguity and improve interoperability. This is defacto the ultimate goal in REST, having one client that is able to interact with a sheer amount of other services without having to be adapted to each single API explicitely.
The beauty of REST is, it isn't limited to a single representation form, i.e. JSON, in fact there is almost an infinite number of possible representation formats that could be exchanged in a REST environment. Plain application/json is a terrible media-type for REST applications IMO as it doesn't include any defintions in regards to links and forms and doesn't describe the semantics of certain fields that may be shipped in requests and responses. The lack of semantical description usually leads to typed resources where a recipient expects that receiving data from i.e. /api/users returns some specific user data, that may differ from host to host. If you skim through IANA's media type registry you will find a couple of media-type formats you could have used to transfer user-related data and any client supporting these representation formats whold be able to interact with this enpoint without any issues. Fielding himself claimed that
A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). (Source)
Through content-type negotiation client and server will negotiate about a representation format both support and understand. The question therefore shouldn't be which one to support but how many you want to support. The more media-type your API or client is able to exchange payloads for, the more likely it will be to interact with other participants.
Most of those so-called REST APIs are in reality just RPC services exposed via HTTP that may or may not respect and support certain HTTP operations. HTTP thereby is just a transport layer whose domain is the transfer of files or data over the Web. Plenty of people still believe that you shouldn't put verbs in URIs when in reality a script or process usually doesn't (and shouldn't) care whether a URI contains a verb or not. The URI itself is just a pointer a client will follow and invoke when it is interested in receiving the payload. We humans are also not that much interested in the URI itself in regards to the content it may return after invoking that URI. The same holds true for arbitrary clients. It is more important what you ship along with that URI. On the Web a link can be annotated with certain text and/or link relation names that set the links content in relation to the current page. It may hint a client that certain content may be invoked before the whole response was parsed as it is quite likely that the client will also want to know about that. preload i.e. is such a link-relation name that hints the client about that. If certain domain-specific terms exist one might use an extension scheme as defined by Web linking or reuse common knowlege or special microformats.
The whole interaction in a REST environment is similar to playing a text-based computer game or following a certain process flow (i.e. ordering and paying produts) defined by an application domain protocol, that can be designed as a state machine. The client is therefore guided through the whole process. It basically just follows the orders the server gave it, with some choices to break out of the process (i.e. cancel the order before paying).
SOAP on the otherhand is, as you've stated, an XML-based RPC protocol reusing a subset of HTTP to exchange requests and responses. The likelihood that when you change something within your WSDL plenty of clients have to be adapted and recompiled are quite high. SOAP even defines its own security mechanism instead of reusing TLS, which requires explicit support by the clients therefore. As you have a one-to-one communication model due to the state that may be kept in process, scaling SOAP services isn't that easy. In a REST environment this is just a matter of adding a load-balancer before the server and then mirroring the server n-times. The load-balancer can send the request to any of the servers due to the stateless constraint
What is the 'mixture' typically referred to as (besides, maybe, a bad API). Is there a word for it? or a concept I can refer to that'll make most developers understand what I'm referring to, without having to say the entire paragraph I did above?
Is it just "JSON SOAP API?", "A Service-based REST API?" - what would you call it?
The general term for an API that communicates on top of HTTP would be Web API or HTTP API IMO. This article also uses this term. It also lists XML-RPC and JSON-RPC besides SOAP. I do agree with Voice though that you'll receive 5 answers on asking 4 people about the right term to use. While it would be convenient to have a respective term available everyone would agree upon, the reality shows that people are not that interested in a clear separation. Just look here at SO on the questions taged with rest. There is nothing wrong with not being "RESTful", though one should avoid the term REST for truly RPC services. Though I think we are already in a situation where the term REST can't be rescued from misusage and marketing purposes.
For something that requires external documentation to use and that ships with its own custom, non-standardized representation format or that just exposes CRUD for domain objects I'd add -RPC to it, as this is more or less what it is at its heart. So if the API sends JSON and the representation to expect is documented via Swagger or some other external documentationJSON-RPC would probably the most fitting name IMO.
To sum up this post, I hope I could shed some light on what REST truly is and how your observation is flawed by all those pragmatic attempts that unfortunately are RPC through and through. If you change something within their implementation, how many clients will break? In addition to that you can't reuse the client that you've implemented for API A to interact with API B (of a different company or vendor) out of the box and therefore have to either adapt your client or create a new one solely for that API. This is true RPC and therfore should be reflected in the name somehow to hint developers about future expectations. Unfortunately, the process of naming things propperly, especially in regards to REST, seems already lost. There is a fine but tiny group who attempt to spread the true meaning, like Voice, Cassio and some others, though it is like fighting windmills. The best advice here would be to first discuss the naming conventions and what each participant understand on which term and then agree on a naming scheme everyone agrees on to avoid future confusion.
My understanding of SOAP vs REST
...
Is that a broadly correct assessment?
No.
REST is an "architectural style", which is to say a coordinated collection of architectural constraints. The World Wide Web is an example of an application built using the REST architectural style.
SOAP is a transport agnostic message protocol specification, based on XML Information Set
If so, what do I call an API that is a mixture?
I don't think you are going to find an authoritative terminology here. Colloquially, you are likely to hear the broad umbrella term "web api" to describe an HTTP API that isn't "RESTful".
The whole space is rather polluted by semantic diffusion.

What is difference between REST and API?

I want to know the main difference between REST and API. Sometimes I see REST API in programming documents, then is REST or API same as REST API? I would like to know more about relation between REST, API and REST API.
REST is a type of API. Not all APIs are REST, but all REST services are APIs.
API is a very broad term. Generally it's how one piece of code talks to another. In web development API often refers to the way in which we retrieve information from an online service. The API documentation will give you a list of URLs, query parameters and other information on how to make a request from the API, and inform you what sort of response will be given for each query.
REST is a set of rules/standards/guidelines for how to build a web API. Since there are many ways to do so, having an agreed upon system of structuring an API saves time in making decisions when building one, and saves time in understanding how to use one.
Other popular API paradigms include SOAP and GraphQL.
Note that the above attempts to answer the question in regards to how the terms are commonly used in web development. Roman Vottner has offered a different answer below which offers good insights into the original definition of the term REST with more technical precision than I have provided here.
REST mostly just refers to using the HTTP protocol the way it was intended. Use the GET HTTP method on a URL to retrieve information, possibly in different formats based on HTTP Accept headers. Use the POST HTTP method to create new items on the server, PUT to edit existing items, DELETE to delete them. Make the API idempotent, i.e. repeating the same query with the same information should yield the same result. Structure your URLs in a hierarchical manner etc.
REST just is a guiding principle how to use URLs and the HTTP protocol to structure an API. It says nothing about return formats, which may just as well be JSON.
That is opposed to, for example, APIs that send binary or XML messages to a designated port, not using differences in HTTP methods or URLs at all.
There is no comparison in REST and API, REST is an API type.
API, in general, is a set of protocols deployed over an application software to communicate with other software components (Like browser interacting with servers) and provide an interface to services which the application software
offers to several live consumers.
And Rest is a form of principle which an API follows in which the server provides information whatever the client desires to interact with services.
REST basically is a style of web architecture that governs the behavior of clients and servers. While API is a more general set of protocols and is deployed over the software to help it interact with some other software.
REST is only geared towards web applications. And mostly deals with HTTP requests and responses. This makes it practically usable by any programming language and easy to test.
API is an acronym for Application Programming Interface and defines a set of structures (i.e. classes) one has to implement in order to interact with a service the API was exposed for. APIs usually expose operations that can be invoked including any required or supported arguments as well as the expected responses. Classical examples here are Corba IDL, SOAP or RMI in the Java ecosystem but also RPC-like usages of Web systems specified in documentation like Swagger or OpenAPI.
REST (REpresentational State Transfer) on the contrary was specified by Fielding in his doctoral thesis where he analyzed how the whole user interactions occurs on the Web. He realized that on the Web only a transport protocol, a naming scheme for stuff as well as a well defined exchanged format is needed to exchange messages or documents. These three parts therefore define the interface to interact with peers in such a ecosystem. The transport layer is covered by HTTP while the naming scheme is defined by URI/IRI. Contrary to traditional RPC protocols which usually only support one syntax, REST is actually independent from a particular syntax. To upkeep interoperability both client and server though need do negotiate about it, which HTTP itself supports through the Accept request and Content-Type response headers. As long as client and server support HTTP, URI/IRI and a set of negotiated representation formats, defined by backing hypermedia capable media-types, they will be able to interact with each other. In a more narrow sense REST therefore has no API other than HTTP, URI/IRI and the respective media types.
However, things are unfortunately not that easy. Most people unfortunately understand something very different in terms of REST or REST API. While URIs should not convey any semantics itself, after all they are just pointers to a resource, plenty of programmers attribute more importance to URIs than they should. Some clients i.e. will attempt to extract some knowledge off of URIs or consider URIs to return responses that represent a certain type. I.e. it may seem natural to consider an URI such as https://api.acme.org/users/1 to return a representation that describes a particular user of that particualar system. An external documentation may specify that a JSON structure is returned that follows a given template such as
{
"id": 1,
"firstName": "Roman",
"lastName": "Vottner",
"role": "Admin",
...
}
can be expected, however, such a thing is closer to RPC than it is to REST. Neither is the response self-descriptive, as required by REST, nor does it follow a representation format that follows a well defined media type that defines the syntax and each of the elements that may form a message. Clients therefore are usually tailor-made for exactly one particular system (or REST API if you will) and can't be used to interact with different systems out of the box without further manual integration/updates. External documentation such as OpenAPI or Swagger are used to describe the available endpoints, the payload-templates that a server will be able to process as well as the expected responses, depending on the input. These documentation therefore is the truth and thus defines the API a client can look up or even use to autogenerate stub classes to interact with the server-side, similar to SOAP.
I therefore don't agree with the answer given by dave. While for RPC systems or the common understood term of REST API his definition may be suitable, for actual REST architectures his explanation isn't fitting at all and thus, IMO at least, not correct either. REST isn't a collection of rules, standards and/or guidelines. It is a set of few constraints that just ensure that peers in such an architecture avoid coupling, support future evolution and become more robust to change.
API is basically a set of functions and procedures which allow one application to access the feature of other application
REST is a set of rules or guidelines to build a web API.
It is basically an architectural style for networked applications on the web which is limited to client-server based applications.
Read more at: https://www.freelancinggig.com/blog/2018/11/02/what-is-the-difference-between-api-and-rest-api/

Can I have a REST element URI without a collection URI?

a basic REST question.. I design a REST API and would like to be able to get a list of book recommendations based on a book id (i.e. client sends book id=w to server and server replies with a list of recommended books, id=x,y,z).
I see two ways to do this:
/recommendation?bookId=thetitle
/recommendation/thetitle
Option 2 seems a bit cleaner to me but I'm not sure if it would be considered good REST design? Because /recommendation/thetitle looks like an element URI, not a collection URI (although in this case it would return a collection). Also, the first part of the resource (/recommendation) would not make any sense by itself.
Thankful for any advice.
URL patterns of this kind have nothing to do with REST. None of the defining properties of REST requires readable URLs.
At the same time, one of the core principles (HATEOAS), if followed properly, allows API clients (applications, not people!) to browse the API and obtain every link required to perform a desired transition of application state or resource state based on a well known message format.
If you feel your API must have readable URLs, it's a good sign that its design probably isn't RESTful at all. This implies the need for a developer to understand the URL structure and hardcode it somewhere in a client application. Something that REST is supposed to avoid by principle.
To quote Roy Fielding's blog post on the subject:
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC’s functional coupling].
Obviously, nothing stops you from actually making URLs meaningful regardless of how RESTful your API actually is. Even if it's for a purpose not dictated by REST itself (viewing the logs left by a client of a properly RESTful API could be easier for a human if they're readable, off the top of my head).
Finally, if you're fine with developing a Web API that's not completely RESTful and you expect developers of clients to read this kind of docs and care about path building, you might actually benefit from comprehensible URLs. This can be very useful in APIs of the so-called levels 0-3, according to Richardson's maturity model.
What's important in terms of REST is how you're leveraging the underlying protocol (HTTP in this case) and what it allows you to do. If we consider your examples from this perspective, /recommendation/thetitle seems preferable. This is because the use of query parameters may prevent responses from being cached by browsers (important if you're writing a JS client) or proxies, making it harder to reuse existing tools and infrastructure.

RESTful API runtime discoverability / HATEOAS client design

For a SaaS startup I'm involved in, I am building both a RESTful web API and a couple of client apps on different platforms that consume it. I think I've got the API figured out, but now I'm turning to the clients. As I've been reading about REST, I see that a key part of REST is discovery, but there seems to be a lot of debate between two different interpretations of what discovery really means:
Developer discovery: The developer hard-codes copious amounts of API details into the client, such as resource URI's, query parameters, supported HTTP methods, and other details that they've discovered through browsing the docs and experimenting with the API's responses. This type of discovery IMHO necessitates cool linkage and the API versioning question, and leads to hard coupling of the client code to the API. Not much better than if using a well-documented collection of RPC's it seems.
Runtime discovery - The client app itself is able to figure out everything it needs with little or no out-of-band information (presumably, only a knowledge of the media types the API deals with.) Links can be hot. But to make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in. There are possibly other difficulties I haven't thought of yet since I haven't gotten to that point in development. But I do like the idea of loose coupling.
Runtime discovery seems to be the holy grail of REST, but I'm seeing precious little discussion about how to implement such a client. Almost all REST sources I've found seem to assume Developer discovery. Anyone know of some Runtime discovery resources? Best practices? Examples or libraries with real code? I'm working in PHP (Zend Framework) for one client. Objective-C (iOS) for the other.
Is Runtime discovery a realistic goal, given the present set of tools and knowledge in the developer community? I can write my client to treat all of the URI's in an opaque manner, but how to do this most efficiently is a question, especially over low-bandwidth connections. Anyway, URI's are only part of the equation. What about link templating in the Runtime context? How about communicating what methods are supported, aside from making a lot of OPTIONS requests?
This is definitely a tough nut to crack. At Google, we've implemented our Discovery Service that all our new APIs are built against. The TL;DR version is we generate a JSON Schema-like spec that our clients can parse - many of them dynamically.
That results means easier SDK upgrades for the developer and easy/better maintenance for us.
By no means the perfect solution, but many of our devs seem to like.
See link for more details (and make sure to watch the vid.)
Fascinating. What you are describing is basically the HATEOAS principle. What is HATEOAS you ask? Read this: http://en.wikipedia.org/wiki/HATEOAS
In layman's terms, HATEOAS means link following. This approach decouples your client from specific URL's and gives you the flexibility to change your API without breaking anyone.
You did your home work and you got to the heart of it: runtime discovery is holy grail. Don't chase it.
UDDI tells a poignant story of runtime discovery: http://en.wikipedia.org/wiki/Universal_Description_Discovery_and_Integration
One of the requirements that should be satisfied before you can call an API 'RESTful' is that it should be possible to write a generic client application on top of that API. With the generic client, a user should be able to access all the API's functionality. A generic client is a client application that does not assume that any resource has a specific structure beyond the structure that is defined by the media type. For example, a web browser is a generic client that knows how to interpret HTML, including HTML forms etc.
Now, suppose we have a HTTP/JSON API for a web shop and we want to build a HTML/CSS/JavaScript client that gives our customers an excellent user experience. Would it be a realistic option to let that client be a generic client application? No. We want to provide a specific look-and-feel for every specific data element and every specific application state. We don't want to include all knowledge about these presentation-specifics in the API, on the contrary, the client should define the look and feel and the API should only carry the data. This implies that the client has hard-coded coupling of specific resource elements to specific layouts and user interactions.
Is this the end of HATEOAS and thus the end of REST? Yes and no.
Yes, because if we hard-code knowledge about the API into the client, we loose the benefit of HATEOAS: server-side changes may break the client.
No, for two reasons:
Being "RESTful" is a property of the API, not of the client. As long as it is possible, in theory, to build a generic client that offers all capabilities of the API, the API can be called RESTful. The fact that clients don't obey the rules, is not the API's fault. The fact that a generic client would have a lousy user experience is not an issue. Why is it important to know that it is possible to have a generic client, if we don't actually have that generic client? This brings me to the second reason:
A RESTful API offers clients the option to choose how generic they want to be, i.e. how resilient to server-side changes they want to be. Clients which need to provide a great user experience may still be resilient to URI changes, to changes in default values and more. Clients doing batch jobs without user interaction may be resilient to other kinds of changes.
If you are interested in practical examples, checkout my JAREST paper. The last section is about HATEOAS. You will see that with JAREST, even highly interactive and visually attractive clients can be quite resilient to server-side changes, though not 100%.
I think the important point about HATEOAS is not that it is some holy grail client-side, but that it isolates the client from URI changes - it is assumed you are using known (or developer discovered custom) Link Relations that will allow the system to know which link for an object is the editable form. The important point is to use a media type that is hypermedia aware (e.g. HTML, XHTML, etc).
You write:
To make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in.
If that link template is supplied in the previous request, then there is no out-of-band information. For example a HTML search form uses link templating (/search?q=%#) to generate a URL (/search?q=hateoas), but nothing is known by the client (the web browser) other than how to use HTML forms and GET.

REST's 'resource communication mechanisms' and 'on-the-fly' improvement of a client's knowledge of them

I'm trying to come to terms with REST, as defined by Roy Fielding. Recently I've been trying to wrap my mind around:
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
The concept I'm interested in is in this quote:
The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand).
Specifically, what is knowledge of "resource communication mechanisms", how is that knowledge described in documentation/specs and realised in an implemntation?
Then, how best to improve that knowledge 'on-the-fly'?
I think I understand addressing 'the client's knowledge of media types'.
I have some guesses (PUT,GET, etc.) but would appreciate any suggestions, examples or pointers to RESTful API's that explicitly adress the issues in that quote. If it helps I'm thinking about these issues in the context of HTTP+JSON, I appreciate REST isn't limited to HTTP+*.
The Sun Cloud API has previously been cited as good RESTful design, I couldn't see where or how it addressed these specific issues - maybe a case of not seeing the wood for the trees?
Clarification:
What puzzles me is if PUT,GET,etc. are these mechanisms, this suggests a client knows which to apply to specific hyperlinks within some <media-type>, and this seems fragile, and might suggest hypertext-links map (directly) to resources.
Resource Communication Mechanisms
By "resource communication mechanisms", I believe Roy is referring to HTTP requests and HTTP verbs. He is just saying it without being specify to HTTP because REST is not dependent on HTTP. I would say that for 99.99% of all REST services, the resource communication mechanism is documented in RFC2616.
The Sun Cloud API meets these requirements because all a client needs to understand to use the API is how to do HTTP requests and the semantics of the returned media types. For example if a client does not understand what is contained in a document of type application/vnd.com.sun.cloud.Cloud+json then it will not be able to use the API.
This is in contrast with services like OData and SData that do not define new media-types, but assume a client knows how to extract domain data out of an Atom feed and expects the client to construct URLs based on a set of rules that define the URI space. This is in direct violation of Roy's recommendations.
Improved on the fly
To be honest, I can only guess at what Roy is alluding to here. I could imagine a scenario where downloaded javascript could be used to construct an url based on user input. This could prevent the server from having to explicitly generate an url for each element in a list.
Also, certain valid transitions could be enabled or disabled on the fly based on user input. Consider the case where you do not want to enable a submit button until the user has entered all the required fields. The retrieved document contains the link to allow the transition, but the downloaded code controls when and if the user can select the link.
Downloaded code could also, be used to dynamically change the verb on a link. If you wish to edit a resource, it could do a GET, if you want to delete that resource, you do a DELETE. This would allow the representation to only contain a single link but be able to perform multiple operations.