I need to programmatically interact with a WebObjects website and extract data from the responses. The particular WebObjects site I am scraping uses component actions and stores sessions in cookies (not urls). This means that all urls look something like this:
http://example.com/WOApp/WebObjects/WOApp.woa/wo/7.0.0.0.29.1.1.1
My first questions are:
Does urls like this not completely destroy local and shared caching opportunities (cachable constraint in REST)? I imaging the only effective caching with such urls is the WebObjects server itself.
Isn't addressability broken as well? Each resource does have a unique endpoint, but it changes constantly. Furthermore (I think) that WebObjects also makes too old URLs invalid since they "time-out" after a period of time. I'm not sure whether this applies only to urls with sessions though.
Regarding the scraping I am not sure whether it's possible to extract any meaningful endpoints from the website. For example, with a normal website I would look through the HTML and extract the POST urls, then use them in my scraper by posting directly to them instead of going through the normal request-response cycle.
In this case I obviously cannot use any URLs extracted from the HTML since they are dynamically generated on each request, but I read something about being able to access WebObjects components directly if the security settings have not been set to disallow this (see https://developer.apple.com/legacy/library/documentation/LegacyTechnologies/WebObjects/WebObjects_3.5/PDF/WebObjectsDevGuide.pdf, p. 53 "Limitations on Direct requests"). I don't understand exactly how to do this though or if it's even possible.
If it's not possible what would be a good approach then? The only options I can think of is:
Using a full-blown browser client to interact with the website (e.g. WatiR or Selenium) and extract & process the HTML from their responses
Manually extracting the dynamic end-points by first request the page where they are on and then find the place in the HTML where they're located. Then use them afterwards as if they were "static".
I am interested in opinions on how to approach this scenario since I don't believe any of the solutions above are particularly good.
You've asked a number of questions, and I'll see if I can cover each in turn.
Does urls like this not completely destroy local and shared caching
opportunities (cachable constraint in REST)? I imaging the only
effective caching with such urls is the WebObjects server itself.
There is, indeed, a page cache within the WebObjects application server, and you're right to observe that these component action URLs probably thwart any other kind of caching. Additionally, even though the session ID is not present in the URL, you'd need the session ID in the cookie to re-create the same page, so having just that URL would get you a session restoration error from the application server.
Isn't addressability broken as well? Each resource does have a unique
endpoint, but it changes constantly.
Well, yes, on the face of it this is true. You've given a component action URL as an example, and they're tied to the session.
Furthermore (I think) that
WebObjects also makes too old URLs invalid since they "time-out" after
a period of time. I'm not sure whether this applies only to urls with
sessions though.
Again, all true. Component action URLs generate sessions, and sessions time out.
At this point, let me take a quick diversion. I'm assuming you're not the owner of the WebObjects application—you're talking about having to scrape a WebObjects app, and you've identified some ways in which this particular app doesn't conform to REST principles. You're completely right—a fully component-action-based WebObjects application won't be RESTful. WebObjects pre-dates REST by a few years. Having said that, there are ways in which a WebObjects application can be completely RESTful:
Using session-less direct actions gives a degree of REST-like behaviour, and would certainly solve the problems you identify with caching, addressability and expiry.
Using the ERRest framework to create a 100% RESTful application.
Of course, none of this will help you if you're just trying to scrape a legacy application.
Regarding the scraping I am not sure whether it's possible to extract
any meaningful endpoints from the website. For example, with a normal
website I would look through the HTML and extract the POST urls, then
use them in my scraper by posting directly to them instead of going
through the normal request-response cycle.
Again, if it's a fully component action-based application, you're right—all those URLs will be dynamically generated and useless to you.
In this case I obviously cannot use any URLs extracted from the HTML
since they are dynamically generated on each request, but I read
something about being able to access WebObjects components directly if
the security settings have not been set to disallow this…
That's talking about getting a component to render directly from its template with some restrictions:
As you note, the application can easily prevent it from happening at all.
As mentioned on p.53, the user input and action-invocation phases of rendering the component are skipped, which probably means this approach would be limited to rendering a component that didn't have any dynamic content anyway. This might be of some very limited use to you, though you'd need to know the component names you were interested in, and they wouldn't normally be exposed anywhere.
I'm not sure you're going to find anything better than the types of high-level functional approaches you've already suggested above, such as automating at the browser level with Selenium. If what you need is REST-style direct addressability of resources within the application, you're not going to get that unless you can re-write the application to use direct actions or ERRest where you need them.
A little late, but could help.
I use the Apache's mod_ext_filter (little modified) to pre/post filter the requests/responses from our WebObjects application. The filter calls PHP scripts and can read the dynamical hyperrefs and other things from the HTML pages. The scripts can also modify the HTTP requests, so we can programatically add/remove parameters from the request to implement new workflows in front of the legacy app and cleanup the requests before they will reach WebObjects. It is also possible to handle an additional database within the scripts and store some things over multiple requests.
So you can get the dynamically created links (maybe a button's name or HTML form destination) and can recognize these names within the request.
It is also possible to "remote control" such applications with little scripts like "click on the third button on the page". The only thing you need is a DOM parser to get the structure of the HTML pages and then rebuild the actions which the browser would do (i.e. create the HTTP request manually and send it as POST to the extracted form destination href). The only problem is the Javascript code, which we analyze and reprogram within PHP (i.e. enable/disable input elements, so they will not be transmitted within the requests)
There were some problems within the WebObjects Adapter Module for Apache. It still uses Content-Length within the HTTP header, which you cannot change in mod_ext_filter. If you change the HTML or the parameters within the request, the length of the content will not longer match. But it is possible to change that.
Theoretically it could also be possible to control such an closed-source legacy application from a new UI on a tablet or smartphone, which delegates the user interaction to the backend WebObjects app.
The scripts depends on the page structure, so if your WebObjects app will be changed, you have to correct some things in the scripts (i.e. third button could be now the fourth button).
It should also be possible to add a Restful interface in front of the application and query the data from the legacy app by the filter scripts.
Related
We have one POST API live in production. Now we have a requirement to accept Localization information and proceed with execution accordingly.
e.g. if distanceUnit is "KM" then process all incoming data in Kilometers.
There are three options I could think of to accept localization information.
As a http header i.e. localization: {"distanceUnit": "km"}
As a part of payload itself.
Request parameter.
I like the 1st option as
it doesn't change api contract.
It's easier for other apis to send this info in case they need to be localized in future.
Localization is a part of content negotiation so I don't think it should be part of payload/query parameter.
Any opinions here would be helpful to zero in on 1st or second option.
Thanks.
While accept-language, as indicated by the proposed link Kit posted, may be attempting, this only supports registered languages, maintained by IANA, the standadization gremium of the Web, but not certain generic configuration options out of the box. It may be attempting to default to miles for i.e. Accept-Language: us and use km elsewhere, American scientists may have certain issues with your application then if they want to use km instead of miles. But if this might not be the case, this clearly could be an option you might consider. In regards to custom HTTP headers, I wouldn't recommend using those as the problem with custom HTTP headers in general is that arbitrary generic HTTP clients do not support these which somehow contradicts the idea why one should use a REST architecture.
Let us transfer your problem to the Web domain for a second and see how we usually solve that task there. As REST is basically just a generalized approach to the common way we humans interact with the Web, any concepts used on the Web also apply to a REST architecture. Thus, designing the whole interaction flow as if your application interacts on a typical Web page is just common practice (or at least should be).
On the Web a so called Web form is used to "teach" a Web client (a.k.a. Browser) what data the server expects as input. It not only teaches the client about the respective properties the server either expects or supports for a certain resource but also which HTTP method to use, about the target URI to send the request to and about the media-type to use, which implicitly is often just given as application/x-www-form-urlencoded but may also be multipart/form-data.
The usage of forms and links fall into the HATEOAS constraint where these concpets allow clients to progress through their task, i.e. of buying an item in a Web shop or administrating users in a system, without the need of ever having to consult an external documentation at all. Applications here basically just use the build-in hypermedia capabilities to progress through their tasks. Clients usually follow some kind of predefined processes where the server instructs clients on what they need to do in order to add an item to the shopping cart or on how to add or edit a user while still just operating on a generic HTML document that by itself isn't tailored to the respective task at hands. This approach allows Web clients to basically render all kinds of pages and users to interact with those generic pages. If something in that page representation changes your browser will automatically adept and render the new version on the next request. Hence, the system is able to evolve over time and adapt to changes easily. This is probably one of the core reasons why anyone wants to use a REST architecture basically.
So, back to the topic. On the Web a server would advertise to a client that it supports various localization information with above mentioned forms. A user might be presented a choice or dropdown option where s/he can select the appropriate option. The user usually does not care how this input is transferred to the server or about the internals of the server at all. All s/he cares for is that the data will be available after the request was submitted (in case of adding or updating a resource). This also holds true for application in a REST architecture.
You might see a pattern here. REST and the browsable Web are basically the same thing. The latter though focuses on human interaction while the primer one should allow applications to "surf the Web" and follow allong processes outlined by the server (semi-)automatically. As such it should be clear by now that the same concepts that apply to the browsable Web also apply to REST and applications in that REST architecture.
I like the 1st option as ... it doesn't change api contract
Clients shouldn't bind to a particular API as this creates coupling, which REST tries to avoid at all costs. Instead of directly binding to an API, the Web and as such also REST should use contracts build on hyper media types that define the admissible syntax and semantics of messages exchanged. By abstracting the contract away from the API itself to the media-type a client can support various contracts simultaneously. The generalization of the media-type furthermore allows to i.e. express various different things with the same media type and thus increase the likelihood for reusage and thus a better integration support into application layers.
Supporting various media-types is similar to speaking different languages. By being able to speak various languages you just increase the likelihood that you will be able to communicate with other people (services) out of the box without the need of learning those languages before. A client can tell a server via the Accept header which media-types it is able to "speak( (a.k.a. process) and the server will either respond with either of these or respond with a 406 Not Acceptable. That error response is, as Jim Webber put it, coordination data that at all times tells you whether everything went well or in case of failures gives you feedback on what went wrong.
In order to stay future-proof I therefore would suggest to design the configuration around hypertext enabled media types that support forms, i.e. HTML forms, applicaiton/hal-forms+json or application/ion+json. If in future you need to add further configuration options adding these is just a trivial task. Whether that configuration is exposed as own resource which you just link to, as embedded part within the resource or not return to the client at all is also a choice you have. If the same configuration may be used by multiple resources it would be benefitial to expose it as own resource and then just create a reference from the resource to that configuration but as mentioned these are design decisions you have to make.
If the POST request body is the only place where this is used, and you never have to do GET requests and automatically apply any conversion, my preference would probably go to adding it to the body.
It's nice to have a full document that contains all the information to describe itself, without requiring external out-of-band data to fully interpret its meaning.
You might like to define your schema to always include the unit in relevant parts of the document, for example:
distance: [5, 'km']
or, as you said, do it once at the top of the doc.
I'm surprised to find so little mention of this dilemma online, and it makes me wonder if I'm totally missing something.
Assume I have a singleton resource called Settings. It is created on init/install of my web server, but certain users can modify it via a REST API, lets say /settings is my URI. I have a GET operation to retrieve the settings (as JSON), and a PATCH operation to set one or more of its values.
Now, I would like to let the user reset this resource (or maybe individual properties of it) to default - the default being "whatever value was used on init", before any PATCH calls were done. I can't seem to find any "best practice" approach for this, but here are the ones I have come up with:
Use a DELETE operation on the resource. It is after all idempotent, and its pretty clear (to me). But since the URI will still exist after DELETE, meaning the resource was neither removed nor moved to an inaccessible location, this contradicts the RESTful definition of DELETE.
Use a POST to a dedicated endpoint such as /settings/reset - I really dislike this one because its the most blatantly non-RESTful, as the verb is in the URI
Use the same PATCH operation, passing some stand-in for "default" such as a null value. The issue I have with this one is the outcome of the operation is different from the input (I set a property to null, then I get it and it has a string value)
Create a separate endpoint to GET the defaults, such as /setings/defaults, and then use the response in a PATCH to set to those values. This doesn't seem to contradict REST in any way, but it does require 2 API calls for seemingly one simple operation.
If one of the above is considered the best practice, or if there is one I haven't listed above, I'd love to hear about it.
Edit:
My specific project has some attributes that simplify this question, but I didn't mention them originally because my aim was for this thread to be used as a reference for anyone in the future trying to solve the same problem. I'd like to make sure this discussion is generic enough to be useful to others, but specific enough to also be useful to me. For that, I will append the following.
In my case, I am designing APIs for an existing product. It has a web interface for the average user, but also a REST (ish) API intended to meet the needs of developers who need to automate certain tasks with said product. In this oversimplified example, I might have the product deployed to a test environment on which i run various automated tests that modify the /settings and would like to run a cleanup script that resets /settings back to normal when I'm done.
The product is not SaaS (yet), and the APIs are not public (as in, anyone on the web can access them freely) - so the audience and thus the potential types of "clients" I may encounter is rather small - developers who use my product, that is deployed in their private data center or AWS EC2 machines, and need to write a script in whatever language to automate some task rather than doing it via UI.
What that means is that some technical considerations like caching are relevant. Human user considerations, like how consistent the API design is across various resources, and how easy it is to learn, are also relevant. But "can some 3rd party crawler identify the next actions it can perform from a given state" isn't so relevant (which is why we don't implement HATEOAS, or the OPTIONS method at all)
Let's discuss your mentioned options first:
1: DELETE does not necessarily need to delete or remove the state contained in the resource targeted by the URI. It just requires that the mapping of target URI to the resource is removed, which means that a consecutive request on the same URI should not return the state of the resource further, if no other operation was performed on that URI in the meantime. As you want to reuse the URI pointing to the client's settings resource, this is probably not the correct approch.
2: REST doesn't care about the spelling of the URI as long as it is valid according to RFC3986. There is no such thing as RESTful or RESTless URI. The URI as a whole is a pointer to a resource and a client should refrain from extracting knowledge of it by parsing and interpreting it. Client and server should though make use of link relation names URIs are attached to. This way URIs can be changed anytime and client will remain to be able to interact with the service further. The presented URI however leaves an RPC kind of smell, which an automated client is totally unaware of.
3: PATCH is actually pretty-similar to patching done by code versioning tools. Here a client should precalculate the steps needed to transform a source document to its desired form and contain these instructions into a so called patch document. If this patch document is applied by someone with the state of a document that matches the version used by the patch document, the changes should be applied correctly. In any other cases the outcome is uncertain. While application/json-patch+json is very similar to the philosophy on a patch-document containing separate instructions, application/merge-patch+json has a slightly different take on it by defining default rules (nulling out a property will lead to a removal, including a property will lead to its adding or update and leaving out properties will ignore these properties in the original document)
4: In this sense first retrieving the latest state from a resource and locally updating it/calculating the changes and then send the outcome to the server is probably the best approach of the ones listed. Here you should make use of conditional requests to guarantee that the changes are only applied on the version you recently downloaded and prevent issues by ruling out any intermediary changes done to that resource.
Basically, in a REST architecture the server offers a bunch of choices to a client that based on his task will chose one of the options and issue a request to the attached URI. Usually, the client is taught everything it needs to know by the server via form representations such as HTML forms, HAL forms or ION.
In such an environment settings is, as you mentioned, a valid resource on its own, so is also a default settings resource. So, in order to allow a client to reset his settings it is just a matter of "copying" the content of the default settings resource to the target settings resource. If you want to be WebDAV compliant, which is just an extension of HTTP, you could use the COPY HTTP operation (also see other registered HTTP operations at IANA). For plain HTTP clients though you might need a different approach so that any arbitrary HTTP clients will be able to reset settings to a desired default one.
How a server wants a client to perform that request can be taught via above mentioned form support. A very simplistic approach on the Web would be to send the client a HTML page with the settings pre-filled into the HTML form, maybe also allow the user to tweak his settings to his wishes beforehand, and then click a submit button to send the request to the URI present in the action attribute of the form, which can be any URI the server wants. As HTML only supports POST and GET in forms, on the Web you are restricted to POST.
One might think that just sending a payload containing the URI of the settings resource to reset and optionally the URI to the default settings to a dedicated endpoint via POST is enough and then let it perform its magic to reset the state to the default one. However, this approach does bypass caches and might let them believe that the old state is still valid. Caching in HTTP works as such that the de-facto URI of a resource is used as key and any non-safe operations performed on that URI will lead to an eviction of that stored content so that any consecutive requests would directly go to the server instead of being served by the cache instead. As you send the unsafe POSTrequest to a dedicated resource (or endpoint in terms of RPC) you miss out on the capability to inform the cache about the modification of the actual settings resource.
As REST is just a generalization of the interaction model used on the human Web, it is no miracle that the same concepts used on the Web also apply onto the application domain level. While you can use HTML here as well, JSON-based formats such as application/hal+json or the above mentioned HAL forms or ION formats are probably more popular. In general, the more media-type your service is able to support, the more likely the server will be to server a multitude of clients.
In contrast to the human Web, where images, buttons and further stuff provide an affordance of the respective control to a user, arbitrary clients, especially automated ones, usually don't coop with such affordances good. As such other ways to hint a client on the purpose of a URI or control element need to be provided, such as link relation names. While <<, <, >, >> may be used on a HTML page link to indicate first, previous, next and last elements in a collection, link relation here provide first, prev, next and last as alternatives. Such link relations should of course be either registered with IANA or at least follow the Web linking extension approach. A client looking up the URI on a prev relation will know the purpose of the URI as well as still be able to interact with the server if the URI ever changes. This is in essence also what HATEOAS is all about, using given controls to navigate the application though the state machine offered by the server.
Some general rules of thumb in designing applications for REST architectures are:
Design the interaction as if you'd interact with a Web page on the human Web, or more formally as a state machine or domain application protocol, as Jim Webber termed it, a client can run through
Let servers teach clients on how requests need to look like via support of different form types
APIs shouldn't use typed resources but instead rely on content type negotiation
The more media type your API or client supports the more likely it will be to interact with other peers
Long story short, in summary, a very basic approach is to offer a client a pre-filled form with all the data that makes up the default settings. The target URI of the action property targets the actual resource and thus also informs caches about the modification. This approach is on top also future-proof that clients will be served automatically with the new structure and properties a resource supports.
... so the audience and thus the potential types of "clients" I may encounter is rather small - developers who use my product, that is deployed in their private data center or AWS EC2 machines, and need to write a script in whatever language to automate some task rather than doing it via UI.
REST in the sense of Fielding's architectural style shines when there are a multitude of different clients interacting with your application and when there needs to be support for future evolution inherently integrated into the design. REST just gives you the flexibility to add new features down the road and well-behaved REST clients will just pick them up and continue. If you are either only interacting with a very limited set of clients, especially ones under your control, of if the likelihood of future changes are very small, REST might be overkill and not justify the additional overhead caused by the careful desing and implementation.
... some technical considerations like caching are relevant. Human user considerations, like how consistent the API design is across various resources, and how easy it is to learn, are also relevant. But "can some 3rd party crawler identify the next actions it can perform from a given state" isn't so relevant ...
The term API design already indicates that a more RPC-like approach is desired where certain operations are exposed user can invoke to perform some tasks. This is all fine as long as you don't call it REST API from Fielding's standpoint. The plain truth here is that there are hardly any applications/systems out there that really follow the REST architectural style but there are tons of "bad examples" who misuse the term REST and therefore indicate a wrong picture of the REST architecture, its purpose as well as its benefits and weaknesses. This is to some part a problem caused by people not reading Fielding's thesis (carefully) and partly due to the overall perference towards pragmatism and using/implementing shortcuts to get the job done ASAP.
In regards to the pragmatic take on "REST" it is hard to give an exact answer as everyone seems to understand different things about it. Most of those APIs rely on external documentation anyway, such as Swagger, OpenAPI and what not and here the URI seems to be the thing to give developers clue about the purpose. So a URI ending with .../settings/reset should be clear to most of the developers. Whether the URI has an RPC-smell to it or whether or not to follow the semantics of the respective HTTP operations, i.e. partial PUT or payloads within GET, is your design choice which you should document.
It is okay to use POST
POST serves many useful purposes in HTTP, including the general purpose of “this action isn’t worth standardizing.”
POST /settings HTTP/x.y
Content-Type: text/plain
Please restore the default settings
On the web, you'd be most likely to see this as a result of submitting a form; that form might be embedded within the representation of the /settings resource, or it might live in a separate document (that would depend on considerations like caching). In that setting, the payload of the request might change:
POST /settings HTTP/x.y
Content-Type: application/x-www-form-urlencoded
action=restoreDefaults
On the other hand: if the semantics of this message were worth standardizing (ie: if many resources on the web should be expected to understand "restore defaults" the same way), then you would instead register a definition for a new method token, pushing it through the standardization process and promoting adoption.
So it would be in this definition that we would specify, for instance, that the semantics of the method are idempotent but not safe, and also define any new headers that we might need.
there is a bit in it that conflicts with this idea of using POST to reset "The only thing REST requires of methods is that they be uniformly defined for all resources". If most of my resources are typical CRUD collections, where it is universally accepted that POST will create a new resource of a given type
There's a tension here that you should pay attention to:
The reference application for the REST architectural style is the world wide web.
The only unsafe method supported by HTML forms was POST
The Web was catastrophically successful
One of the ideas that powered this is that the interface was uniform -- a browser doesn't have to know if some identifier refers to a "collection resource" or a "member resource" or a document or an image or whatever. Neither do intermediate components like caches and reverse proxies. Everybody shares the same understanding of the self descriptive messages... even the deliberately vague ones like POST.
If you want a message with more specific semantics than POST, you register a definition for it. This is, for instance, precisely what happened in the case of PATCH -- somebody made the case that defining a new method with additional constraints on the semantics of the payload would allow a richer, more powerful general purpose components.
The same thing could happen with the semantics of CREATE, if someone were clever enough to sit down and make the case (again: how can general purpose components take advantage of the additional constraints on the semantics?)
But until then, those messages should be using POST, and general purpose components should not assume that POST has create semantics, because RFC 7231 doesn't provide those additional constraint.
In learning about REST architecture, I've noticed that Hypermedia seems to be an important part of the uniform interface constraint to become RESTful; however, I'm having a hard time understanding how this concept of hypermedia is to be consumed by a client based off of the definition of hypermedia found across the web.
From what I understand about REST, hypermedia is/are basically the links that are provided to a client in a rest response containing the representation of a resource (so that the client only has to know about the base entry point url to a REST service). The links are essentially there to help the client know what options it has regarding the representation it received (for example, if I request the /children/ resource, I might receive an xml list of children that also contains links to /children/youngest, /children/oldest/, children/create, etc... This is exactly what i have a hard time understanding... Why does the server have to return these "hypermedia" links to the client? Shouldn't the client have already known about those links? The client isn't going to read the links on its own and follow them correctly... Somebody has to write client code (html and/or javascript for example) beforehand. What good do these links do for the client if the client should have already known about them? What am i missing?
Hypermedia is indeed an important part of creating RESTful interfaces. However, I see a full hypermedia client (sometimes referred to as HATEOAS) as just a point on the spectrum of becoming more RESTful. The Richardson Maturity Model describes 3 steps to becoming more RESTful and I've found these as good guidelines when considering how much of a RESTful patten to adopt on particular projects.
IMO, the best example of a full hypermedia client is a web browser. The web browser understands that the html <a href> tag represents a link. The key to a link is that a user can discover functionality as they move through their journey. So, when designing a RESTful API, the same principle can be applied.
Why does the server have to return these "hypermedia" links to the
client?
So, one reason is that clients (if written to expose links) can allow state transitions to be discoverable.
Another benefit to using links is your API becomes responsible for the structure and data of each link. So, the server-side API is generating the links for each request. This means that the server API can make changes (version or structure) to the links without breaking the client.
You can build a RESTful API without links (Richardson Level 2), but then what if you want to change the URL for one of your API requests? You need to either implement a versioning strategy (through the url or headers) or you need to create a new url for the new feature. Through time, this can get out of hand.
Think about how browsing a web-page works. REST is very similar (although you're actually describing a related topic HATEOAS). The intent is that your client will retrieve the initial data, then have some way of deciding whether or not to request the referenced data. In a browser, some references (such as css, javascript, images) are immediately requested, and others, such as hrefs, are simply displayed to the user, and only retrieved if the user chooses to click the link. The same is true of your REST client -- you will decide for yourself, based on your requirements, which references to automatically request, and which to display to a user for decision making.
If you have an app that shows a person, along with their address, then your app would be written to automatically find the address reference in the person data and retrieve it. If your app shows a person and an option to show related data, then you'd show the person along with hyperlinks (or something similar) that allows the user to choose to retrieve and display the address data.
The issue really comes when you think about how the client visually represents the information it is retrieving. For this, the REST style suggests that you should allow the server to return javascript to dynamically parse the new data (for example, the address in my example above). Maybe the returned data also contains css-like references that will help. Maybe it'll contain javascript links to allow dynamic user interaction (which sounds dangerous to me, but what the hell, it works for the web).
Fielding defined the uniform interface constraint in order to decouple the clients from the implementation of the service. This is just the same as defining a new interface in any oo language. You define it to decouple a class from the implementation of its dependency.
To decouple the client from the URI structure you need links in the response, which contain semantic annotation and the URI. So your client can decide which link to follow based on the semantic annotation (e.g. link relation or a term in a related RDF vocab). This way it does not need to know anything about the URI structure.
You need to decouple clients from the implementation details of the services to make them reusable. At the end you can create general clients like a web browsers today. The difference that these clients will understand what they do, so they won't necessary rely on humans.
What good do these links do for the client if the client should have already known about them?
They make state transitions more directly a part of the interface. They tell the client whether these links or state transitions are available to be used (or not if they are not present!) in the current state.
If the client did not take notice of this, it could potentially still try to generate and follow an URL for an option (state transition) that was actually not available. It might be necessary to replicate the logic to determine what is possible in a particular state on both sides to be sure to avoid this.
If an invalid link/state transition is attempted, the server should report back an error.
On the other hand, if the client never allowed generating/following a link not provided by the server, it could have awareness and provide feedback about the problem earlier (normally resulting in a clearer situation or error message), and while the server should handle the invalid request correctly, in case it didn't it would be less of a problem as it is much less likely to happen in the first place.
Something like this.
var discountDetails = new AddDiscountForm()
{
promotionId = 123456,
discountType = "Promotion",
discount = UnitValue.FromFloat(14.00)
};
await journey
.SelectRoot()
.SelectRelation("product-finder", new { barcode = "23409-23097" })
.SelectItem(0)
.SelectRelation("add-discount", discountDetails)
.RunAsync();
In fact I took the idea from my response and built it.
https://github.com/lukepuplett/surfdude-csharp
I am currently re-writing an old web-application and I want it to be RESTful. Now, one important philosophy behind a RESTful app, is that each request to the end-point has to be stateless.
With the application I am aiming for a common codebase for the API as for normal browsing. In other words, I want to avoid special URLs like http://api.domain.tld or http://domain.tld/api. I intend to interpret the HTTP Accept header for this.
One challenge I came up with, are request parameters, that a user browsing the page usually only chooses once. A good example for this is the language. Again, I can use the Accept-Language header to pick an initial language. But what if the user wishes to change this? It would be unusable if the user needed to switch the language after each request.
In my opinion, this is really a request parameter, and should be passed on as such. For example: http://domain.tld/resource?lang=en. So once the user switched the language, I would need to append this parameter to each URL on the page.
In a way, this makes the browsing-session stateful. Are there any "best practices" for this? How would you approach this. One idea I have, is to store these "global" parameters in the session, but add them to each URL nevertheless. If only to make the API easily discoverable.
On a sidenote: I am currently building the web-page using Flask which provides a method url_for to build URLs. I am considering overriding this, so each generated URL will have the parameter. But this is not a Flask specific problem. This is something most RESTful services should consider, so I will tag it neither with python, nor flask!
State Transfer
REST doesn't require each request to be stateless. The requirement is that the server does not have to manage state on behalf of the client. In effect, each request has to carry sufficient state to allow the server to process it.
Your approach, providing the user language is perfectly sensible. Others might prefer to retrieve it from a shared database but this can have some scalability concerns.
Roy Fielding writes
A REST API must not define fixed
resource names or hierarchies (an
obvious coupling of client and
server). Servers must have the freedom
to control their own namespace.
Instead, allow servers to instruct
clients on how to construct
appropriate URIs, such as is done in
HTML forms and URI templates, by
defining those instructions within
media types and link relations.
How do you do this for system-to-system interfaces? Say the client wants to create an order in the server at http://my.server.org How is it supposed to learn that for creating an order it is supposed to use the url http://my.server.org/newOrder and not http://my.server.org/nO or anything else?
For a human interface (i.e. browser), I guess the server would provide some form of link (possibly in a form element) and the text around and in that link would tell a user which of the forms on that page is the correct one for creating an order (as supposed to creating a user or navigating to some search result)
What are the mechanics used for implementing this on the client side? And also: are they actually used or does the majority of people just hardwire the urls into the client?
How do you do this for
system-to-system interfaces? Say the
client wants to create an order in the
server at http://my.server.org How is
it supposed to learn that for creating
an order it is supposed to use the url
http://my.server.org/newOrder and not
http://my.server.org/nO or anything
else?
It doesn't learn. Machine clients, generally, can't "learn". Not yet at least, we're still pre-Skynet. You have to "teach" them.
But what the key is that you don't teach them URLs. You teach them relations.
Consider, in HTML...
<a rel="order" href="http://my.server.org/newOrder"/>
and
<a rel="order" href="http://my.server.org/nO"/>
You'll notice that the rel is the same, "order", but the URL is not.
In a "perfect" world, you system will have a single entry point, say, http://my.server.org/ and from there the client can find all of the rels that it needs to know about.
In practice, many systems have several "well known", and defined entry points from which the client can start, just as an expediency so the client does not alway have to start at the root of the system. These well known entry points have an implied commitment from the provider that these URLs won't be changing any time soon. That they're long lived, and the server will support them very well.
But once passed the entry point, any URL you find likely does not have such a promise behind it. The URL can be a one use only URL. It could be directed to different machines for, say, load balancing. Who knows. But as a consumer of the service, you really don't care what the URL is, you only care about the relation. The relation tells you the detail of the URL to use.
The documentation of your hypermedia API explains how to apply the uniform interface to each of the rels that your client will encounter. The client can't "intuit" that either, it has to be taught.
Basically, by teaching the client how to navigate the relations that it will or MAY find in the payloads it processes is how the client manipulates the hypermedia API. The payloads contain sign posts to show the way, but the server dictates where those sign posts go.
As for how often it is used, in the machine to machine world, likely not very much. Most systems aren't large enough where the URLs change enough to matter, and the clients are so few that changing the clients is not a significant burden. So most just hard code away.
But then, in the end, you just have bad clients. Nothing a REST system can do with a bad client. It can't tell them apart at runtime anyway.
No matter how you publish an API (to be consumed by machines), it is possible to make breaking changes.
When wrapping your API behind a UI (such as HTML forms), you have the freedom to change the URI without breaking the user, but that is because the user is consuming an abstraction you provided. Change the URL schema without changing your form, and you'll still break the client.
A couple ways to avoid breaking machine clients (basically, supporting backward-compatibility):
Build in some sort of URL versioning
Do redirection from old URL schemas to your new schema
We've quite sucessfully approached it the following way: expose a WADL file at the very root URL of the application describing the media types as well as where to find links in it and their semantics. I know this (WADL) is something seen critical by some in the REST community but I always felt intimidated by very URL focus of WADL only. Beyond all the religious debates we liked having a well defined way of documenting representations. There is a way to get around the URL focus of WADL and rather point out where links can be found in the representation and then rather document that. See that blog post (currently down because of maintenance so you might want to look at it in the Google cache) for details on the approach.
This results in only a single URL to be known by the client as he can find out about it accessing the WADL, and from then on just learn about the representation and where to find links, what HTTP method needs what parameters when being invoked and so on.