Question
How should a well-designed application handle 301 "moved permanently" redirects upon http POST to restful api?
Context
Our hosted application offers a restful api
We have a customer extensively who uses our restful api in their own 'on-premise' application [fwiw, installed at dozens of sites-- with no easy way to update]
We are migrating our application to a new data centers and in the process we will switch them (and others) from 'plain text http' to 'ssl-encrypted https'. [The physical location data center of course is irrelevant, but the new data center has more stringent security rules and thus mandates https]
Our front end (haproxy/nginx) will send a 301 'Moved Permanently'
This 301 redirect preserves query parameters but loses post data
I know that "restful api's should be permanent", but, merde arrive (that's shit happens in French)--Empires crumble, Berlin Wall falls, Oracle buys Sun etc.
The Problem
Their application makes HTTP post calls to our restful api. When the front end returns 'http status 301', their application does not 're-post' to the new url, and the update fails.
Questions
How should error handling handle 301's as well as other http statuses? ( pseudocode will suffice)
Should our 'front end' do something different for '301'?
To fulfill your "well-designed" requirement in the sense of "pure RESTfulness", the client should re-send the request to the new URI. A response code of 301 indicate the resource has moved, and cannot be used to fulfill the request, so there's really no fallback position.
If the client attempts to re-post, but loses data, that's a client bug. The "correct" behavior of a client varies by your requirements: it could treat the redirect as a recoverable error case, and repost transparently; it could repost while instructing the user to update the endpoint; or it could fail with an appropriate error message.
I have problem with OpenRouteService API (Direct Routing Request via GET described in here http://wiki.openstreetmap.org/wiki/OpenRouteService#Direct_Routing_Request_.28via_GET.29).
My request is:
http://openls.geog.uni-heidelberg.de/route?start=18.609%2C53.02&end=18.749%2C53.49&via=18.01%2C53.12&lang=pl&distunit=KM&routepref=Pedestrian&&weighting=Recommended&avoidAreas&useTMC=false&noMotorways=false&noTollways=false&noUnpavedroads=false&noSteps=false&noFerries=false&instructions=false
(from Toruń in Poland to Grudziądz via Bydgoszcz).
Unfortonatly, I get error:
"validation error: Expected element 'EndPoint#http://www.opengis.net/xls' instead of 'viaPoint#http://www.opengis.net/xls' here in element WayPointList#http://www.opengis.net/xls"
If I put antyhing in "via=" this error appear.
When I change "via=" to empty value:
http://openls.geog.uni-heidelberg.de/route?start=18.609%2C53.02&end=18.749%2C53.49&via=&lang=pl&distunit=KM&routepref=Pedestrian&&weighting=Recommended&avoidAreas&useTMC=false&noMotorways=false&noTollways=false&noUnpavedroads=false&noSteps=false&noFerries=false&instructions=false
all works fine.
Is it problem with my request or api isn't working correctly?
It seems that the frontend API of OpenRouteService receives the GET requests correctly, but after it builds the request in xml and relays it to the backend server the server fails to validate the request correctly. It should be a problem with the backend server.
The frontend php code is here for reference.
The web frontend, which POSTs the xml directly to another backend server, seems to work correctly with via points.
I don't make this very general question post lightly, but I've maxed out what I can figure out on my own.
I'm starting up the QA Test Automation at my new company (they don't automate anything currently) and they've elected to use SoapUI for this procedure.
The application they're developing is a REST application (I don't really have any idea what that means) so I'm trying to build a REST request and Test Suite to reach our internal test server (which gets me XML that I am not allowed to post here - but it does reach the server successfully!) and then try to do a Login/Logout test.
I'm asking for help with methodology, because I have no idea where to start. I;ve Googled and trolled their support forums and looked in every corner of YouTube. Everyone is doing something different enough that I can't relate or use it.
Does anybody out there use SoapUI and test functional login in a REST application? I can write HTML/CSS and I'm pretty Java savvy, so I can do technical stuff if I know what to look for and what to learn.
Feeling overwhelmed. This was not in my job description when I started.
You should start with REST, and after that with SoapUI.
It is hard to catch the essence of REST.
It is like the hybrid of SOAP and a simple HTML driven web application. By SOAP you describe your web service with a WSDL. By a web application you send back hypermedia, so you don't have to write a WSDL or any descriptor to your application. This is convention over configuration...
REST uses the same approach, so it sends back hypermedia as well, but it sends not HTML, because it is not machine processable. The hypermedia sent by a REST API is usually an XML or a JSON derivative, for example ATOM+XML, JSON-LD, etc... If your web service does not send back hyperlinks, then it is not a real REST service just a SOAP web service with some REST constraints. There is a big difference. By SOAP you have to know everything about the operation name and the parameters if you want to send a request. If something changes, then your SOAP client breaks immediately. By REST your automated client follows links, checks their link-relation, or the bound linked data and recognizes which link is what it was looking for. So the modification of the link's url is irrelevant in the client, because it follows the vocabulary of the application, for example: hydra is a project which tries to describe these application level semantics in a general way, and tries to bind it to open linked data.
So at first you have to check that you have a real REST API, which follows the HATEOAS principle, or just a REST like SOAP web service. This is very important if you want to write end to end tests against it. By testing REST, you have to follow the links in your tests returned by the web API. By testing REST like SOAP, you have to build the links yourself in your tests... How to build such a link? I am sure you got a description of your REST API, but a link looks usually something like this in a JSON format:
{
rel: "link-relations",
method: "METHOD",
href: "domain/api-root/version/resource-path?map-reduce",
data: {...},
title: "...",
...
}
Ofc. there is some difference by every hypermedia, so you have to check your XML hypermedia type, how it represents links... The link-relations and maybe other attributes bind your data to the semantics of your REST API. The METHOD is always a verb, usually: GET, POST, PUT, PATCH, DELETE, maybe OPTIONS, and so on... There are only a few REST verbs, each of them has a specific meaning. In the url: The domain is the domain name of your application, e.g. https://example.com. The api-root is the root of your REST API, usually /api. The version is the version number of the currently used API, usually /v1. Only non backward compatible vocabulary changes should affect this version number. The resource-path is the path of your resource, usually /users or /users/inf3rno, etc... By REST you have resources. Each of them has a unique resource-path, and as you can see, every word in that path is a noun. So resources are something you can modify or display with a verb. For example a GET /users/inf3rno should return a representation of my profile page, and a PATCH /users/inf3rno {nick: "Leslie"} will turn my nick name: inf3rno into Leslie. By REST every resource should have only a single resource-path, so this is always a unique identifier, therefore the previous example with PATCH was not so perfect if you want to have multiple users with the same nick... The map-reduce in the queryString of the url, and it contains the sorting, pagination and filtering settings of the resource you want to modify or display. For example you can retrieve some data of every user with a first name: "Leslie" with GET /users?filters="firstName: 'Leslie'"&page=3&count=25. There is a difference between the following url-s: /users?id="inf3rno" and /users/inf3rno. The first one points to a collection resource and filters the result by its representation, the second one points to a single item resource. So a GET should return a collection representation with a single item by the first one, and an item representation by the seconds one. By the resource modifying methods there is no difference between the 2 urls... So it is recommended to add only a unique identifier to the resource-path if you want to select an item resource from a collection. By reducing the collection representation in any other ways, you have to add the filters to the queryString. The data part contains the params from the input fields. The title is the title of the link, and so on... You can use url-templates of you want to put input params to the url as well...
By REST the client maintains the session, and it sends the credentials (username, password) with every request. This is because the REST service is like John Snow, it does not know anything about the session or the identity of the user. It has to authenticate every request. To do that it uses a credentials -> permissions cache. This is a good approach, because the service scales very well if it does not have to maintain the session, which is part of the application state (the state of the client)... The REST service maintains only the resource state, which is not dependent on the clients...
The response to your REST requests is usually a hypermedia which contains the links you can follow and the data you requested. By REST like SOAP web services you get only the data in a JSON or XML format. Every response should contain a proper status header. The most frequent status codes are:
200 - ok (by successful PUT, PATCH and GET)
201 - created (by successful POST)
202 - accepted (by async request with eventual consistency)
204 - no content (by successful DELETE)
206 - partial content (by pagination with range headers)
301 - moved permanently (by migration)
304 - not modified (by cache)
400 - bad request (by invalid input)
401 - unauthorized (if no password given, or wrong username or password)
403 - access denied (if your account does not have permission to perform the task)
404 - not found (by unknown resource)
409 - conflict (by concurrency issues or duplicated request or db constraint problems)
410 - gone (if the resource was present before, but it is already deleted)
415 - unsupported media type (if the client wants the response in an unknown media type)
500 - internal server error (if the request was okay, but something went wrong by processing it)
By any error you have to send a detailed error message with a custom error code, which is understandable for the users, not just the developers...
That's how a REST API looks like.
To test it with e2e tests you have to set fixtures send REST requests and check their response. So it is like any other test... The SoapUI is not necessarily the best tool to do that, I read many complaints about it... I personally never used it, but it is not so hard to write your custom testing system. You need a testing framework, which can compare expected and actual values. You need something to send HTTP requests, or simply mock out the HTTP framework of the REST API. You need something for the fixture. By integration tests you can mock out the business logic and the HTTP framework as well, so by those you just inject the mock dependencies and check the calls. By e2e tests you need a test data set, and compare it with the result XML in your case... If you want to e2e test your client, you can use selenium if it is HTML based, maybe with nightwatch.js. By testing a real REST API, you'll need an automated browser, like selenium for your REST API implementation, which can select and follow the proper links. If you are developing the REST API you would write a browser like that anyways if you want an example client for your 3rd party client developers.
There seems to be a lot of confusion about the correct http status code to return if the user tries to access a page which requires the user to login.
So basically what status code will be send when I show the login page?
I'm pretty sure we need to use a status code in the 4xx range.
I'm not talking about HTTP authentication here, so that's at least 1 status code we aren't going to use (401 Unauthorized).
Now what should we use? The answers (also here on SO) seem to vary:
According to the answer here we should use 403 Forbidden.
But in the description of the status code is:
Authorization will not help and the request SHOULD NOT be repeated.
Well that doesn't look like the right one. Since authorization WOULD help.
So let´s check out some other answer. The answer here even doesn't use the 4xx range at all but rather uses 302 Found
The description of the 302 Found status code:
The requested resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. This response is only cacheable if indicated by a Cache-Control or Expires header field.
I think that also isn't what I want. Since it is not the requested resource which resides under a different URI. But rather a completely different resource (login page vs authenticated content page).
So I moved along and picked another answer surprisingly with yet another solution.
This answer suggest we choose 400 Bad Request.
The description of this status code is:
The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.
I think the server understood the request just fine, but just refuses to give access before the user is authenticated.
Another answer also says a 403 response is correct, however it ends with:
If this is a public facing website where you are trying to deny access based on a session cookie [that's what I do], 200 with an appropriate body to indicate that log in is needed or a 302 temporary redirect to a log in page is often best.
So 403 is correct, but 200 or 302 is THE BEST.
Hey! That's what I am looking for: THE BEST solution. But shouldn't the best be the same as the correct one? And why would it be the best?
Thanks to all who have made it this far into this question :)
I know I shouldn't worry too much about it. And I think this question is more hypothetical (not really, but used it because of lack of a better word).
But this question is haunting me for some time now.
And if I would have been a manager (who just picked up some cool sounding words as they always do) I would have said: but, but, but, but restfulness is important. :-)
So: what is the right way™ of using a status code in the above situation (if any)?
tl;dr
What is the correct http status code response when a user tries to access a page which requires login?
If the user has not provided any credentials and your API requires them, return a 401 - Unauthorized. That will challenge the client to do so. There's usually little debate about this particular scenario.
If the user has provided valid credentials but they are insufficient to access the requested resource (perhaps the credentials were for a freemium account but the requested resource is only for your paid users), you have a couple of options given the looseness of some of the HTTP code definitions:
Return 403 - Forbidden. This is more descriptive and is typically understood as, "the supplied credentials were valid but still were not enough to grant access"
Return 401 - Unauthorized. If you're paranoid about security, you might not want to give the extra information back to the client as was returned in (1) above
Return either 401 or 403 but with helpful information in the response body describing the reasons why access is being denied. Again, that information might be more than you would want to provide in case it helps attackers somewhat.
Personally, I've always used #1 for the scenario where valid credentials have been passed but the account they're associated with doesn't have access to the requested resource.
You ask for "the best", "the right way", and "the correct", in turn, which makes answering this question difficult because those criteria are not necessarily interchangeable and may, in fact, conflict -- especially where RESTfulness is concerned.
The "best" answer depends on your application. Are you building a Plain Old Browser-Based (POBB) web-application? Are you building a native client (ex. iOS or Android) and hitting a service over the Web? Are you making heavy use of AJAX to drive web-page updates? Is curl the intended client?
Let's assume you are building a traditional web application. Let's look at how Google does it (output chopped for brevity):
$ curl -v http://gmail.com/
< HTTP/1.1 301 Moved Permanently
< Location: http://mail.google.com/mail/
< Content-Type: text/html; charset=UTF-8
< Content-Length: 225
< ...
Google first redirects us to the "true" URL for GMail (using a 302 redirect).
$ curl -v http://mail.google.com/mail/
< HTTP/1.1 302 Moved Temporarily
< Location: https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=http://mail.google.com/mail/&scc=1<mpl=default<mplcache=2
< Content-Type: text/html; charset=UTF-8
< Content-Length: 352
< ...
And then it redirects us to the login page (using a 302 redirect).
$ curl -v 'https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=http://mail.google.com/mail/&scc=1<mpl=default<mplcache=2'
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< ...
The login page itself is delivered with the 200 status code!
Why this way?
From a user-experience perspective, if a user goes to a page they can't view because they are not authenticated, you want to take the user to a page that allows them to correct this (via logging in). In this example, the login page stands alone and is just another page (which is why 200 is appropriate).
You could throw up a 4XX page with an explanation and a link to the login page. That might, in fact, seem more RESTful. But it's a worse user experience.
Ok, but is there a case where something like 403 makes sense? Absolutely.
First, though, note that 403 isn't well-defined in the specification. In order to understand how it should be used, you need to look at how it's implemented in the field.
403 is commonly used by web servers like Apache and IIS as the status code for pages returned when the browser requests a directory listing (a URI ending in "/") but the server has directory listings disabled. In this case, 403 is really a specialized 404, and there isn't much you can do for the user except let him/her know what went wrong.
However, here's an example of a site that uses the 403 to both signal to the user that he/she doesn't have sufficient privilege and what action to take to correct the situation (check out the full response for details):
curl -v http://www.w3.org/Protocols/rfc2616/
< HTTP/1.1 403 Forbidden
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 1564
< ...
(As an aside, 403 is also seen in web-based APIs, like Twitter's API; here, 403 means "The request is understood, but it has been refused. An accompanying error message will explain why. This code is used when requests are being denied due to update limits.")
As an improvement, let's assume, however, that you don't want to redirect the user to a login page, or force the user to follow a link to the login page. Instead, you want to display the login form on the page that the user is prevented from seeing. If they successfully authenticate, they see the content when the page reloads; if they fail, they get the login form again. They never navigate to another URL.
In this case, a status code of 403 makes a lot of sense, and is homologous to the 401 case, with the caveat that the browser won't pop up a dialog asking the user to authenticate -- the form is in the page itself.
This approach to authentication is not common, but it could make sense, and is IMHO preferable to the pop-up-a-javascript-modal-to-log-in solutions that developers try to implement.
It comes down to the question, do you want to redirect or not?
Additional: thoughts about the 401 status code...
The 401 status code -- and associated basic/digest authentication -- has many things going for it. It's embraced by the HTTP specification, it's supported by every major browser, it's not inherently un-RESTful... The problem is, from a user experience perspective, it's very very unattractive. There's the un-stylable, cryptic pop-up dialog, lack of an elegant solution for logging out, etc. If you (or your stakeholders/clients) can live with those issues (a big if) then it might qualify as the "correct" solution.
Agreed. REST is just a style, not a strict protocol. Many public web services deviate from this style. You can build your service to return whatever you want. Just make sure your clients know how what return codes to expect.
Personally, I have always used 401 (unauthorized) to indicate an unauthenticated user has requested a resource that requires a login. I then require the client application to guide the user to the login.
I use 400 (bad request) in response to a logon attempt with invalid credentials.
HTTP 302 (moved) seems more appropriate for web applications where the client is a browser. Browsers typically follow the re-direct address in the response. This can be useful for guiding the user to a logon page.
I'm not talking about HTTP authentication here, so that's at least 1 status code we aren't going to use (401 Unauthorized).
Wrong. 401 is part of Hypertext Transfer Protocol (RFC 2616 Fielding, et al.), but not limited to HTTP authentication. Furthermore, it's the only status code indicating that the request requires user authentication.
302 & 200 codes could be used and is easier to implement in some scenarios, but not all. And if you want to obey the specs, 401 is the only correct answer there is.
And 403 is indeed the most wrong code to return. As you correctly stated...
Authorization will not help and the request SHOULD NOT be repeated.
So this is clearly not suitable to indicate that authorization is an option.
I would stick to the standard: 401 Unauthorized
-
UPDATE
To add a little more info, lifting the confusion related to...
The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource.
If you think that's going to stop you from using a 401, you have to remember there's more:
"The field value consists of at least one challenge that indicates the authentication scheme(s) and parameters applicable to the Request-URI."
This "indicating the authentication scheme(s)" means you can opt-in for other auth-schemes!
The HTTP protocol (RFC 2616) defines a simple framework for access authentication schemes, but you don't HAVE to use THAT framework.
In other words: you're not bound to the usual WWW-Auth. You only just MUST indicate HOW your webapp does it's authorization and offer the according data in the header, that's all. According to the specs, using a 401, you can choose your own poison of authorization! And that's where your "webapp" can do what YOU want it to do when it comes to the 401 header and your authorization implementation.
Don't let the specs confuse you, thinking you HAVE to use the usual HTTP authentication scheme. You don't! The only thing the specs really enforce: you just HAVE/MUST identify your webapp's authentication scheme and pass on related parameters to enable the requesting party to start potential authorization attempts.
And if you're still unsure, I can put all this into a simple but understandable perspective: let's say you're going to invent a new authorization scheme tomorrow, then the specs allow you to use that too. If the specs would have restricted implementation of such newer authorization technology implementations, those specs would've been modified ages ago. The specs define standards, but they do not really limit the multitude of potential implementations.
Your "TL;DR" doesn't match the "TL" version.
The proper response for requesting a resource that you need authorization to request, is 401.
302 is not the proper response, because, in fact, the resource is not available some place else. The original URL was correct, the client simply didn't have the rights. If you follow the redirect, you do not actually get what you're looking for. You get dropped in to some ad hoc workflow that has nothing to do with the resource.
403 is incorrect. 403 is the "can't get there from here" error. You simply can't see this, I don't care who you are. Some would argue 403 and 404 are similar. The difference is simply with 403, the server is saying "yea, I have it, but you can't", whereas 404 says "I know nothing about what you're talking about." Security wonks would argue that 404 is "safer". Why tell them something they don't need to know.
The problem you are encountering has nothing to do with REST or HTTP. Your problem is trying to set up some stateful relationship between the client and server, manifested in the end via some cookie. The whole resource -> 302 -> Login page is all about user experience using the hack that's known as the Web Browser, which happens to be both, in stock form, a lousy HTTP client and a lousy REST participant.
HTTP has an authorization mechanism. The Authorization header. The user experience around it, in a generic browser, is awful. So no one uses it.
So there is not proper HTTP response (well there is, 401, but don't/can't use that). There is not proper REST response, as REST typically relies on the underlying protocol (HTTP in this case, but we've tackled that already).
So. 302 -> 200 for the login page is all she wrote. That's what you get. If you weren't using the browser, or did everything via XHR or some other custom client, this wouldn't be an issue. You'd just use Authorization header, follow the HTTP protocol, and leverage a scheme like either DIGEST or what AWS uses, and be done. Then you can use the appropriate standards to answer questions like these.
As you point out, 403 Forbidden is explicitly defined with the phrase "Authorization will not help", but it is worth noting that the authors were almost certainly referring here to HTTP authorization (which will indeed not help as your site uses a different authorization scheme). Indeed, given that the status code is a signal to the user agent rather than the user, such a code would be correct insofar as any authorization the agent attempts to provide will not assist any further with the required authorization process (c.f. 401 Unauthorized).
However, if you take that definition of 403 Forbidden literally and feel it is still inappropriate, perhaps 409 Conflict might apply? As defined in RFC 2616 §10.4.10:
The request could not be completed due to a conflict with the current
state of the resource. This code is only allowed in situations where
it is expected that the user might be able to resolve the conflict
and resubmit the request. The response body SHOULD include enough
information for the user to recognize the source of the conflict.
Ideally, the response entity would include enough information for the
user or user agent to fix the problem; however, that might not be
possible and is not required.
There is indeed a conflict with the current state of the resource: the resource is in a "locked" state and such conflict can only be "resolved" through the user providing their credentials and resubmitting the request. The body will include "enough information for the user to recognize the source of the conflict" (it will state that they are not logged-in) and indeed will also include "enough information for the user or user agent to fix the problem" (i.e. a login form).
Your Answer:
401 Unauthorized especially if you do not care or will not be redirecting people to a login page
-or-
302 Found to imply there was the resource but they need to provide credentials to be returned to it. Do this only if you will be using a redirect and make sure to provide appropriate information in the body of the response.
Other Suggestions:
401 Unauthorized is generally used for resources the user does not have access to after handling authentication.
403 Forbidden is a little obscure to me in honesty. I use it when I lock down resources from the file system level, and like your post said, "authorization does not help".
400 Bad Request is inappropriate as needing to login does not represent malformed syntax.
I believe 401 is the correct status code to return from failed authorization. Reference RFC 2616 section-14.8
It reads "A user agent that wishes to authenticate itself with a server-- usually, but not necessarily, after receiving a 401 response"