Play 2.3 WS withFollowRedirects(true) does not work - scala

The following code should post a form to an endpoint (which returns 302) and, after following the redirect, parse the url of the page and return some information from there.
val start = System.currentTimeMillis()
val requestHolder = WS.url(conf("login.url"))
.withRequestTimeout(loginRequestTimeOut)
.withFollowRedirects(true) //This appears to have no effect...
requestHolder.post(getMap(username, password))
.map(resp =>{
Logger.debug(resp.status.toString)
val loginResponse = getResponse(resp)
val end = System.currentTimeMillis()
Logger.debug("Login for the user: "+username+", request took: " + (end - start) + " milliseconds.")
loginResponse
})
The problem is that .withFollowRedirects(true) appears to have no effect on the query. The status of the response is 302 and the request does not follow the redirect.
I've gone through the process manually using httpie and following the redirects does lead to the correct page.
Any help or insight would be much appreciated.

POST redirection isn't as well supported as GET redirection. W3 specification says:
If the 301 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.
Some browsers don't do that, and just ignore. Have a look also at the 307 status:
307 Temporary Redirect (since HTTP/1.1)
In this case, the request should be repeated with another URI; however, future requests should still use the original URI. In contrast to how 302 was historically implemented, the request method is not allowed to be changed when reissuing the original request. For instance, a POST request should be repeated using another POST request.
There is also a discussion about this on Programmer Stack Exchange.

I've had a lot of trouble with withFollowRedirects and POST.
At some point, while fighting to make things work, I had .withFollowRedirects(false) in my code, then removed it during cleanups & things broke. My current guess is that if this option is not explicitly made false, the default behavior is to follow redirects (302 in my case) with some faulty mechanism. Perhaps the default mechanism uses POST again with same arguments. But in my case, interacting with Google App Script (GAS), one needs to use GET to retrieve JSON output of a POST.
Whatever the mechanism was doing, I was getting 400 with no further diagnostics.
After wasting hours, I realized that .withFollowRedirects(false) was in fact truly needed: it disabled Play's messing with redirects, I was able to see the 302 response & handle the following GET manually with success.

Related

MATLAB Not properly sending HTTP POST requests

I'm currently trying to build a MATLAB based system to interface with the API of my stock broker. I'm however running into quite some issues with sending the http post requests to the server.
I already have it working perfectly when testing with POSTMAN, but for some reason it keeps refusing my MATLAB send requests. I now testing the actual requests through PIPEDREAM which lets me view the http request.
Image of the good and bad requests:
The Left is an image of my postman requests which it perfectly processes as JSON strings. However my MATLAB requests are not processed properly and also are 10 characters longer than the actual string value.
The (trimmed) code to send the requests can be seen here.
% http request classes
import matlab.net.*
import matlab.net.http.*
% prepare payload
username = "usr";
password = "XXXXXXXXXXXXX";
login_payload = struct("username", username, "password", password);
request = RequestMessage('POST', [ bunchOfHeaders ], jsonencode(login_payload));
% Send request to login api
[login_resp, c, h] = request.send("https://trading.somebroker.com/login/secure/login");
Does anyone have any clue what could be happening here? If I set the content-lenght to the "correct" length (same as length(login_payload)) it says my length is wrong even though my postman requests seem to not struggle with this.
Found the answer... Matlabs http stuff is absolutely braindead.
I had a closer look into the raw intercepted messages (pipedream just sends your request back to you and you can view it with string(login_resp)).
For some god darn reason matlab encases the json string with "s which makes the receiver treat the whole body as a string. This is caused by setting the "content-type" to "application/json". changing the content-type to "text/plain" did not encase it in "s and completely solved my issue

What status code should we return if the POST succeeds but does not result in creating anything new?

We have an endpoint which when you post create a new version of resource and returns a 201 and the location of the newly created resource. It determines the new version number based on a comparison of the current version and the version being posted (using a semver like ruleset).
If the version you post is identical to the existing version then no version number is updated. What should we return in this case?
We could just return a 201 even though we have not technically created anything.
I don't want to return a 409 as its not really a conflict, like when you post something with the same id. If you posted the same thing when the existing version was slightly different then you would happily get a 201.
We could just return a 200, but then that would seem weird, and increases the response codes that the users have to deal with
Does the idempotency of the 201 response matter?
Any better suggestions?
How about 303 - See Other? Seems to fit. I draw your attention to this sentence
from the spec at https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
This method exists primarily to allow the output of a POST-activated script to redirect the user agent to a selected resource.
That sounds like what you want to do to me. Here's the rest of it.
10.3.4 303 See Other
The response to the request can be found under a different URI and SHOULD be retrieved using a GET method on that resource. This method exists primarily to allow the output of a POST-activated script to redirect the user agent to a selected resource. The new URI is not a substitute reference for the originally requested resource. The 303 response MUST NOT be cached, but the response to the second (redirected) request might be cacheable.
The different URI SHOULD be given by the Location field in the response. Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s).
Note: Many pre-HTTP/1.1 user agents do not understand the 303
status. When interoperability with such clients is a concern, the
302 status code may be used instead, since most user agents react
to a 302 response as described here for 303
I am a bit puzzled by the other answers as some get it almost right. So, let's clear up things a bit. If all requests are indeed performed with the POST method, in the context of ReSTfulness, they are supposed to modify state on the target server. Otherwise, the meaning of POST is a bit relaxed as you can see in RFC 7231, sec. 4.3.3.
Since the intent of the requests is to create a new version of a resource, they have failed if a version with the given presentation already exists. This would disqualify any 2xx-class response codes. From section 6.3:
The 2xx (Successful) class of status code indicates that the client's request was successfully received, understood, and accepted.
If you absolutely wanted to, you could go for 202/Accepted, which "is intentionally noncommittal." This is a bit of a stretch, though, as this status code is intended for queued processing. I would advise against it.
The 204/No Content code suggested by others is a bit of a poor choice. It were absolutely correct if you POSTed to the resource you were updating, though.
As the result is neither informational (1xx) nor a fault by the server (5xx). Let us have a look at the 3xx class first. From section 6.4:
The 3xx (Redirection) class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request.
One of the most prominent one here would be 304/Not Modified. While sounding like a perfect fit, this code is unfortunately not applicable here. It can only be returned in response to conditional GET or HEAD requests.
302/Found may sound like the next best fit. However, this code is intended for temporary redirects, which is in all likelyhood not what you want.
As has been suggested here, 303/See Other is indeed a good choice:
The 303 (See Other) status code indicates that the server is redirecting the user agent to a different resource [...] which is intended to provide an indirect response to the original request.
[...]
This status code is applicable to any HTTP method. It is primarily used to allow the output of a POST action to redirect the user agent to a selected resource
All other 3xx codes are dealing with various forms of redirects that hardly relate to the situation here.
A final look, 4xx-class of status codes. From RFC 7231, sec. 6.5:
The 4xx (Client Error) class of status code indicates that the client seems to have erred. Except when responding to a HEAD request, the server SHOULD send a representation containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method.
Very few of these are actually deeling with the request body. Two of those who do would stand out here: One is 400/Bad Request, which is by design overly broad. It is - if you will - a catch-all solution. However, this would imply that the request body is malformed (as in syntactically incorrect) in some way, which is probably not the case.
More interesting is 409/Conflict. From the RFC (emphasis mine):
The 409 (Conflict) status code indicates that the request could not be completed due to a conflict with the current state of the target resource.
The wording of the definition places this code close to the PUT method, but is not exclusive. To reiterate the definition of the 4xx codes:
These status codes are applicable to any request method.
422/Unprocessable Entity is a contender, but it implies a semantic error, which really isn't the case here.
Ultimately (drumroll) the final piece of the puzzle could be found in section 4.3.3:
If the result of processing a POST would be equivalent to a representation of an existing resource, an origin server MAY redirect the user agent to that resource by sending a 303 (See Other) response with the existing resource's identifier in the Location field.
Note the "MAY." So you can really choose between 303 and 409. I feel 409 were the better fit, as clearly an earlier request introduced a state that is incompatible with the current one. OTOH, 303 may be the politer way to go and is closer to the standard. Either way, as a consumer of your API, I would really like to know if my requests failed. And be it by not having any effect whatsoever.
If nothing has been created by the operation, 201 is not suitable for that:
6.3.2. 201 Created
The 201 (Created) status code indicates that the request has been fulfilled and has resulted in one or more new resources being created. [...]
See below some options you could consider if the operation succeeds:
6.3.1. 200 OK
The 200 (OK) status code indicates that the request has succeeded. The payload sent in a 200 response depends on the request method. For the methods defined by this specification, the intended meaning of the payload can be summarized as:
[...]
POST: a representation of the status of, or results obtained from, the action;
[...]
Aside from responses to CONNECT, a 200 response always has a payload, though an origin server MAY generate a payload body of zero length. If no payload is desired, an origin server ought to send 204 (No Content) instead. [...]
6.3.5. 204 No Content
The 204 (No Content) status code indicates that the server has successfully fulfilled the request and that there is no additional content to send in the response payload body. [...]
If the operation fails:
6.5.1. 400 Bad Request
The 400 (Bad Request) status code indicates that the server cannot will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).
11.2. 422 Unprocessable Entity
The 422 (Unprocessable Entity) status code means the server understands the content type of the request entity (hence a 415 (Unsupported Media Type) status code is inappropriate), and the syntax of the request entity is correct (thus a 400 (Bad Request) status code is inappropriate) but was unable to process the contained instructions. For example, this error condition may occur if an XML request body contains well-formed (i.e., syntactically correct), but semantically erroneous, XML instructions.
A 201 Created should be used whenever you creating a new resource without doubt.
As defined in HTTP Method Definitions RFC, either200 Ok or 204 No Contentis an appropriate response if the operation does not create a new resource depending on the response body content.
The action performed by the POST method might not result in a resource
that can be identified by a URI. In this case, either 200 (OK) or 204
(No Content) is the appropriate response status, depending on whether
or not the response includes an entity that describes the result.
If a resource has been created on the origin server, the response
SHOULD be 201 (Created) and contain an entity which describes the
status of the request and refers to the new resource, and a Location
header (see section 14.30).
Now, coming back to your original question about what to use when the operation is successful and there is nothing to return, you should use 204 No Content. This status code is specifically meant for scenarios where the requested operation is successfully completed but there is no additional relevant information that the server can provide.
The 204 (No Content) status code indicates that the server has successfully fulfilled the request and that there is no additional content to send in the response payload body.
I don't think that for this case the idempotency is an issue, because the state of the system is not the same as it was in the inicial request, because the entity now exists, so you can respond with a different code.
200 should be fine, but it is a little weird as you said.
I have never use this, but I read that for some case you should redirect with a 302, to make a get for other resource, in this case I think this apply, return a 302 and make a get pointing for the old semver, assuming that you have a get endpoint for this entity.
If the POST'd resource has the same version number, but different data, then a 409 would be fitting. But if the data is the same as what's already stored, then a 409 might not be required. Through process of elimination I would say 200 OK would be fine.
We could just return a 200, but then that would seem weird, and increases the response codes that the users have to deal with
If this is a costly concern, consider eliminating the 201, not the 200. The fact of the matter is that for any decently complex service there may be at some point a situation where a 20X (where X is not 0) is applicable. So does that mean we code with each of the 20X responses in mind and spend time checking if our services has situations where 20X is preferred over 200? I suspect not. So unless there is a specific reason to respond with a 20X, for example to deal with a specific use case, then just use 200 and reduce the amount of coding and documenting required. I suspect for most scenarios, the calling client does not care.
Ultimately, the correct answer probably depends on whatever client is consuming your API. If you are building the client too, you could do whatever you prefer... (Just don't get too crazy.)
Assuming you are writing the API and client:
My opinion/suggestion is...
If there IS a new version number: The 201 HTTP status code would fit will.
If there is NOT a new version number: The 200 or 204 HTTP status code would fit well.
If there is no benefit to the client knowing the version number has changed or is the same: Send the 200 HTTP status code.
If you don't control the client consuming your API: Obviously defer to what they expect.
You may also wish review all of the HTTP Status Codes in the HTTP RFC spec. The above status codes also link directly to the relevant section.
201 : when new version is created
202 : when existing version is updated
204 : when request is accepted but no processing is done
by def, 204 = No Content
The server has fulfilled the request but does not need to return an
entity-body, and might want to return updated metainformation. The
response MAY include new or updated metainformation in the form of
entity-headers, which if present SHOULD be associated with the
requested variant.
If the client is a user agent, it SHOULD NOT change its document view
from that which caused the request to be sent. This response is
primarily intended to allow input for actions to take place without
causing a change to the user agent's active document view, although
any new or updated metainformation SHOULD be applied to the document
currently in the user agent's active view.
The 204 response MUST NOT include a message-body, and thus is always
terminated by the first empty line after the header fields.
So its slight tangential to your needs but I think its the best fit.

Play framework - retrieving the Date header in the request

I need to access the Date: header when I handle the request, but this seems to be "swallowed" by the framework; any other header (even made up FooBar ones) show up and I can get them, but this gives me None (I'm using Postman to send a simple GET request - everything else works just fine):
println("Date: " + request.headers.get("Date").getOrElse("no date!"))
returns "no date!" no matter how I try to send something sensible.
I'm wondering whether this gets processed before the request object reaches my Action.
I need the actual string value sent, as this should be part of the request's signature - so an equivalent Date object representing the same value would not be of much use (as it needs to be part of the hash, to avoid replay attacks).
Just as a test, I replaced the Date header with a Date-Auth one, and this one shows up just fine:
ArrayBuffer((Date-Auth, ArrayBuffer(Wed, 15 Nov 2014 06:25:24 GMT))
Any ideas or suggestions greatly appreciated!
Are you sure there is a Date Header in your request (tested with tools like firebug or wireshark)?
Browsers do not need to send a Date header.
RFC 2616 (HTTP 1.1) from the Date section (14.18)
Clients SHOULD only send a Date header field in messages that include an entity-body, as in the case of the PUT and POST requests, and even then it is optional. A client without a clock MUST NOT send a Date header field in a request.
I stand corrected - it turns out that Chrome blocks a whole bunch of headers:
http://www.getpostman.com/docs/requests
I wrote a Python Flask test server and, in fact, the Date header is not there.
That page has also a fix, which works just fine with Postman Version 0.10.4.3 and Interceptor(1).
sorry for wasting everyone's time!
1 Incidentally, IMO Postman is the best REST client and has now also some awesome looks, beyond incredible functionality. If you're working with REST APIs, I highly recommend it.

Picking HTTP status codes for errors from REST-ful services

When a client invokes my REST-ful service, it needs to know if the response came back was 'from me' or rather a diagnosis from the containing web server that something awful happened.
One theory is that, if my code is called, it should always return an HTTP OK(=200), and any errors I've got to return should be just represented in the data I return. After all, it's my code that gets the response, not the naked browser.
Somewhat self-evidently, if I'm using REST to generate HTML read directly by a browser, I absolutely must return an error code if there's an error. In the case I care about, it's always Javascript or Java that is interpreting the entrails of the response.
Another possibility is that there is some family of HTTP status codes that I could return with a high confidence that it/they would never be generated by a problem in the surrounding container. Is this the case?
I use the following:
GET
200 OK
400 Bad Request (when input criteria not correct)
POST
202 Accepted (returned by authorization method)
401 Unauthorized (also returned by authorization)
201 Created (when creating a new resource; I also set the location header)
400 Bad Request (when data for creating new entity is invalid or transaction rollback)
PUT
Same as POST
201 Ok
400 Bad Request
DELETE
200 OK
404 Not Found (same as GET)
I would not know how to avoid that some container returns codes like 404.
4xx codes are meant to handle client errors along with possibly some entity that describes the problem in detail (and thus would mean a combination of both of your mentioned approaches). Since REST relies on HTTP and the according semantics of status as well as methods, always returning 200 in any possible case is a violation of this principle in my opinion.
If you for instance have a request such as http://foo.com/bar/123 which represents a bar ressource with id=123 and you return 200 with some content, the client has no chance to figure out if this was the intended response or some sort of error that occured. Therefore one should try to map error conditions to status codes as discussed in REST: Mapping application errors to HTTP Status codes for example.

HTTP GET with request body

I'm developing a new RESTful webservice for our application.
When doing a GET on certain entities, clients can request the contents of the entity.
If they want to add some parameters (for example sorting a list) they can add these parameters in the query string.
Alternatively I want people to be able to specify these parameters in the request body.
HTTP/1.1 does not seem to explicitly forbid this. This will allow them to specify more information, might make it easier to specify complex XML requests.
My questions:
Is this a good idea altogether?
Will HTTP clients have issues with using request bodies within a GET request?
https://www.rfc-editor.org/rfc/rfc2616
Roy Fielding's comment about including a body with a GET request.
Yes. In other words, any HTTP request message is allowed to contain a message body, and thus must parse messages with that in mind. Server semantics for GET, however, are restricted such that a body, if any, has no semantic meaning to the request. The requirements on parsing are separate from the requirements on method semantics.
So, yes, you can send a body with GET, and no, it is never useful to do so.
This is part of the layered design of HTTP/1.1 that will become clear again once the spec is partitioned (work in progress).
....Roy
Yes, you can send a request body with GET but it should not have any meaning. If you give it meaning by parsing it on the server and changing your response based on its contents, then you are ignoring this recommendation in the HTTP/1.1 spec, section 4.3:
...if the request method does not include defined semantics for an entity-body, then the message-body SHOULD be ignored when handling the request.
And the description of the GET method in the HTTP/1.1 spec, section 9.3:
The GET method means retrieve whatever information ([...]) is identified by the Request-URI.
which states that the request-body is not part of the identification of the resource in a GET request, only the request URI.
Update
The RFC2616 referenced as "HTTP/1.1 spec" is now obsolete. In 2014 it was replaced by RFCs 7230-7237. Quote "the message-body SHOULD be ignored when handling the request" has been deleted. It's now just "Request message framing is independent of method semantics, even if the method doesn't define any use for a message body" The 2nd quote "The GET method means retrieve whatever information ... is identified by the Request-URI" was deleted. - From a comment
From the HTTP 1.1 2014 Spec:
A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.
While you can do that, insofar as it isn't explicitly precluded by the HTTP specification, I would suggest avoiding it simply because people don't expect things to work that way. There are many phases in an HTTP request chain and while they "mostly" conform to the HTTP spec, the only thing you're assured is that they will behave as traditionally used by web browsers. (I'm thinking of things like transparent proxies, accelerators, A/V toolkits, etc.)
This is the spirit behind the Robustness Principle roughly "be liberal in what you accept, and conservative in what you send", you don't want to push the boundaries of a specification without good reason.
However, if you have a good reason, go for it.
You will likely encounter problems if you ever try to take advantage of caching. Proxies are not going to look in the GET body to see if the parameters have an impact on the response.
Elasticsearch accepts GET requests with a body. It even seems that this is the preferred way: Elasticsearch guide
Some client libraries (like the Ruby driver) can log the cry command to stdout in development mode and it is using this syntax extensively.
Neither restclient nor REST console support this but curl does.
The HTTP specification says in section 4.3
A message-body MUST NOT be included in a request if the specification of the request method (section 5.1.1) does not allow sending an entity-body in requests.
Section 5.1.1 redirects us to section 9.x for the various methods. None of them explicitly prohibit the inclusion of a message body. However...
Section 5.2 says
The exact resource identified by an Internet request is determined by examining both the Request-URI and the Host header field.
and Section 9.3 says
The GET method means retrieve whatever information (in the form of an entity) is identified by the Request-URI.
Which together suggest that when processing a GET request, a server is not required to examine anything other that the Request-URI and Host header field.
In summary, the HTTP spec doesn't prevent you from sending a message-body with GET but there is sufficient ambiguity that it wouldn't surprise me if it was not supported by all servers.
GET, with a body!?
Specification-wise you could, but, it's not a good idea to do so injudiciously, as we shall see.
RFC 7231 §4.3.1 states that a body "has no defined semantics", but that's not to say it is forbidden. If you attach a body to the request and what your server/app makes out of it is up to you. The RFC goes on to state that GET can be "a programmatic view on various database records". Obviously such view is many times tailored by a large number of input parameters, which are not always convenient or even safe to put in the query component of the request-target.
The good: I like the verbiage. It's clear that one read/get a resource without any observable side-effects on the server (the method is "safe"), and, the request can be repeated with the same intended effect regardless of the outcome of the first request (the method is "idempotent").
The bad: An early draft of HTTP/1.1 forbade GET to have a body, and - allegedly - some implementations will even up until today drop the body, ignore the body or reject the message. For example, a dumb HTTP cache may construct a cache key out of the request-target only, being oblivious to the presence or content of a body. An even dumber server could be so ignorant that it treats the body as a new request, which effectively is called "request smuggling" (which is the act of sending "a request to one device without the other device being aware of it" - source).
Due to what I believe is primarily a concern with inoperability amongst implementations, work in progress suggests to categorize a GET body as a "SHOULD NOT", "unless [the request] is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported" (emphasis mine).
The fix: There's a few hacks that can be employed for some of the problems with this approach. For example, body-unaware caches can indirectly become body-aware simply by appending a hash derived from the body to the query component, or disable caching altogether by responding a cache-control: no-cache header from the server.
Alas when it comes to the request chain, one is often not in control of- or even aware, of all present and future HTTP intermediaries and how they will deal with a GET body. That's why this approach must be considered generally unreliable.
But POST, is not idempotent!
POST is an alternative. The POST request usually includes a message body (just for the record, body is not a requirement, see RFC 7230 §3.3.2). The very first use case example from RFC 7231 (§4.3.3) is "providing a block of data [...] to a data-handling process". So just like GET with a body, what happens with the body on the back-end side is up to you.
The good: Perhaps a more common method to apply when one wish to send a request body, for whatever purpose, and so, will likely yield the least amount of noise from your team members (some may still falsely believe that POST must create a resource).
Also, what we often pass parameters to is a search function operating upon constantly evolving data, and a POST response is only cacheable if explicit freshness information is provided in the response.
The bad: POST requests are not defined as idempotent, leading to request retry hesitancy. For example, on page reload, browsers are unwilling to resubmit an HTML form without prompting the user with a nonreadable cryptic message.
The fix: Well, just because POST is not defined to be idempotent doesn't mean it mustn't be. Indeed, RFC 7230 §6.3.1 writes: "a user agent that knows (through design or configuration) that a POST request to a given resource is safe can repeat that request automatically". So, unless your client is an HTML form, this is probably not a real problem.
QUERY is the holy grail
There's a proposal for a new method QUERY which does define semantics for a message body and defines the method as idempotent. See this.
Edit: As a side-note, I stumbled into this StackOverflow question after having discovered a codebase where they solely used PUT requests for server-side search functions. This were their idea to include a body with parameters and also be idempotent. Alas the problem with PUT is that the request body has very precise semantics. Specifically, the PUT "requests that the state of the target resource be created or replaced with the state [in the body]" (RFC 7231 §4.3.4). Clearly, this excludes PUT as a viable option.
You can either send a GET with a body or send a POST and give up RESTish religiosity (it's not so bad, 5 years ago there was only one member of that faith -- his comments linked above).
Neither are great decisions, but sending a GET body may prevent problems for some clients -- and some servers.
Doing a POST might have obstacles with some RESTish frameworks.
Julian Reschke suggested above using a non-standard HTTP header like "SEARCH" which could be an elegant solution, except that it's even less likely to be supported.
It might be most productive to list clients that can and cannot do each of the above.
Clients that cannot send a GET with body (that I know of):
XmlHTTPRequest Fiddler
Clients that can send a GET with body:
most browsers
Servers & libraries that can retrieve a body from GET:
Apache
PHP
Servers (and proxies) that strip a body from GET:
?
What you're trying to achieve has been done for a long time with a much more common method, and one that doesn't rely on using a payload with GET.
You can simply build your specific search mediatype, or if you want to be more RESTful, use something like OpenSearch, and POST the request to the URI the server instructed, say /search. The server can then generate the search result or build the final URI and redirect using a 303.
This has the advantage of following the traditional PRG method, helps cache intermediaries cache the results, etc.
That said, URIs are encoded anyway for anything that is not ASCII, and so are application/x-www-form-urlencoded and multipart/form-data. I'd recommend using this rather than creating yet another custom json format if your intention is to support ReSTful scenarios.
I put this question to the IETF HTTP WG. The comment from Roy Fielding (author of http/1.1 document in 1998) was that
"... an implementation would be broken to do anything other than to parse and discard that body if received"
RFC 7213 (HTTPbis) states:
"A payload within a GET request message has no defined semantics;"
It seems clear now that the intention was that semantic meaning on GET request bodies is prohibited, which means that the request body can't be used to affect the result.
There are proxies out there that will definitely break your request in various ways if you include a body on GET.
So in summary, don't do it.
From RFC 2616, section 4.3, "Message Body":
A server SHOULD read and forward a message-body on any request; if the
request method does not include defined semantics for an entity-body,
then the message-body SHOULD be ignored when handling the request.
That is, servers should always read any provided request body from the network (check Content-Length or read a chunked body, etc). Also, proxies should forward any such request body they receive. Then, if the RFC defines semantics for the body for the given method, the server can actually use the request body in generating a response. However, if the RFC does not define semantics for the body, then the server should ignore it.
This is in line with the quote from Fielding above.
Section 9.3, "GET", describes the semantics of the GET method, and doesn't mention request bodies. Therefore, a server should ignore any request body it receives on a GET request.
Which server will ignore it? – fijiaaron Aug 30 '12 at 21:27
Google for instance is doing worse than ignoring it, it will consider it an error!
Try it yourself with a simple netcat:
$ netcat www.google.com 80
GET / HTTP/1.1
Host: www.google.com
Content-length: 6
1234
(the 1234 content is followed by CR-LF, so that is a total of 6 bytes)
and you will get:
HTTP/1.1 400 Bad Request
Server: GFE/2.0
(....)
Error 400 (Bad Request)
400. That’s an error.
Your client has issued a malformed or illegal request. That’s all we know.
You do also get 400 Bad Request from Bing, Apple, etc... which are served by AkamaiGhost.
So I wouldn't advise using GET requests with a body entity.
According to XMLHttpRequest, it's not valid. From the standard:
4.5.6 The send() method
client . send([body = null])
Initiates the request. The optional argument provides the request
body. The argument is ignored if request method is GET or HEAD.
Throws an InvalidStateError exception if either state is not
opened or the send() flag is set.
The send(body) method must run these steps:
If state is not opened, throw an InvalidStateError exception.
If the send() flag is set, throw an InvalidStateError exception.
If the request method is GET or HEAD, set body to null.
If body is null, go to the next step.
Although, I don't think it should because GET request might need big body content.
So, if you rely on XMLHttpRequest of a browser, it's likely it won't work.
If you really want to send cachable JSON/XML body to web application the only reasonable place to put your data is query string encoded with RFC4648: Base 64 Encoding with URL and Filename Safe Alphabet. Of course you could just urlencode JSON and put is in URL param's value, but Base64 gives smaller result. Keep in mind that there are URL size restrictions, see What is the maximum length of a URL in different browsers? .
You may think that Base64's padding = character may be bad for URL's param value, however it seems not - see this discussion: http://mail.python.org/pipermail/python-bugs-list/2007-February/037195.html . However you shouldn't put encoded data without param name because encoded string with padding will be interpreted as param key with empty value.
I would use something like ?_b64=<encodeddata>.
I wouldn't advise this, it goes against standard practices, and doesn't offer that much in return. You want to keep the body for content, not options.
You have a list of options which are far better than using a request body with GET.
Let' assume you have categories and items for each category. Both to be identified by an id ("catid" / "itemid" for the sake of this example). You want to sort according to another parameter "sortby" in a specific "order". You want to pass parameters for "sortby" and "order":
You can:
Use query strings, e.g.
example.com/category/{catid}/item/{itemid}?sortby=itemname&order=asc
Use mod_rewrite (or similar) for paths:
example.com/category/{catid}/item/{itemid}/{sortby}/{order}
Use individual HTTP headers you pass with the request
Use a different method, e.g. POST, to retrieve a resource.
All have their downsides, but are far better than using a GET with a body.
What about nonconforming base64 encoded headers? "SOMETHINGAPP-PARAMS:sdfSD45fdg45/aS"
Length restrictions hm. Can't you make your POST handling distinguish between the meanings? If you want simple parameters like sorting, I don't see why this would be a problem. I guess it's certainty you're worried about.
I'm upset that REST as protocol doesn't support OOP and Get method is proof. As a solution, you can serialize your a DTO to JSON and then create a query string. On server side you'll able to deserialize the query string to the DTO.
Take a look on:
Message-based design in ServiceStack
Building RESTful Message Based Web Services with WCF
Message based approach can help you to solve Get method restriction. You'll able to send any DTO as with request body
Nelibur web service framework provides functionality which you can use
var client = new JsonServiceClient(Settings.Default.ServiceAddress);
var request = new GetClientRequest
{
Id = new Guid("2217239b0e-b35b-4d32-95c7-5db43e2bd573")
};
var response = client.Get<GetClientRequest, ClientResponse>(request);
as you can see, the GetClientRequest was encoded to the following query string
http://localhost/clients/GetWithResponse?type=GetClientRequest&data=%7B%22Id%22:%2217239b0e-b35b-4d32-95c7-5db43e2bd573%22%7D
IMHO you could just send the JSON encoded (ie. encodeURIComponent) in the URL, this way you do not violate the HTTP specs and get your JSON to the server.
For example, it works with Curl, Apache and PHP.
PHP file:
<?php
echo $_SERVER['REQUEST_METHOD'] . PHP_EOL;
echo file_get_contents('php://input') . PHP_EOL;
Console command:
$ curl -X GET -H "Content-Type: application/json" -d '{"the": "body"}' 'http://localhost/test/get.php'
Output:
GET
{"the": "body"}
Even if a popular tool use this, as cited frequently on this page, I think it is still quite a bad idea, being too exotic, despite not forbidden by the spec.
Many intermediate infrastructures may just reject such requests.
By example, forget about using some of the available CDN in front of your web site, like this one:
If a viewer GET request includes a body, CloudFront returns an HTTP status code 403 (Forbidden) to the viewer.
And yes, your client libraries may also not support emitting such requests, as reported in this comment.
If you want to allow a GET request with a body, a way is to support POST request with header "X-HTTP-Method-Override: GET". It is described here : https://en.wikipedia.org/wiki/List_of_HTTP_header_fields. This header means that while the method is POST, the request should be treated as if it is a GET. Body is allowed for POST, so you're sure nobody willl drop the payload of your GET requests.
This header is oftenly used to make PATCH or HEAD requests through some proxies that do not recognize those methods and replace them by GET (always fun to debug!).
An idea on an old question:
Add the full content on the body, and a short hash of the body on the querystring, so caching won't be a problem (the hash will change if body content is changed) and you'll be able to send tons of data when needed :)
Create a Requestfactory class
import java.net.URI;
import javax.annotation.PostConstruct;
import org.apache.http.client.methods.HttpEntityEnclosingRequestBase;
import org.apache.http.client.methods.HttpUriRequest;
import org.springframework.http.HttpMethod;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;
#Component
public class RequestFactory {
private RestTemplate restTemplate = new RestTemplate();
#PostConstruct
public void init() {
this.restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestWithBodyFactory());
}
private static final class HttpComponentsClientHttpRequestWithBodyFactory extends HttpComponentsClientHttpRequestFactory {
#Override
protected HttpUriRequest createHttpUriRequest(HttpMethod httpMethod, URI uri) {
if (httpMethod == HttpMethod.GET) {
return new HttpGetRequestWithEntity(uri);
}
return super.createHttpUriRequest(httpMethod, uri);
}
}
private static final class HttpGetRequestWithEntity extends HttpEntityEnclosingRequestBase {
public HttpGetRequestWithEntity(final URI uri) {
super.setURI(uri);
}
#Override
public String getMethod() {
return HttpMethod.GET.name();
}
}
public RestTemplate getRestTemplate() {
return restTemplate;
}
}
and #Autowired where ever you require and use, Here is one sample code GET request with RequestBody
#RestController
#RequestMapping("/v1/API")
public class APIServiceController {
#Autowired
private RequestFactory requestFactory;
#RequestMapping(method = RequestMethod.GET, path = "/getData")
public ResponseEntity<APIResponse> getLicenses(#RequestBody APIRequest2 APIRequest){
APIResponse response = new APIResponse();
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
Gson gson = new Gson();
try {
StringBuilder createPartUrl = new StringBuilder(PART_URL).append(PART_URL2);
HttpEntity<String> entity = new HttpEntity<String>(gson.toJson(APIRequest),headers);
ResponseEntity<APIResponse> storeViewResponse = requestFactory.getRestTemplate().exchange(createPartUrl.toString(), HttpMethod.GET, entity, APIResponse.class); //.getForObject(createLicenseUrl.toString(), APIResponse.class, entity);
if(storeViewResponse.hasBody()) {
response = storeViewResponse.getBody();
}
return new ResponseEntity<APIResponse>(response, HttpStatus.OK);
}catch (Exception e) {
e.printStackTrace();
return new ResponseEntity<APIResponse>(response, HttpStatus.INTERNAL_SERVER_ERROR);
}
}
}