We use talend to build ETLs. One of the components we use is tRESTClient, to interact to a REST service in our platform. We use talend version 7.1.
Recently this REST service has changed the version of its HTTP server library (libmicrohttpd, the REST service is based in C/C++). In particular, has moved from limicrohttpd 0.9.48 to 0.9.70. And since we did the change, the integration in the ETL has started to fail. We see message likes this in the talend traces:
--------------------------------------
time=2020-08-05 08:26:01| lvl=INFO | ver=1.0| Inbound Message
----------------------------
ID: 1
Response-Code: 204
Encoding: ISO-8859-1
Content-Type:
Headers: {Cache-Control=[no-cache, no-store, max-age=0, must-revalidate], connection=[keep-alive], Date=[Wed, 05 Aug 2020 06:26:03 GMT], Expires=[0], fiware-correlator=[894f7ff0-d6e4-11ea-b1bd-fa163ec74401], Pragma=[no-cache], Referrer-Policy=[same-origin], Server=[nginx], Strict-Transport-Security=[max-age=31536000; includeSubdomains;], X-Content-Type-Options=[nosniff], X-Frame-Options=[SAMEORIGIN], X-XSS-Protection=[1; mode=block]}
--------------------------------------
Exception in component tRESTClient_1 (VLCI_ETL_MEDIOAMBIENTE_INSERT_CB)
javax.ws.rs.client.ResponseProcessingException: Problem with reading the data, class org.dom4j.Document, ContentType: */*.
at org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:437)
at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:379)
at org.apache.cxf.jaxrs.impl.ResponseImpl.readEntity(ResponseImpl.java:314)
at org.apache.cxf.jaxrs.impl.ResponseImpl.readEntity(ResponseImpl.java:304)
…
We are still investigating the problem but we suspect that one of the changes introduced in libmicrohttpd could be related. In particular, with libmicrohttd 0.9.48 (when the ETL works) “Content-Length: 0” header was included in 204 responses. With libmicrohttpd 0.9.70 (when ETL doesn’t work) such “Content-Length: 0” header is not included.
Is this a known issue? I mean, is tRESTClient (version 7.1) expected to fail if “Content-Lengh: 0” is not included in 204 responses?
Note that the HTTP specification is pretty clear regarding Content-Length usage in 204 responses. From RFC 7321 (which as far as I know is the normative standard on HTTP) section 6.1:
A server MUST NOT send a Content-Length header field in any response with a status code of 1xx (Informational) or 204 (No Content)”
so if tRESTClient is not supporting it, then I understand is a bug in talend (maybe solved in post-7.1 releases?)
Any feedback on this is really welcomed! Thank you for your attention.
Finally, it seems we have solved the issue in the following way.
In the failing situation the component was using this configuration:
We have changed Accept Type from "JSON" to "Any" in the following way:
And now it is working.
My guess is that with Accept Type "JSON" the component is waiting for a response payload and this is problematic in the case of 204 No Content. Using "Any" removes that restriction.
Maybe it worked by chance with the non-standard content-length: 0 header in this case... It's a bit weird anyway, I think the tRESTClient should deal with this situation better than this. Maybe is a behavior already fixed in the post-7.1 versions.
Any feedback or comment on this solution would be welcomed. Thanks!
Related
We have a REST API to fetch binary files from the server.
The requests look like
GET /documents/e62dd3f6-18b0-4661-92c6-51c7258f9550 HTTP/1.1
Accept: application/octet-stream
For every response indicating an error, we'd like to give a reason in JSON.
The problem is now, that as the response is not of the same content type as the client requested.
But what kind of response should the server produce?
Currently, it responds with a
HTTP / 1.1 406 Not Acceptable
Content-Type: application/json
{
reason: "blabla"
...
}
Which seems wrong to me, as the underlying issue is, that the resource is not existing and not the client requesting the wrong content type.
But the question is, what would be the right way to deal with such situations?
Is it ok, to respond with 404 + application/json although application/octet-stream was requested
Is it ok, to respond with 406 + application/json, as the client did not specify an application/json as an acceptable type
Should spec been extended so that the client should use the q-param - for example, application/octet-stream, application/json;q=0.1
Other options?
If no representation can be found for the requested resource (because it doesn't exist or because the server wishes to "hide" its existence), the server should return 404.
If the client requests a particular representation in the Accept header and the server is not available to provide such representation, the server could either:
Return 406 along with a list of the available representations. (see note** below)
Simply ignore the Accept header and return a default representation of the resource.
See the following quote from the RFC 7231, the document the defines the content and semantics of the HTTP/1.1 protocol:
A request without any Accept header field implies that the user agent will accept any media type in response. If the header field is present in a request and none of the available representations for the response have a media type that is listed as acceptable, the origin server can either honor the header field by sending a 406 (Not Acceptable) response or disregard the header field by treating the response as if it is not subject to content negotiation.
Mozilla also recommends the following regarding 406:
In practice, this error is very rarely used. Instead of responding using this error code, which would be cryptic for the end user and difficult to fix, servers ignore the relevant header and serve an actual page to the user. It is assumed that even if the user won't be completely happy, they will prefer this to an error code.
** Regarding the list of available representations, see this answer.
I have been trying to create a REST/Json Virtual service on CA LISA 7.5 (we can’t update), using request response pairs. The request response looks like the following:
Name-req
GET /cods_party_web/party/111700 HTTP/1.1
Pragma: no-cache
Cache-Control: no-cache
x-abc-outlet-id: 017879
x-abc-user-id: CTM
x-abc-consent-level: 2
x-abc-application-id: 00028
x-abc-outlet-id-type: OU_ID
x-abc-user-id-type: 1
x-IBM-Client-Id: XXX....
x-IBM-Client-Secret: XXX...
Name-rsp
HTTP/1.1 200 {"party":{"partyId":111700,"foreNames":["Julie","Pamela",""],"lastName":"Duncan","initials":["J"],"...lots of content......."type":"EMAIL"}],"associatedOU":null}
When I try to build the virtual service image, no matter what options I select, my VS image response is either in hex as shown below or it is blank.
I remember having this problem 1 year ago, and was able to get the reponse to look like below but I can’t remember how I did it.
Success response
Not found response.
Many thanks in advance
It's not returning hex - those are just column numbers for an empty binary response. I think the problem is your response document is not properly formed HTTP - there's no reason phrase in the status line, and you need two line feeds after the status line. Try this:
HTTP/1.1 200 OK
{"party":{"partyId":111700,"foreNames":["Julie","Pamela",""],"lastName":"Duncan","initials":["J"],"...lots of content......."type":"EMAIL"}],"associatedOU":null}
I understand that you can't upgrade, so this doesn't really help you, but LISA 9.5 doesn't have this issue -- the response looks like it's supposed to.
On the other hand, CA has released a free, simpler version of LISA that also successfully generates a VS from your example. Check it out here:
http://educationcontent.ca.com/A01/index.html
I'm designing a simple versioned RESTful API. The version will be specified as part of the media type in the Accept header. I wonder what are the proper HTTP status codes for these cases:
version not yet supported, e.g.
Accept: application/json; version=999.0
version no longer supported
Accept: application/json; version=0.1
Seems you should use 406 Not Acceptable for both cases.
406 Not Acceptable The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.
According to ASP.NET repo, that would either be 400 or 405, depending on the situation:
Status Code
Error Code
Summary
400
ApiVersionUnspecified
An API version is required, but was not specified
400, 405
UnsupportedApiVersion
The specified API version is not supported
400
InvalidApiVersion
An API version was specified, but it is invalid
400
AmbiguousApiVersion
An API version was specified multiple times with different values
For more info see: https://github.com/dotnet/aspnet-api-versioning/wiki/Error-Responses
I am calling a SOAP web service as client.
Following is content-type value of response
Content-Type: text/xml
I requested customer to add UTF-8 to response as follow:
Content-Type: text/xml;charset=utf-8
But customer says that it can be from client side. Is this possible? Can I determine content type of server as client?
PS: I noticed that the cited RFC2376 is obsolete by RFC3023 (conservative enough) and then RFC7303 that I'm omitting to evaluate now in involved current use and content, so the relevance of the following might not be that definitive, I'm feeling to delete it.
You have everything formal in RFC2376 XML Media Types: Section 3.1 text/xml Registration
See also Section 6 Examples of that RFC, particularly Section 6.4 text/xml with Omitted Charset
The server side (your customer) is STRONGLY RECOMMENDED to use charset parameter that they are not currently using.
And if charter is omitted XML processors MUST use the default charset value of "us-ascii"
You are right asking the customer to specify charset, the "MUST" in the RFC is a strong requirement that limits also your adaptability from client side when they are not sending us-ascii.
What is the appropriate way of giving an estimate for request completion when the server returns a 202 - Accepted status code for asynchronous requests?
From the HTTP spec (italics added by me):
202 Accepted
The request has been accepted for processing, but the processing has not been completed. [...]
The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.
Here are some of thoughts:
I have glanced at the max-age directive, but using it would be abusing Cache-Control?
Return the expected wait time in the response body?
Add an application specific X- response header, but the X-headers was deprecated in RFC 6648?
Add a (non X-) specific response header? If so, how should it be named? The SO question Custom HTTP headers : naming conventions gave some ideas, but after the deprecation it only answers how HTTP headers are formatted, not how they should be named.
Other suggestions?
Definitely do not abuse existing HTTP headers for this. Since it's your own server, you get to define what the response looks like. You can (and should) pick whatever response works best for the intended recipient of this information and return the actual information in the response body.
For example, if you are only interested in displaying a human-readable message then you could return text/plain saying "Your request is likely to be processed in the next 30 minutes.".
At the other end of the spectrum, you might want to go all the REST way and return application/json, perhaps formatted like this (I totally made this up on the spot):
{
"status": "pending",
"completion": {
"estimate": "Thu Sep 08 2011 12:00:00 GMT-0400",
"rejected-after": "Fri Sep 09 2011 12:00:00 GMT-0400",
},
"tracking": {
"url": "http://server/status?id=myUniqueId"
}
}
You can use the Location header to specify the URL of the status monitor. Things like current status and estimate can either go in custom headers (which noone but your own software would use), or in the response body (which a web browser would display to a user, at least).
Although not explicitly mentioned specifically for the 202 - Accepted response code, the Retry-After header seems to be a suitable option. From the documentation:
The Retry-After response-header field can be used [...] to indicate how long the service is expected to be unavailable to the requesting client.