I want to upload a file from InputStream over HTTP, and for this, I am using the new HttpClient provided as part of JDK11. When I try to set the Content-Length while creating a post request I get an IllegalArgumentException saying restricted header name: "Content-Length" and if I try to upload a file from InputStream without the Content-Length header I get an internal server error from the server where I want to upload the file. Is there any option to set the Content-Length in the request in Java 11?
CodeI am using for creating HttpRequest:
var postRequest = HttpRequest.newBuilder()
.POST(HttpRequest.BodyPublishers.ofInputStream(() -> inputStream))
.uri(new URI(url))
.header(HttpHeaders.CONTENT_LENGTH, Long.toString(inputStreamSupplier.getFileSize()))
.header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_OCTET_STREAM_VALUE)
.build();
Note: It won't be possible to update to Java 12 to allow the restricted headers.
I could also use another library, just wanted to know if there is an option to use the classes from JDK before switching to RestTemplate from Spring. (yes, it's deprecated, As the alternative uses spring boot can't use it at the moment)
Simply use fromPublisher:
var bodyPublisher = BodyPublishers
.fromPublisher(BodyPublishers.ofInputStream(()-> inputStream)), length);
Note that you must ensure that the InputStream delivers exactly length bytes.
Related
I have to perform POST requests from Jmeter. I use default HTTPRequest sampler, where I specify the JSON structure that is understandable by the testing app. One part of this JSON has to contain binary data from a pdf file.
For reading the file I use BeanShellSampler in the setUp thread group:
File file = new File(bsh.args[0]);
try {
FileInputStream fis = new FileInputStream(file);
byte[] array = new byte[(int)file.length()];
log.info("String is read.");
fis.read(array);
vars.put("fileEntity", new String(array, "cp1252"));
} catch (e) {
e.printStackTrace();
log.error(e.getMessage());
}
The problem is that when I look at the request with Fiddler, I see that difference, how the binary object is represented there in comparison with Postman's requests:
Postman
Jmeter
I think that there is something wrong with the encoding when I create a String object in the BeanShellSampler. What encoding is correct?
I tried to use RawDataSource plugin but it doesn't help for two reasons:
It fails to read my file, saying "Error reading next chunk"
It uses the same approach that I do to read the file, but uses UTF8 encoding. I tried this encoding also, but without any success.
My expectation is that your fis.read(array); function relies on default value of the file.encoding system property which may or may not be cp1252.
I would recommend introducing an InputStreamReader and explicitly specify the encoding there like:
InputStreamReader isr = new InputStreamReader(fis,"cp1252");
Also be aware that starting from JMeter 3.1 it's recommended to use JSR223 Test Elements and Groovy language for scripting mainly because Groovy performs much better comparing to Beanshell.
I need to read gzipped json payload in a POST request in my SPringBoot app which accepts json data. How to do that in order to keep the application generic as there may be other clients in future sending data in plain json or other compression formats? I suppose this should be handled by the server itself so is there any way to instruct the embedded Tomcat to unzip the payload?
My SpringBoot application runs on embedded Tomcat 9.0.17.
The controller accepts JSON payload in a POST request.
#RequestMapping(value = "/update/v1", method = RequestMethod.POST, produces = "application/json", consumes = "application/json")
public ResponseEntity<String> receiveUpdates(#RequestBody String update) {
We recently changed our content provider and the new one is sending payload in "gzip" format (with header content-encoding=gzip) and without any content-type header. As a result it gives the following error
'error': 'Unsupported Media Type', 'message': "Content type '' not supported"
If I change my consume type to MediaType.ALL_VALUE, my controller starts receiving the request but the payload itself is gzipped. I can handle it in my service layer but that would make it specific to gzipped data.
This problem could be solved by introducing a Filter to handle gzipped payload as mentioned here and here.
But I believe there should be a way to instruct the Tomcat to handle this and serve unzipped data.
I enabled gzip compression for all the responses in my web service (Play 2.4) by following those instructions. Easy to set up, and I can see it works like a charm having checked with curl and wireshark that the responses are sent compressed.
Now I want to be a good developer and add an integration test to make sure no one breaks HTTP compression next week. Here's where the fun begins! My test looks like this:
"use HTTP compression" in {
forAll(endPoints) { endPoint =>
val response = await(
WS.url(Localhost + port + "/api" + endPoint).withHeaders("Accept-Encoding" -> "gzip").get()
)
response.header("Content-Encoding") mustBe Some("gzip")
}
}
However, the test fails as WS's response headers don't include content enconding information, and the body is returned as plain text, uncompressed.
[info] - should use HTTP compression *** FAILED ***
[info] forAll failed, because:
[info] at index 0, None was not equal to Some("gzip") (ApplicationSpec.scala:566)
Checking the traffic in wireshark when running this test I can clearly see the server is returning a gzip-encoded response, so it looks like WS is somehow transparently decompressing the response and stripping the content-encoding headers? Is there a way I can get the plain, compressed response with full headers so I can check whether the response is compressed or not?
I don't think you can do that. If I'm not mistaken , the problem here is that Netty return the content already uncompressed, so the header is removed also.
There is a configuration in AsyncHTTPClient to set that (setKeepEncoding), but unfortunately this only works in version 2.0 and newer, and Play 2.4 WS lib uses version 1.9.x.
Either way, the client Play gives you is already configured, and I don't know if you are able to tweak it. But you can create a new client to emulate that behavior:
// Converted from Java code: I have never worked with those APi's in Scala
val cfg = new AsyncHttpClientConfig.Builder().addResponseFilter(new ResponseFilter {
override def filter[T](ctx: FilterContext[T]): FilterContext[T] = {
val headers = ctx.getRequest.getHeaders
if (headers.containsKey("Accept-Encoding")) {
ctx.getResponseHeaders.getHeaders.put("Content-Encoding", List("gzip"))
}
ctx
}
}).build()
val client: NingWSClient = NingWSClient(cfg)
client.url("...") // (...)
Again, this is just emulating the result you need. Also, probably a more clever logic than just add gzip as Content-Encoding (ex: put the first algorithm requested in "Accepts Encoding") is advised.
Turns out we can't really use Play-WS for this specific test because it already returns the content uncompressed and stripped of the header (see #Salem's insightful answer), so there's no way to check whether the response is compressed.
However it's easy enough to write a test that checks for HTTP compression using standard Java classes. All we care about is whether the server answers in (valid) GZIP form when sending a request with Accept-Encoding: gzip. Here's what I ended up with:
forAll(endPoints) { endPoint =>
val url = new URL(Localhost + port + "/api/" + endPoint)
val connection = url.openConnection().asInstanceOf[HttpURLConnection]
connection.setRequestProperty("Accept-Encoding", "gzip")
Try {
new GZIPInputStream(connection.getInputStream)
} must be a 'success
}
I have an application using embedded Jetty to serve both the frontend of the site but also the API.
It's written in Scala, but that's not really relevant. My file that sets everything up looks like this:
val servlet = new ServletHolder(Servlet)
val frontend = new ServletHolder(new DefaultServlet())
frontend.setInitParameter("resourceBase", "frontend")
val handler = new ServletContextHandler()
handler.addServlet(servlet, "/api/*")
handler.addServlet(frontend, "/*")
val server = new Server(8080)
server.setHandler(handler)
server.start()
However, static content served from / requires that I use file extensions for my static content - I would like to use content negotiation so that these are not required. Is this possible?
You are not serving static files if you want to use content negotiation.
Content Negotiation is a fundamental concept of HTTP, and it really for the content that is being served from the (hand waving) "resource", that you are requesting.
Serving static files is a specialized form of "resource", where the mime-type / content-type is known based on its file extension.
The DefaultServlet (which is doing the static file serving in your example) has 1 more feature on top of the this specialization, but not for content-type, but rather content-encoding (you can pre-compress your static resources by creating a <filename>.gz file that sits next to the original file, if the requesting client indicates that they can accept gzip, then this <filename>.gz is served instead of the uncompressed <filename> version.
In order to accomplish this goal, you'll need to write something that serves the static files in a way that makes sense for you.
What you'll need if you want to do this yourself.
A new servlet on url-pattern /*
The new servlet's .init() creates an in-memory data structure that houses all of the known static files you have, plus their extensions, and mime-types.
The new servlet's .doGet() will handle any incoming requests by seeing if there is an acceptable resource to serve based on this in-memory lookup. Serve the actual content you want from this in-memory lookup.
Don't forget to support ETag, Ranged requests, Server response Cache, HTTP Cache, and Gzip.
I am using Mule ESB version 1.3 and as soap engine it has AXIS1.4 embedded.Mule uses the code below to send stream as ws attachment. I checked they are still using it in newer versions.
File temp = File.createTempFile("soap", ".tmp");
temp.deleteOnExit(); // TODO cleanup files earlier (IOUtils
// has a file tracker)
FileOutputStream fos = new FileOutputStream(temp);
msgContext.getRequestMessage().writeTo(fos);
fos.close();
contentLength = (int)temp.length();
payload = new FileInputStream(temp);
Here the content of soap message with attachment is saved on a file and handle for this file is given to the rest of system. I did not get the logic behind this. Why not just use
javax.activation.DataHandler
inputstream and instead first saving and then reading message from file. As far as I understand Axis1.4 itself not working that way. Am I missing something here ?
After some modifications to make mule send file I came accross the below difference between axis and mule client generated messages. Except this line generated messages are same.
Mule generate this header:
Content-Type: multipart/related;type="application/xop+xml"
Axis generate this header which is correct:
Content-Type: multipart/related;type="application/xop+xml"; start="<2F4952A019F62AB6704A0C06DB5E8AA1>"; start-info="text/xml; charset=utf-8"; .boundary="----=_Part_0_1836994030.1387284330292"
What is causing this difference ?