Httpclient throws Timeout waiting for connection from pool exception - httpclient

Now am working on an exception for our uploading image service, the scenario below:
We have a web page, user from all over the world can upload their images to our server, the image normally keep about 3MB. Now we held a promotion, so the images number uploading to our server is extremely huge, which, however caused the server throws the exception as "org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool".
We use apache httpclient as the core uploading middleware, its version is 4.5.*, we correctly handled the response by using the method mentioned in this article.
The code like below:
if (returnType != StorageHttpResponse.class && response != null) {
EntityUtils.consumeQuietly(response.getEntity());
httpRequest.abort();
}
Also, the max connection pool for the service is 128 and the max connection time out is 50000 ms. We upload the images by using stream mode, not directly upload the image file.
So here, I correctly handled the response entity by consuming it in finally code block, but I still can't stop the service throw connection pool timeout exception.
Any other stuffs that I need to add to my service? Do I really using redis to make a queue to user's uploading requests and post handling?
Whole code here:
public <T> T excute(Request request, Class<T> returnType) {
Preconditions.checkState(!isShutDown, "JSSHttpClient is destory!");
HttpRequestBase httpRequest = new HttpRequestBuild(this.credential).build(request);
HttpResponse response = null;
try {
response = this.client.execute(httpRequest);
if (errorHandler.hasError(request, response)) {
int statusCode = response.getStatusLine().getStatusCode();
log.warn("Unexpected response," + request + " http code [" + statusCode + "]");
errorHandler.handleError(response);
}
if (returnType != null && returnType != StorageHttpResponse.class) {
return JsonMessageConverter.read(returnType, response);
}
if (returnType == StorageHttpResponse.class) {
return (T) new StorageHttpResponse(response);
}
} catch (IOException e) {
Throwables.propagate(e);
} finally {
if (returnType != StorageHttpResponse.class && response != null) {
EntityUtils.consumeQuietly(response.getEntity());
httpRequest.abort();
}
}
return null;
}

you can set the parameters in either properties or yml file like below.
http:
pool:
size: 100
sockettimeout: 20000
defaultMaxPerRoute: 200
maxPerRoutes:
-
scheme: http
host: localhost
port: 8080
maxPerRoute: 100
-
scheme: https
host: {{URL}}
port: -1
maxPerRoute: 200

Finally, we solved this not by using code. Because we all know if the response not consumed directly, the connection of a request will not released. So in our code, we offen consume the response first.
We solved this problem not by using better code but slightly modify some parameters like maxconnectionpoolsize, maxconnectionperroute and maxconnectiontimeout based on our business scenario. Then running it and all seems ok now. Hope this helps you.

Related

Vertx - threads are stuck while sending response back to client

I'm using vertx-4.2.6 to build a proxy service which takes requests from clients (for ex: browser, standalone apps etc), invoke a single thirdparty server, gets the response and send the same response back to client who initiated the request.
In this process, I'm using shared Webclient across multiple requests, i'm getting response from thirdparty quickly (mostly in milli seconds) but sometimes the response is not returned back to client and stucks at ctx.end(response).
Whenever i restart my proxy server, it serves requests sometimes without any issues but time goes on, lets say by EOD, for new requests client seeing 503 error -service unavailable I'm using one MainVerticle with 10 instances. I'm not using any worker threads.
Below is the pseudo code:
MainVerticle
DeploymentOptions depOptions = new DeploymentOptions();
depOptions.setConfig(config);
depOptions.setInstances(10);
vertx.deployVerticle(MainVerticle.class.getName(), depOptions);
.....
router.route("/api/v1/*")
.handler(new HttpRequestHandler(vertx));
HttpRequestHandler
public class HttpRequestHandler implements Handler<RoutingContext> {
private final Logger LOGGER = LogManager.getLogger( HttpRequestHandler.class );
private WebClient webClient;
public HttpRequestHandler(Vertx vertx) {
super(vertx);
this.webClient=createWebClient(vertx);
}
private WebClient createWebClient(Vertx vertx) {
WebClientOptions options=new WebClientOptions();
options.setConnectTimeout(30000);
WebClient webClient = WebClient.create(vertx,options);
return webClient;
}
#Override
public void handle(RoutingContext ctx) {
ctx.request().bodyHandler(bh -> {
ctx.request().headers().remove("Host");
StopWatch sw=StopWatch.createStarted();
LOGGER.info("invoking CL end point with the given request details...");
/*
* Invoking actual target
*/
webClient.request(ctx.request().method(),target_port,target_host, "someURL")
.timeout(5000)
.sendBuffer(bh)
.onSuccess(clResponse -> {
LOGGER.info("CL response statuscode: {}, headers: {}",clResponse.statusCode(),clResponse.headers());
LOGGER.trace("response body from CL: {}",clResponse.body());
sw.stop();
LOGGER.info("Timetaken: {}ms",sw.getTime()); //prints in milliseconds
LOGGER.info("sending response back to client...."); //stuck here
/*
* prepare the final response and return to client..
*/
ctx.response().setStatusCode(clResponse.statusCode());
ctx.response().headers().addAll(clResponse.headers());
if(clResponse.body()!=null) {
ctx.response().end(clResponse.body());
}else {
ctx.response().end();
}
LOGGER.info("response SENT back to client...!!"); //not getting this log for certain requests and gives 503 - service unavailable to clients after 5 seconds..
}).onFailure(err -> {
LOGGER.error("Failed while invoking CL server:",err);
sw.stop();
if(err.getCause() instanceof java.net.ConnectException) {
connectionRefused(ctx);
}else {
invalidResponse(ctx);
}
});
});
Im suspecting issue might be due to shared webclient. But i'm not sure. I'm new to Vertx and i'm not getting any clue what's going wrong. Please suggest if there are any options to be set on WebClientOptions to avoid this issue.

Why can not I read bytes from the TcpClient in C#?

Why can not I read bytes from the TcpClient in C#?
Here is the error I am getting:
Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.
Here is how I start my TcpClient:
public static async void Start()
{
TcpListener server = null;
try
{
server = new TcpListener(IPAddress.Loopback, 13000);
server.Start();
var client = await server.AcceptTcpClientAsync();
var stream = client.GetStream();
var bytes = Convert.FromBase64String("ABCD");
await stream.WriteAsync(bytes, 0, bytes.Length);
client.Close();
}
catch (Exception e)
{
throw;
}
finally
{
if(server != null)
{
server.Stop();
}
}
}
Here is how I run a request to the TcpClient:
try {
var response = (new HttpClient()).GetByteArrayAsync("http://localhost:13000").Result;
return Convert.ToBase64String(response);
} catch(Exception e) {
throw;
}
The return Convert.ToBase64String(response); line is never reached. While I see the quoted above error message inside the Exception e if I hit a breakpoint on the throw line.
Also, during debug the Start() method completes just fine. I.e. it starts, then wait for a request, gets a request, writes to the TclClient and at the end runs the server.Stop(); command.
I am expecting my code to work, because I took it and modified from the official documentation over here.
I tried to check out a few resources which would tackle my exception, but none of them did help.
E.g. I tried to use the question.
First answer tells nothing useful actually, but just plays around with words and at the end states that one can do nothing about the exception (please, correct me if I am missing a point in the answer).
And the second answer tells an impossible in my case problem. Because, I am sure there is nothing running on the 13000 port.
Your client code is using HttpClient, which sends an HTTP request and expects an HTTP response. But your server is not an HTTP server, it is just a plain TCP server, so the client is likely to fail and forcibly close the connection when it doesn't receive a properly formatted HTTP response.
The "official documentation" whose example you modified is not using HttpClient at all, it is using TcpClient instead.
If you want to use HttpClient in your client, then you should use HttpListener instead of TcpListener in your server.

S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream

I am getting this warning when doing s3 GET Object request
WARN S3AbortableInputStream: Not all bytes were read from the
S3ObjectInputStream, aborting HTTP connection. This is likely an error
and may result in sub-optimal behavior. Request only the bytes you
need via a ranged GET or drain the input stream after use
I looked at AmazonS3: Getting warning: S3AbortableInputStream:Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection
I do not have a clear understanding of what needs to be done.
Below is the code snippet I am using
try {
s3obj = s3Client.getObject(s3Bucket, objectSummary.getKey)
val name = s3obj.getKey.substring(dirPath.length + 1)
val filename = name.replace(".xml", "")
endpoints += base_url + filename
} catch {
case e#(_: IOException |_ : JSONException | _: Exception) =>
println(e)
} finally {
if (s3obj != null)
s3obj.close()
}
Basically what it does is, if S3 client tries to close an InputStream without consuming it fully after reading N bytes, it gives the warning message because they need to close the HttpRequest and the underlying http connection with removing the connection from the connection pool.
This could happen when:
Web application reads data from S3 initially and S3 DataStore normally caches the data in local file system and while this is happening, if web visitor disconnects the HTTP connection, and so if Tomcat indicates it, then the http processing thread will be halted and as a result, the InputStream will be closed.
Please refer the following link:
https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html
If you add
s3obj.abort()
to your exception block, it should resolve the issue. It would look like this:
} catch {
case e#(_: IOException |_ : JSONException | _: Exception) =>
println(e)
if (s3obj != null) {
s3obj.abort()
}
} finally {
if (s3obj != null)
s3obj.close()
}
It is a complexity of the AWS SDK in that it wants to avoid the normal close() operations to short cut streaming lots of data when it is not necessary.

jmeter send email assertion

I am new to Jmeter and am looking for a way to send emails out for every failed assertion.
Test Structure:
Thread Group:
Transaction Controller:
Http Request:
Http Request:
Http Request:
Thread Group:
Transaction Controller:
Http Request:
Http Request:
Http Request:
Each Http request contains a response assertion. I would like to capture and send one email that contains all of the failed assertions within a transaction controller. Is there a way to do this? I tried using adding a SMTP Sampler within a thread group with a child Bean PreProcessor that contains this code:
import org.apache.jmeter.assertions.AssertionResult;
try {
AssertionResult[] results = prev.getAssertionResults();
StringBuilder body = new StringBuilder();
for (AssertionResult result : results) {
body.append(result.getName());
body.append(result.getFailureMessage());
body.append(System.getProperty("line.separator"));
}
vars.put("body", body.toString());
}
catch (Throwable ex) {
log.error("Error in Beanshell", ex);
throw ex;
}
When i do this it will only send an email of the last failed assertion instead of all failed assertions.
Add a global JSR223 Listener (it is much better to use Groovy rather than Beanshell, see Groovy is the New Black article for details)
Put the following code into "Script" area (it is basically your code but amended to read previous samplers results from the ${body} variable and append new results to them
import org.apache.jmeter.assertions.AssertionResult;
try {
AssertionResult[] results = prev.getAssertionResults();
StringBuilder body = new StringBuilder();
String previousBody = vars.get("body");
if (previousBody != null) {
body.append(previousBody);
}
for (AssertionResult result : results) {
body.append(result.getName());
body.append(result.getFailureMessage());
body.append(System.getProperty("line.separator"));
}
vars.put("body", body.toString());
} catch (Throwable ex) {
log.error("Error in Groovy", ex);
throw ex;
}
We have a test that prints out request body information to the console on failure. This is added to each sampler individually and works quite nicely. You could update this code send a failure message instead of printing to the console.
//Loop through assertions and check for failed
for (int i = 0; i < results.length; i++){
if (results[i].isFailure() || results[i].isError()){
failed_assertion = true;
System.out.println("\n********** Request Body ***************\n"+ctx.getCurrentSampler().getUrl().getPath()+ctx.getCurrentSampler().getQueryString()+'\n'+ctx.getCurrentSampler().getArguments().getArgument(0).getValue()+"\n");
System.out.println("************ ERROR DETECTED ***********\n"+prev.getResponseDataAsString()+"\n****************************************\n");
break;
}
}
If that doesn't work you could construct an IF block on each sampler. Similar to what they're doing here.

Spring cloud performance tunning

I'm doing a performance test against a Spring Cloud application. When number of concurrent users exceeds 150, it starts to give "Forwarding error"
{"timestamp":1458685370986,"status":500,"error":"Internal Server Error","exception":"com.netflix.zuul.exception.ZuulException","message":"Forwarding error"}
Which parameter I should adjust to get rid of the error?
You should post your logs for the error, without that we can only guess what the exact error is. As Forwarding error reported by ZuulExcetption is a generic error.
See this link for the RibbonRoutingFilter.forward() method which actually reports this error. I'm adding the code here for the backup.
private HttpResponse forward(RestClient restClient, String service, Verb verb, String uri, Boolean retryable,
MultiValueMap<String, String> headers, MultiValueMap<String, String> params,
InputStream requestEntity) throws Exception {
Map<String, Object> info = this.helper.debug(verb.verb(), uri, headers, params,
requestEntity);
RibbonCommand command = new RibbonCommand(service, restClient, verb, uri, retryable,
headers, params, requestEntity);
try {
HttpResponse response = command.execute();
this.helper.appendDebug(info, response.getStatus(),
revertHeaders(response.getHeaders()));
return response;
}
catch (HystrixRuntimeException ex) {
info.put("status", "500");
if (ex.getFallbackException() != null
&& ex.getFallbackException().getCause() != null
&& ex.getFallbackException().getCause() instanceof ClientException) {
ClientException cause = (ClientException) ex.getFallbackException()
.getCause();
throw new ZuulException(cause, "Forwarding error", 500, cause
.getErrorType().toString());
}
throw new ZuulException(ex, "Forwarding error", 500, ex.getFailureType()
.toString());
}
}
As you can see that only viable place where the error can be generated is in command.execute(), where command is an instance of HystrixCommand. Here is a link for the execute() method in HystrixCommand.
Below is the code for backup.
public R execute() {
try {
return queue().get();
} catch (Exception e) {
throw decomposeException(e);
}
}
Here the queue() is a Future instance
Most common error that can occur with the Future is a timeout exception. Since here Future instance queue() is not bound by any timetout value, it can go on waiting for ever.
However most of the time API which make use of Future have a thread monitoring the time they take and they interrupt it after a certain period of time. Same is done by Ribbon.
If yours indeed is a timeout issue then an easy solution is to increase Ribbon timeout value by using following property.
ribbon.ReadTimeout=10000
//or
<client-name>.ribbon.ReadTimeout=10000
Time out majorly can occur if the tomcat server which hosts the service which is proxied by the Zuul has too much load. It's whole thread pool is exhausted thus resulting in the next requests having to wait for long time.
This can probably be alleviated by change the number of threads that your service tomcat has by using following property.
server.tomcat.max-threads=0
By default it's set to 0, which leaves it to the embedded server's default. In tomcat's case it's 200. See the reference maxThreads property in tomcat.
Note: To increase the thread pool size we have to make sure that the machine has that capacity to provide resources if that many threads were to be in execution simultaneously.