Read timed out when using Hystrix on RPC - spring-cloud

I use #HystrixCommand to package a RPC,if not use Hystrix it will run normally。This RPC is not a microservice instead a normall http
#Override
#HystrixCommand(commandProperties = {#HystrixProperty( name ="execution.isolation.thread.timeoutInMilliseconds",value="5000")},fallbackMethod="doAccountQueryFallback")
public CBIBAcctQueryResponseVO doAccountQuery(#RequestBody CBIBAcctQueryRequestVO cbibAcctQueryRequestVO) {
//initialize message
// ....
try {
responseMessage = (AcctTypeInqResponse) HTTPUtil.sendHttp("http://********", inqRequest, *****.class);
} catch (RuntimeException e) {
//...
}
logger.debug("success");// 1
return cbibAcctQueryResponseVO;
}
public CBIBAcctQueryResponseVO doAccountQueryFallback(#RequestBody CBIBAcctQueryRequestVO cbibAcctQueryRequestVO){
//....
}
the error infomation like:
feign.RetryableException: Read timed out executing POST http://****/doAccountQuery
at feign.FeignException.errorExecuting(FeignException.java:67)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:104)
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
at com.sun.proxy.$Proxy153.doAccountQuery(Unknown Source)
...
...
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
I change the value of the #HystrixCommand properties. Though I set a long time it will also show this exception,do // 1 and do not call fallback method.
When I set 100 ,it show this exception,do // 1 and call fallback method.
I suspect is it send some times?
When I set 1 ,it does not show this exception and call fallback method.
it's actually expected.
the doAccountQuery is a mircoservice and there is a normal http request in the doAccountQuery. I want to package this normal http request with Hystrix only(not use feign).but once i add # #HystrixCommand on the doAccountQuery.the microservice doAccountQuery itself will timed out,but it also will return the result (retransmission? only the first transmission timed out ? I dont know)
1 also try
ribbon:
ReadTimeout: 60000
ConnectTimeout: 60000
but useless
Besides sometimes the exception will become
connect timed out and not do //1 and even not call fallback method!
I want have some debug suggest thanks!

Related

ErrorDecoder not catching SocketTimeoutException

With feign builder I am making call to external URL. I have set readTimeout as 2seconds, I am getting the SocketTimeoutException but it's not going to ErrorDecoder.
Feign Builder Configuration:
Options options = new Options(10000, TimeUnit.MILLISECONDS, 2000,
TimeUnit.MILLISECONDS, false);
return Feign.builder()
.logLevel(level)
.client(client)
.retryer(Retryer.NEVER_RETRY)
.options(options)
.errorDecoder(feignErrorDecoder())
.exceptionPropagationPolicy(UNWRAP);
private ErrorDecoder feignErrorDecoder() {
return (methodKey, response) -> {
return new MyCustomException("ERROR_TIMEOUT",
"Timeout Occurred: " + response.status());
};
}
When I call service exception is not going to ErrorDecoder. java.lang.reflect.UndeclaredThrowableException coming.
ErrorDecoder is not called when an IOException (SocketTimeoutException) is thrown. See SynchronousMethodHandler#executeAndDecode(...). In general decoders are only called when a response is returend by the api-call. The ErrorCoder is only called when the http error code is not 2xx and 4xx.

How to catch/Capture "javax.net.ssl.SSLHandshakeException: Failed to create SSL connection" while sending message over java vert.x Eventbus

I am trying to use SSL over eventbus. To test the failure case I tried sending message to the eventbus from another verticle in same cluster by passing some different keystore.
I am getting below exception on console but it is not failing the replyHandler hence my code is not able to detect the SSL exception.
my code:
eb.request("ping-address", "ping!", new DeliveryOptions(), reply -> {
try {
if (reply.succeeded()) {
System.out.println("Received reply " + reply.result().body());
} else {
System.out.println("An exception " + reply.cause().getMessage());
}
} catch (Exception e) {
System.out.println("An error occured" + e.getCause());
}
});
Exception on console:
**javax.net.ssl.SSLHandshakeException: Failed to create SSL connection**
at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:109)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:341)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:327)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:319)
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1249)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1230)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:813)
Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1329)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
... 20 more
But handler is failing for timeout after 30 sec.
Timed out after waiting 30000(ms) for a reply. address: __vertx.reply.8419a431-d633-4ba8-a12e-c41fd5a4f37a, repliedAddress: ping-address
I want to capture the SSL exception immediately and handle it. Please guide me how can I Capture/catch this exception.
I tried with below code. Below one is able to handle the exception and I am not getting reply result from called event-bus. Reply result is always null. (value is always null)
MessageProducer<Object> ms = eb.sender("ping-address");
ms.write("ping!", reply -> {
if (reply.succeeded()) {
reply.map(value -> {
System.out.println("Received reply " + value);
return reply;
});
} else {
System.out.println("No reply");
System.out.println("An exception : " + reply.cause().getMessage());
}
});
You can't catch this exception because the Vert.x clustered EventBus implementation buffers messages when the nodes are not connected together. The message could be sent later if the problem is only temporary.
If you want to be notified earlier, you could set a lower timeout in DeliveryOptions.

How to handle UndeliverableException error thrown via blockingFirst() in Micronaut?

Below is a snippet from my micronaut web service:
try {
val result = hClient.exchange(GET<String>("$readEndpoint/$token")).blockingFirst()
logger.error("result")
} catch (e: Exception) {
logger.error(e.message)
}
hClient is a reactive http client injected as #Inject val hClient: RxHttpClient
The endpoint is throwing "Connection reset by peer" exception.
Issue I am facing
Even though I have wrapped code in try and catch, An exception io.reactivex.exceptions.UndeliverableException is thrown and not caught.
I basically get two exceptions thrown, one is caught by catch block with message Error occurred reading HTTP response: Connection reset by peer, another one is flowing up to service with message io.reactivex.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.nio.channels.ClosedChannelException
Reproducible via below code
Keep timeout too less to receive a timeout error.
micronaut:
http:
client:
read-timeout: 1s
#Controller("/")
class TokenController(
#Client("https://hello123456789.com/dummy") #Inject val hClient: RxHttpClient
) {
#Get("/test")
fun refresh(): String {
try {
val result = hClient.exchange(HttpRequest.GET<String>("/token/1234")).blockingFirst()
println("result")
} catch (e: Exception) {
println(e.message)
}
return ""
}
}
Googling told me that I need to add global onError to rxjava but couldn't find how to do that in Micronaut.
Any help is appreciated.

http4s client returns partial payload

I'm using the AsyncHttpClient in http4s-0.19.0-M2 to make a client-call:
for {
resp <- http.expectOr[String](GET(url)){ error =>
error.as[String].map(body => throw new Exception(...)
}
_ <- doSomethingWithResponse(resp)
} yield ()
Occassiaonally the remote end times out, and I see the following in the log:
java.util.concurrent.TimeoutException: Request timeout to remote.server.com after 60000 ms
at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43)
at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50)
at shade.cda.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:670)
at shade.cda.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:745)
at shade.cda.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:473)
at shade.cda.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
However, it looks like doSomethingWithResponse() is still invoked, but with a partial resp string. Is there a way to change this behavior so that the http.expectOr call fails if it can't retrieve the entire payload?

Problem calling Request using RequestBuilder

My Code is
String url = "http: gd.geobytes.com/gd?after=-1&variables=GeobytesCountry,GeobytesCity";
RequestBuilder builder = new RequestBuilder(RequestBuilder.GET, URL
.encode(url));
try {
Request request = builder.sendRequest(null, new RequestCallback() {
public void onError(Request request, Throwable exception) {
Couldn't connect to server (could be timeout, SOP
violation, etc.)
}
public void onResponseReceived(Request request,
Response response) {
System.out.println(response.getText() + "Response");
if (200 == response.getStatusCode()) {
Window.alert(response.getText());
} else {
Window.alert(response.getText());
}
}
});
} catch (RequestException e) {
e.printStackTrace();
}
i receive following error
com.google.gwt.http.client.RequestPermissionException: The URL http://gd.geobytes.com/gd?after=-1&variables=GeobytesCountry,GeobytesCity is invalid or violates the same-origin security restriction
at com.google.gwt.http.client.RequestBuilder.doSend(RequestBuilder.java:378)
at com.google.gwt.http.client.RequestBuilder.sendRequest(RequestBuilder.java:254)
at com.ip.client.IpAddressTest.onModuleLoad(IpAddressTest.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.dev.shell.ModuleSpace.onLoad(ModuleSpace.java:369)
at com.google.gwt.dev.shell.OophmSessionHandler.loadModule(OophmSessionHandler.java:185)
at com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:380)
at com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:222)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.google.gwt.http.client.RequestException: (NS_ERROR_DOM_BAD_URI): Access to restricted URI denied
We use GWT for UI which is served by a Coldfusion server and started seeing this error in Hosted Mode.
Changing the Security Level to Medium-Low in the trusted sites zone in IE fixed it for me.
The "Same Origin Policy" is something that browsers implement for the user's security. If you load javascrip code from one web site, that code can't start sending requests to other web sites. It can only send requests to the same site that the code came from.
More details available at GWT docs.
If you are in control of the server that serves the gwt javascript to the browser, you can have some code on that server that sends a request to gd.geobytes.com for you.