I'm using Spring Cloud gateway and trying to use the circuit breaker. In the gateway I have added a fallback(failover-service) method for the service(cricketer-service) in the gateway(cricketer-gateway).
The failover service works fine individually. Failover service is named as cricketer-failover-service in application.properties.
Gateway config
#Configuration
public class RoutesConfig {
#Bean
public RouteLocator routes(RouteLocatorBuilder builder) {
return builder.routes()
.route(r -> r.path("/cricketers*")
.filters(f -> f.circuitBreaker(c -> c.setName("cricketerFb")
.setFallbackUri("forward:/cricketer-failover")
.setRouteId("cricketer-failover")
))
.uri("lb://cricketer-service")
.id("cricketer-service"))
.route(r -> r.path("/cricketer-failover*")
.uri("lb://cricketer-failover-service")
.id("cricketer-failover-service"))
.build();
}
When I turn the service off and then hit the endpoint, I receive an exception
2020-08-30 14:17:28.670 ERROR 14766 --- [or-http-epoll-3] com.netflix.hystrix.AbstractCommand : Unrecoverable Error for HystrixCommand so will throw HystrixRuntimeException and not apply fallback.
java.lang.Exception: Throwable caught while executing.
at com.netflix.hystrix.AbstractCommand.getExceptionFromThrowable(AbstractCommand.java:1976) ~[hystrix-core-1.5.18.jar:1.5.18]
Stacktrace - https://gist.github.com/nirmalks/4adb8b64955625df6b5a22469f106bfe
Code - https://github.com/nirmalks/cricketer-microservice/tree/hystrix-fallback-err
Related
I have a Spring Cloud Gateway (eureka client) app that uses Spring Cloud Load Balancer (Spring Cloud version: Hoxton.SR6) and I have an instance of a spring boot app (spring boot 2.3 with enabled graceful shutdown, (eureka client).
When I shutdown a spring boot service and perform a request through the gateway then the gateway throws 500 error (connection refused), instead of 503. 503 appears after a 1-2 minutes.
Can anyone clarify if it is an expected behavior?
It seems that the problem comes from eureka-client (1.9.21 version in my case)
AtomicReference<Applications> localRegionApps isn't frequently updated
Thanks!
UPDATE:
I decided to check deeper this 500 error. The result is that my system (ubuntu) gives this error if the port is not used:
curl -v localhost:9722
Rebuilt URL to: localhost:9722/
Trying 127.0.0.1...
TCP_NODELAY set
connect to 127.0.0.1 port 9722 failed: Connection refused
Failed to connect to localhost port 9722: Connection refused
Closing connection 0
So I put in my application.yml:
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
Then when my request is routed to my_route and none of apps uses 9722 then I get an error:
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:9722
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/internal/mail/internal/health-check" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
It seems to be an unexpected exception, since it isn't possible to handle it using a circuit breaker or any gateway filter.
Is it possible to handle this error correctly? I would like to return 503 in this case
One of the easiest ways to map particular exception to particular HTTP status code is providing a custom bean of type org.springframework.boot.web.reactive.error.ErrorAttributes. Here is an example:
#Bean
public ErrorAttributes errorAttributes() {
return new CustomErrorAttributes(httpStatusExceptionTypeMapper);
}
public class CustomErrorAttributes extends DefaultErrorAttributes {
#Override
public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
Map<String, Object> attributes = super.getErrorAttributes(request, options);
Throwable error = getError(request);
MergedAnnotation<ResponseStatus> responseStatusAnnotation = MergedAnnotations
.from(error.getClass(), MergedAnnotations.SearchStrategy.TYPE_HIERARCHY).get(ResponseStatus.class);
HttpStatus errorStatus = determineHttpStatus(error, responseStatusAnnotation);
attributes.put("status", errorStatus.value());
return attributes;
}
private HttpStatus determineHttpStatus(Throwable error, MergedAnnotation<ResponseStatus> responseStatusAnnotation) {
if (error instanceof ResponseStatusException) {
return ((ResponseStatusException) error).getStatus();
}
return responseStatusAnnotation.getValue("code", HttpStatus.class).orElseGet(() -> {
if (error instanceof java.net.ConnectException) {
return HttpStatus.SERVICE_UNAVAILABLE;
}
return HttpStatus.INTERNAL_SERVER_ERROR;
}
}
}
hava a try to define your custom ErrorWebExceptionHandler.
see:
org.springframework.boot.web.reactive.error.ErrorWebExceptionHandler
org.springframework.boot.autoconfigure.web.reactive.error.DefaultErrorWebExceptionHandler
You should use Cloud Circuit Breaker.
For that:
Declare corresponding starter in your pom:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
</dependency>
Declare circuit breaker in application.yaml
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
filters:
- name: CircuitBreaker
args:
name: myCircuitBreaker
fallbackUri: forward:/inCaseOfFailureUseThis
Declare the endpoint which will be called in the case of failure (a connection error, for example)
#RequestMapping("/inCaseOfFailureUseThis")
public Mono<ResponseEntity<String>> inCaseOfFailureUseThis() {
return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("body for service failure case"));
}
I am trying to setup an integration test env for debezium integration (following the instructions in this example) but the test container (default image: confluentinc/cp-kafka:5.2.1) doesn't start but throws an exception.
I am using below mentioned code to create a KafkaContainer bean
#Bean
public KafkaContainer kafkaContainer() {
if (kafkaContainer == null) {
kafkaContainer = new KafkaContainer()
.withNetwork(network())
.withExternalZookeeper("172.17.0.2:2181");
kafkaContainer.start();
}
}
return kafkaContainer;
}
it throws following exception.
***************************
APPLICATION FAILED TO START
***************************
Description:
An attempt was made to call a method that does not exist. The attempt was made from the following location:
org.testcontainers.containers.KafkaContainer.getBootstrapServers(KafkaContainer.java:91)
The following method did not exist:
org/testcontainers/containers/KafkaContainer.getHost()Ljava/lang/String;
The method's class, org.testcontainers.containers.KafkaContainer, is available from the following locations:
jar:file:/home/shubham/.m2/repository/org/testcontainers/kafka/1.14.3/kafka-1.14.3.jar!/org/testcontainers/containers/KafkaContainer.class
The class hierarchy was loaded from the following locations:
org.testcontainers.containers.KafkaContainer: file:/home/shubham/.m2/repository/org/testcontainers/kafka/1.14.3/kafka-1.14.3.jar
org.testcontainers.containers.GenericContainer: file:/home/shubham/.m2/repository/org/testcontainers/testcontainers/1.12.5/testcontainers-1.12.5.jar
org.testcontainers.containers.FailureDetectingExternalResource: file:/home/shubham/.m2/repository/org/testcontainers/testcontainers/1.12.5/testcontainers-1.12.5.jar
Action:
Correct the classpath of your application so that it contains a single, compatible version of org.testcontainers.containers.KafkaContainer
2020-09-10 01:09:49.937 ERROR 72507 --- [ main] o.s.test.context.TestContextManager : Caught exception while allowing TestExecutionListener
Used an older version of org.testcontainers in maven dependencies and it worked. Thanks!
Im trying to connect to Kafka with JMS. I followed this guide to use the Payara Kafka Connector. This worked on Wildfly. But I cant get it to work on OpenLiberty.
The server.xml:
<resourceAdapter id="kafkajmsra" location="${shared.resource.dir}kafka-rar-0.5.0.rar"/>
<jmsTopicConnectionFactory jndiName="JMSTopicFactory">
<properties.kafkajmsra
bootstrapServerConfig="kafka:9092"/>
</jmsTopicConnectionFactory>
<jmsTopic id="kafkaTopic" jndiName="JmsTopic">
<properties.kafkajmsra topicName="demoTopic" />
</jmsTopic>
With those configurations I get a NullPointerException if I try to inject those components. The JNDI names can be found but not with these parameters.
#Resource(lookup = "JMSTopicFactory")
private TopicConnectionFactory jmsTopicFactory;
#Resource(lookup = "JMSTopic")
private Topic jmsTopic;
Am I missing something in the server.xml?
I tried using the default JMS Connector. It does connect to Kafka, but the connection gets refused and on the kafka side it tells me this:
[2020-05-31 20:05:27,134] WARN [SocketServer brokerId=1] Unexpected error from /172.20.0.4; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = -1091633152)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:103)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:448)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:398)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:893)
at kafka.network.Processor.run(SocketServer.scala:792)
at java.lang.Thread.run(Thread.java:748)
EDIT:
I changed the server.xml to look like this now:
<resourceAdapter id="kafkajmsra" location="${shared.resource.dir}/kafka-rar-0.4.0.rar"/>
<connectionFactory jndi="java:app/KafkaConnectionFactory"
interfaceName="fish.payara.cloud.connectors.kafka.api.KafkaConnectionFactory"
resourceAdapter="liberty/wlp/usr/shared/resources/kafka-rar-0.4.0.rar">
</connectionFactory>
and the java code looks like this:
#ApplicationScoped
public class TopicProducer {
private static final Logger LOG = LoggerFactory.getLogger(TopicProducer.class);
public TopicProducer() throws Exception {
LOG.info("Starting TopicProducer");
}
#Resource(lookup = "java:app/KafkaConnectionFactory")
KafkaConnectionFactory kafkaConnectionFactory;
public void send(final String msg) {
try (KafkaConnection connection = kafkaConnectionFactory.createConnection()) {
LOG.info("Send message: {}", msg);
connection.send(new ProducerRecord("demoTopic", msg));
} catch (Exception e) {
LOG.error(e.getMessage(), e);
}
}
}
But now I get a NullPointerException on the #Resource. My guess is that the resource adapter cannot be found.
Trying to inject an EJB into another one using #EJB annotation :
1) The first EJB "TestEJB" has two interfaces (AdditionRemote and AdditionLocal)
and a class implementation (Addition)
2) The second EJB "TestEJB2" has two interfaces (DivisionRemote and DivisionLocal) and a class implementation (Division)
here is the code of the "TestEJB" where i want to inject a reference of TestEJB2 :
#EJB
private Division d;
public Addition() {
// TODO Auto-generated constructor stub
}
#Override
public int add(int a, int b) {
return a+b;
}
#Override
public void call() {
// TODO Auto-generated method stub
d.div(4, 2);
}
However when i use the #EJB annotation in the code i cannot deploy TestEJB anymore.
i get this error msg :
[org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("full-replace-deployment") failed - address: ([]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"TestEJB.jar\".POST_MODULE" => "WFLYSRV0153: Failed to process phase POST_MODULE of deployment \"TestEJB.jar\"
Caused by: java.lang.RuntimeException: WFLYSRV0177: Error getting reflective information for class logic.Addition with ClassLoader ModuleClassLoader for Module \"deployment.TestEJB.jar\" from Service Module Loader
Caused by: java.lang.NoClassDefFoundError: Labsurd/Division;
Caused by: java.lang.ClassNotFoundException: absurd.Division from [Module \"deployment.TestEJB.jar\" from Service Module Loader]"}}
18:45:39,399 INFO [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "TestEJB.jar" with deployment "TestEJB.jar"
18:45:39,400 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 1) WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service jboss.deployment.unit."TestEJB.jar".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "TestEJB.jar"
you must inject one of the interfaces or add #localbean (for no-interface view) to Division
#EJB
private DivisionLocal d;
or
#EJB
private DivisionRemote d;
because the proxy is created for interfaces(when you implement interface the no-interface view will not be created for session bean unless you annotate it with #LocalBean)
I have config the spring cloud gateway hystrix as below:
default-filters:
- name: Hystrix
args:
name: defaultGatewayCommand
fallbackUri: forward:/hystrix-fallback
The problem is, for example, when I throw ResponseStatusException exception in my project, the hystrix fallback will be triggered.
#GetMapping("/hystrix-fallback")
public Mono<ApiErrorResponse> hystrixFallback() {
return Mono.just(new ApiErrorResponse("xxxxxxxx"));
}
I want to catch some error info and build my new ApiErrorResponse, but i don 't how to do it?
#GetMapping("/hystrix-fallback")
public Mono hystrixFallback(ServerWebExchange exchange) {
Exception exception = exchange.getAttribute(ServerWebExchangeUtils.HYSTRIX_EXECUTION_EXCEPTION_ATTR);
return Mono.just(new ApiErrorResponse("xxxxxxxx"));
}