Spring Cloud Gateway 500 when an instance is down - spring-cloud

Question

I have a Spring Cloud Gateway (eureka client) app that uses Spring Cloud Load Balancer (Spring Cloud version: Hoxton.SR6) and I have an instance of a spring boot app (spring boot 2.3 with enabled graceful shutdown, (eureka client).
When I shutdown a spring boot service and perform a request through the gateway then the gateway throws 500 error (connection refused), instead of 503. 503 appears after a 1-2 minutes.
Can anyone clarify if it is an expected behavior?
It seems that the problem comes from eureka-client (1.9.21 version in my case)
AtomicReference<Applications> localRegionApps isn't frequently updated
Thanks!
UPDATE:
I decided to check deeper this 500 error. The result is that my system (ubuntu) gives this error if the port is not used:
curl -v localhost:9722
Rebuilt URL to: localhost:9722/
Trying 127.0.0.1...
TCP_NODELAY set
connect to 127.0.0.1 port 9722 failed: Connection refused
Failed to connect to localhost port 9722: Connection refused
Closing connection 0
So I put in my application.yml:
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
Then when my request is routed to my_route and none of apps uses 9722 then I get an error:
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:9722
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/internal/mail/internal/health-check" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
It seems to be an unexpected exception, since it isn't possible to handle it using a circuit breaker or any gateway filter.
Is it possible to handle this error correctly? I would like to return 503 in this case

Answer 1

One of the easiest ways to map particular exception to particular HTTP status code is providing a custom bean of type org.springframework.boot.web.reactive.error.ErrorAttributes. Here is an example:
#Bean
public ErrorAttributes errorAttributes() {
return new CustomErrorAttributes(httpStatusExceptionTypeMapper);
}
public class CustomErrorAttributes extends DefaultErrorAttributes {
#Override
public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
Map<String, Object> attributes = super.getErrorAttributes(request, options);
Throwable error = getError(request);
MergedAnnotation<ResponseStatus> responseStatusAnnotation = MergedAnnotations
.from(error.getClass(), MergedAnnotations.SearchStrategy.TYPE_HIERARCHY).get(ResponseStatus.class);
HttpStatus errorStatus = determineHttpStatus(error, responseStatusAnnotation);
attributes.put("status", errorStatus.value());
return attributes;
}
private HttpStatus determineHttpStatus(Throwable error, MergedAnnotation<ResponseStatus> responseStatusAnnotation) {
if (error instanceof ResponseStatusException) {
return ((ResponseStatusException) error).getStatus();
}
return responseStatusAnnotation.getValue("code", HttpStatus.class).orElseGet(() -> {
if (error instanceof java.net.ConnectException) {
return HttpStatus.SERVICE_UNAVAILABLE;
}
return HttpStatus.INTERNAL_SERVER_ERROR;
}
}
}

Answer 2

hava a try to define your custom ErrorWebExceptionHandler.
see:
org.springframework.boot.web.reactive.error.ErrorWebExceptionHandler
org.springframework.boot.autoconfigure.web.reactive.error.DefaultErrorWebExceptionHandler

Answer 3

You should use Cloud Circuit Breaker.
For that:
Declare corresponding starter in your pom:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
</dependency>
Declare circuit breaker in application.yaml
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
filters:
- name: CircuitBreaker
args:
name: myCircuitBreaker
fallbackUri: forward:/inCaseOfFailureUseThis
Declare the endpoint which will be called in the case of failure (a connection error, for example)
#RequestMapping("/inCaseOfFailureUseThis")
public Mono<ResponseEntity<String>> inCaseOfFailureUseThis() {
return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("body for service failure case"));
}