Difference in load balancer behavior - spring-cloud

Question

I noticed that load balancer has different behavior pattern in certain scenario.
When using discovery with load balancer and one of the services is restarted on different port - then if services are running on Spring Boot 2.3.8-RELEASE, other service can find and communicate with restarted service, but if services are running on Spring Boot 2.7.8 it will only work if below setting is set to "true":
spring.cloud.loadbalancer.health-check.refetch-instances=true
I tried to create 3x brand new projects in first test using Spring Boot 2.3.8-RELEASE and in second test Spring Boot 2.7.8 to eliminate any specific setting or configuration or my actual project. I was thinking that maybe there is something specific in my actual project causing difference in behavior, but it turns out that vanilla plain setup has exactly this same problem.
In general my expectation was that setup using Spring Boot 2.3.8-RELEASE will have exactly this same behavior in case of restarting a service on different port as Spring Boot 2.7.8 - but that doesn't seems to be a case.
I noticed that load balancer has different behavior pattern in certain scenario.
When using discovery with load balancer and one of the services is restarted on different port - then if services are running on Spring Boot 2.3.8-RELEASE, other service can find and communicate with restarted service, but if services are running on Spring Boot 2.7.8 it will only work if below setting is set to "true":
spring.cloud.loadbalancer.health-check.refetch-instances=true
I tried to create 3x brand new projects in first test using Spring Boot 2.3.8-RELEASE and in second test Spring Boot 2.7.8 to eliminate any specific setting or configuration or my actual project. I was thinking that maybe there is something specific in my actual project causing difference in behavior, but it turns out that vanilla plain setup has exactly this same problem.
In general my expectation was that setup using Spring Boot 2.3.8-RELEASE will have exactly this same behavior in case of restarting a service on different port as Spring Boot 2.7.8 - but that doesn't seems to be a case.
Here are details of my tests.
My setup is 3x spring boot applications - all in this same version (first test with 2.3.8-RELEASE and second with 2.7.8):
Discovery service - Eureka Server
Test service 1 - "caller" - Eureka Client
Test service 2 - "receiver" - also Eureka Client
The difference in behavior can be reproduce following scenario:
Start Discovery service
Start "caller" application - wait for it to register in Discovery
Start "receiver" application - also wait for it to register in Discovery and also wait for "caller" application to fetch latest record from Discovery
Call GET endpoint in "caller" application which in turn calls GET endpoint in "receiver" application
At this stage everything works fine, "caller" can find and call "receiver"
Change server port in "receiver" application to different value and restart "receiver" application
Again, wait for "receiver" application to register in Discovery and "caller" application to fetch latest record from Discovery
At this stage there is a difference in the way how Spring Boot 2.3.8-RELEASE application stack behaves and 2.7.8
What happens is 2.3.8-RELEASE will work fine after changing port in "receiver" application but 2.7.8 will ONLY be able to communicate with "receiver" if below setting is set to "true":
spring.cloud.loadbalancer.health-check.refetch-instances=true
if it's not (and that is the default setting) the 2.7.8 will not be able to ever communicate with "receiver" unless "receiver" port will be reverted to original value and both applications restarted.
Here are some more significant elements of my configuration for Spring Boot 2.3.8-RELEASE:
Discovery service
Main class:
#SpringBootApplication
#EnableEurekaServer
public class EurekaApplication {
public static void main(String[] args) {
SpringApplication.run(EurekaApplication.class, args);
}
}
application.yml:
spring:
application:
name: eureka
eureka:
client:
registerWithEureka: false
fetchRegistry: false
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.8.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.abednarski79</groupId>
<artifactId>eureka-old</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eureka</name>
<description>Eureka service</description>
<properties>
<java.version>11</java.version>
<spring-cloud.version>Hoxton.SR12</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
Caller service
Rest template configuration:
#Configuration
public class RestTemplateConfig {
#Bean
#LoadBalanced
#Qualifier("loadBalanced Rest Template")
RestTemplate loadBalancedRestTemplate() {
return new RestTemplate();
}
#Bean
#Qualifier("rest Template")
RestTemplate restTemplate() {
return new RestTemplate();
}
}
Controller:
#RestController
public class CallerController
{
private Rest Template restTemplate;
public CallerController (#Qualifier("loadBalancedRestTemplate") RestTemplate restTemplate)
{
this.restTemplate= restTemplate;
}
#GetMapping(value = "/call")
public String call()
{
HttpHeaders headers = new HttpHeaders();
HttpEntity<String> entity = new HttpEntity<String> (headers);
restTemplate.exchange("http://receiver/receive", HttpMethod. GET, entity, String.class);
System.out.println("Called.");
return "success";
}
}
pom.xml - only difference vs discovery service is load balancer is added:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>
application.yml:
spring:
application:
name: caller
cloud:
loadbalancer:
ribbon:
enabled: false
eureka:
instance:
prefer-ip-address: true
client:
healthcheck:
enabled: true
serviceUrl:
defaultZone: http://localhost:8001/eureka/
Receiver application
Controller:
#RestController
public class ReceiverController
{
#GetMapping(value = "/receive")
public String receive()
{
System.out.println("Received.");
return "success";
}
}
application.yml:
spring:
application:
name: receiver
eureka:
instance:
prefer-ip-address: true
client:
healthcheck:
enabled: true
serviceUrl:
defaultZone: http://localhost:8001/eureka/
And here are some more significant elements of my configuration for Spring Boot 2.7.8: Discovery service
pom.xml:
<parent>
<groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.8</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.abednarski79</groupId>
<artifactId>eureka</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eureka</name>
<description>Eureka service</description>
<properties>
<java.version>11</java.version>
<spring-cloud.version>2021.0.5</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
Caller service
pom.xml:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.8</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.abednarski79</groupId>
<artifactId>caller</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>caller</name>
<description>Caller service</description>
<properties>
<java.version>11</java.version>
<spring-cloud.version>2021.0.5</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
Rest template configuration:
#Configuration
#LoadBalancerClient(value = "receiver", configuration = LoadBalancerConfig.class)
public class RestTemplateConfig {
#Bean
#LoadBalanced
#Qualifier("loadBalanced RestTemplate")
RestTemplate loadBalanced RestTemplate() {
return new RestTemplate();
}
#Bean
#Qualifier("rest Template")
Rest Template restTemplate() {
return new RestTemplate();
}
and load balancer configuration:
public class LoadBalancerConfig
{
#Bean
ServiceInstanceListSupplier serviceInstanceListSupplier(ConfigurableApplicationContext context, #Qualifier ("restTemplate") RestTemplate restTemplate) {
return ServiceInstanceListSupplier
.builder()
.withBlockingDiscoveryClient()
.withBlockingHealthChecks(restTemplate)
.build(context);
}
}
application.yml:
spring:
application:
name: caller
cloud:
loadbalancer:
health-check:
refetch-instances: false
eureka:
instance:
prefer-ip-address: true
client:
healthcheck:
enabled: true
serviceUrl:
defaultZone: http://localhost:9001/eureka/