Spring cloud sidecar can not un-register nodeJS service once it is shut down - spring-cloud

I suspect this is an issue, can anyone help to have a check?
In my sideCar application, I have application.yml:
server:
port: 5678
spring:
application:
name: nodeservice
sidecar:
port: ${nodeServer.instance.port:3000}
health-uri: http://localhost:${nodeServer.instance.port:3000}/app/health.json
eureka:
instance:
hostname: ${host.instance.name:localhost}
leaseRenewalIntervalInSeconds: 5 #default is 30, recommended to keep default
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
And in my main spring config app, I have:
String url_node = "";
try {
InstanceInfo instance = discoveryClient.getNextServerFromEureka("nodeservice", false);
// InstanceInfo instance = discoveryClient.getNextServerFromEureka("foo", false);
url_node = instance.getHomePageUrl();
} catch (Exception e) {
}
Now I start my nodeJS server, I have in spring app:
url for nodeService is: http://SJCC02MT0NUFD58.local:3000/
This is perfect, but after I shutdown my nodeJS server,
http://localhost:3000/app/health.json url is totally down, BUT, in the main java spring app, I still see the same output there.
So it seemed even if the NodeJS service is no longer available, eureka is still remembering that in memory.
Anything wrong for my configuration?
Another question is why the url being discovered by spring is http://SJCC02MT0NUFD58.local:3000/, not http://localhost:3000? I already configured Eureka.server.instance.host to be localhost.
Thanks

You are seeing the appropriate behavior. Eureka and ribbon are built to be very resilient (AP in CAP). In the case you described, a service had at least one instance, then there were none, the ribbon eureka client keeps the last know list of servers around as a last resort. You're just printing the names, if you try to connect to that service it will fail. This is where you use the Hystrix Circuit Breaker that can provide a fallback in the case that no instances are up.

Related

How to configure zone Spring Cloud Zookeeper

I've a 3 front-end application and 3 back end application, Let us say 1 Virtual Machine hosts both front-end and back end application as shown in below diagram, Each front-end application connects to back end using discovery client powered by zookeeper.
Now I want to create network affinity or zone such that FE1 connects to BE1 if available, if BE1 is down connect to BE2/BE3. Can this be achieved in spring-cloud-zookeeper?
Though this can be done using eureka, but I would prefer to do it using zookeeper.
EDIT
Ok in eureka we can set the zone field and ribbon can do zone affinity in client based on zone field retrieved from eureka for each server. The issue is in zookeeper though ribbon uses the same zonepreference filter but it since zookeeper does not pass the zone info, it always remains UNKNOWN, hence zone filtering is not applied.
As workaround what I tried is pass zone info as metadata while registering service as shown below.
spring:
application:
name: kp-zk-server
cloud:
zookeeper:
discovery:
metadata:
zone: default
Now in client create ribbon configuration as retrieve the zone info from metadata as filter as shown below.
#Configuration
public class DefaultRibbonConfig {
#Value("${archaius.deployment.zone:default}")
private String zone;
private Predicate<Server> filter = server -> {
if (server instanceof ZookeeperServer) {
ZookeeperServer zkServer = (ZookeeperServer) server;
String str = zkServer.getInstance().getPayload().getMetadata().get("zone");
return zone.equals(str);
}
return true;
};
#Bean
public ServerListFilter<Server> ribbonServerListFilter(IClientConfig config) {
return new ServerListFilter<Server>() {
#Override
public List<Server> getFilteredListOfServers(List<Server> servers) {
List<Server> selected = servers.stream().filter(filter).collect(Collectors.toList());
return selected.isEmpty() ? servers : selected;
}
};
}
}
boostrap.yml
archaius:
deployment:
zone: Zone1
spring:
application:
name: kp-zk-consumer
cloud:
zookeeper:
dependency:
enabled: true
resttemplate:
enabled: false
discovery:
enabled: true
default-health-endpoint: /actuator/health
dependencies:
kWebClient:
path: /kp-zk-server
loadBalancerType: ROUND_ROBIN
required: true
#ribbon:
# NIWSServerListFilterClassName: io.github.kprasad99.zk.KZoneAffinityServerFilter
Problem
Now the problem is my custom filter class is not being enabled/used, ribbon is still using the default zone filter, if I define the configuration using #RibbonClients
#RibbonClients(defaultConfiguration = DefaultRibbonConfig.class)
However, if I declare using ribbon.NIWSServerListFilterClassName the filter is not applied, but in this case I cannot set the zone property, need to hardcode the zone property.
As far as I know this isn't possible with Zookeeper out of the box.
However, you could achieve the same result by using spring-cloud-loadbalancer and a custom ServiceInstanceSupplier which extends DiscoveryClientServiceInstanceSupplier and filters the instances based on given metadata that has been set, or return the complete list of discovered instances if none matched the criteria to provide you some fallback.
This is a generic solution that could solve your question even if you're running in the same datacenter for example.
Hope this helps!

Spring Cloud - Registry Service port customization

I'd like to customize the Eureka port with Spring Cloud.
With the default port below, the services registry sees itself right (within the provided GUI)
spring:
application:
name: services-registry
server:
port: 8761
eureka:
instance:
hostname: localhost
nonSecurePort: ${server.port}
client:
register-with-eureka: true
fetch-registry: false
service-url:
default-zone: http://${eureka.instance.hostname}:${server.port}/eureka/
But if I just change server.port to 8787, no service can register itself, not even the services registry itself.
2017-01-09 16:18:21.584 WARN 17496 --- [nfoReplicator-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failure
2017-01-09 16:18:21.584 WARN 17496 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_SERVICES-REGISTRY/xxx.org:services-registry:8787 - registration failed Cannot execute request on any known server
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
...
2017-01-09 16:13:33.299 WARN 17496 --- [nfoReplicator-0] c.n.discovery.InstanceInfoReplicator : There was a problem with the instance info replicator
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
Can someone explain this issue and save my day? Thanks!
Ok, got it... the label after service-url property (which can be aliased as serviceUrl in YML) is a HashMap KEY, not a property label. So it has to be kept as a Camel Case tag in any ways!
eureka.client.service-url.defaultZone=http://[myIP#]:8787/eureka

Spring Cloud | Feign Hytrix | First Call Timeout

I have a service that has uses 3 feign clients. Each time I start my application, I get a TimeoutException on the first call to any feign client.
I have to trigger each feign client at least once before everything is stable. Looking around online, the problem is that something inside of feign or hystrix is lazy loaded and the solution was to make a configuration class that overrides the spring defaults. I've tried that wiith the below code and it is still not helping. I still see the same issue. Anyone know a fix for this? Is the only solution to call the feignclient twice via a hystrix callback?
#FeignClient(value = "SERVICE-NAME", configuration =ServiceFeignConfiguration.class)
#Configuration
public class ServiceFeignConfiguration {
#Value("${service.feign.connectTimeout:60000}")
private int connectTimeout;
#Value("${service.feign.readTimeOut:60000}")
private int readTimeout;
#Bean
public Request.Options options() {
return new Request.Options(connectTimeout, readTimeout);
}
}
Spring Cloud - Brixton.SR4
Spring Boot - 1.4.0.RELEASE
This is all running in docker
Ubuntu - 12.04
Docker - 1.12.1
Docker-Compose - 1.8
I found the solution to be that the default properties of Hystrix are not good. They have a very small timeout window and the request will always time out on the first try. I added these properties to my application.yml file in my config service and now all of my services can use feign with no problems and i dont have to code around the first time timeout
hystrix:
threadpool.default.coreSize: "20"
threadpool.default.maxQueueSize: "500000"
threadpool.default.keepAliveTimeMinutes: "2"
threadpool.default.queueSizeRejectionThreshold: "500000"
command:
default:
fallback.isolation.semaphore.maxConcurrentRequests: "20"
execution:
timeout:
enabled: "false"
isolation:
strategy: "THREAD"
thread:
timeoutInMilliseconds: "30000"

setting spring data jedis connection pool using application.yml file

We are using spring data redis. Below mentioned are property for sentinel configuration
spring.redis.sentinel.master: globalsessions_dev
spring.redis.sentinel.nodes: sentinel01.stage.shutterfly.com:26379,sentinel02.stage.shutterfly.com:26379,sentinel03.stage.shutterfly.com:26379
We would like to use connection pool also to be configured in same manner. Spring redis documentation does not provide details of connection pool yml property.
Thanks in advance.
spring:
profiles: live
redis:
sentinel:
master:
nodes:
host: 192.168.1.1000
port: 6379
password:
pool:
max-wait: -1
max-active: -1
max-idle: -1
min-idle: 16
You could also use Spring tool suite, it has nice autocompletion of yaml properties ;)

Zuul timing out in long-ish requests

I am using a front end Spring Cloud application (micro service) acting as a Zuul proxy (#EnableZuulProxy) to route requests from an external source to other internal micro services written using spring cloud (spring boot).
The Zuul server is straight out of the applications in the samples section
#SpringBootApplication
#Controller
#EnableZuulProxy
#EnableDiscoveryClient
public class ZuulServerApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(ZuulServerApplication.class).web(true).run(args);
}
}
I ran this set of services locally and it all seems to work fine but if I run it on a network with some load, or through a VPN, then I start to see Zuul forwarding errors, which I am seeing as client timeouts in the logs.
Is there any way to change the timeout on the Zuul forwards so that I can eliminate this issue from my immediate concerns? What accessible parameter settings are there for this?
In my case I had to change the following property:
zuul.host.socket-timeout-millis=30000
The properties to set are: ribbon.ReadTimeout in general and <service>.ribbon.ReadTimeout for a specific service, in milliseconds. The Ribbon wiki has some examples. This javadoc has the property names.
I have experienced the same problem: in long requests, Zuul's hystrix command kept timing out after around a second in spite of setting ribbon.ReadTimeout=10000.
I solved it by disabling timeouts completely:
hystrix:
command:
default:
execution:
timeout:
enabled: false
An alternative that also works is change Zuul's Hystrix isolation strategy to THREAD:
hystrix:
command:
default:
execution:
isolation:
strategy: THREAD
thread:
timeoutInMilliseconds: 10000
This worked for me, I had to set connection and socket timeout in the application.yml:
zuul:
host:
connect-timeout-millis: 60000 # starting the connection
socket-timeout-millis: 60000 # monitor the continuous incoming data flow
I had to alter two timeouts to force zuul to stop timing out long-running requests. Even if hystrix timeouts are disabled ribbon will still timeout.
hystrix:
command:
default:
execution:
timeout:
enabled: false
ribbon:
ReadTimeout: 100000
ConnectTimeout: 100000
If Zuul uses service discovery, you need to configure these timeouts with the ribbon.ReadTimeout and ribbon.SocketTimeout Ribbon properties.
If you have configured Zuul routes by specifying URLs, you need to use zuul.host.connect-timeout-millis and zuul.host.socket-timeout-millis
by routes i mean
zuul:
routes:
dummy-service:
path: /dummy/**
I had a similar issue and I was trying to set timeout globally, and also sequence of setting timeout for Hystrix and Ribbon matters.
After spending plenty of time, I ended up with this solution. My service was taking upto 50 seconds because of huge volume of data.
Points to consider before changing default value for Timeout:
Hystrix time should be greater than combined time of Ribbon ReadTimeout and ConnectionTimeout.
Use for specific service only, which means don't set globally (which doesn't work).
I mean use this:
command:
your-service-name:
instead of this:
command:
default:
Working solution:
hystrix:
command:
your-service-name:
execution:
isolation:
strategy: THREAD
thread:
timeoutInMilliseconds: 95000
your-service-name:
ribbon:
ConnectTimeout: 30000
ReadTimeout: 60000
MaxTotalHttpConnections: 500
MaxConnectionsPerHost: 100
Reference
Only these settings on application.yml worked for me:
ribbon:
ReadTimeout: 90000
ConnectTimeout: 90000
eureka:
enabled: true
zuul:
host:
max-total-connections: 1000
max-per-route-connections: 100
semaphore:
max-semaphores: 500
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 1000000
Hope it helps someone!