I've a 3 front-end application and 3 back end application, Let us say 1 Virtual Machine hosts both front-end and back end application as shown in below diagram, Each front-end application connects to back end using discovery client powered by zookeeper.
Now I want to create network affinity or zone such that FE1 connects to BE1 if available, if BE1 is down connect to BE2/BE3. Can this be achieved in spring-cloud-zookeeper?
Though this can be done using eureka, but I would prefer to do it using zookeeper.
EDIT
Ok in eureka we can set the zone field and ribbon can do zone affinity in client based on zone field retrieved from eureka for each server. The issue is in zookeeper though ribbon uses the same zonepreference filter but it since zookeeper does not pass the zone info, it always remains UNKNOWN, hence zone filtering is not applied.
As workaround what I tried is pass zone info as metadata while registering service as shown below.
spring:
application:
name: kp-zk-server
cloud:
zookeeper:
discovery:
metadata:
zone: default
Now in client create ribbon configuration as retrieve the zone info from metadata as filter as shown below.
#Configuration
public class DefaultRibbonConfig {
#Value("${archaius.deployment.zone:default}")
private String zone;
private Predicate<Server> filter = server -> {
if (server instanceof ZookeeperServer) {
ZookeeperServer zkServer = (ZookeeperServer) server;
String str = zkServer.getInstance().getPayload().getMetadata().get("zone");
return zone.equals(str);
}
return true;
};
#Bean
public ServerListFilter<Server> ribbonServerListFilter(IClientConfig config) {
return new ServerListFilter<Server>() {
#Override
public List<Server> getFilteredListOfServers(List<Server> servers) {
List<Server> selected = servers.stream().filter(filter).collect(Collectors.toList());
return selected.isEmpty() ? servers : selected;
}
};
}
}
boostrap.yml
archaius:
deployment:
zone: Zone1
spring:
application:
name: kp-zk-consumer
cloud:
zookeeper:
dependency:
enabled: true
resttemplate:
enabled: false
discovery:
enabled: true
default-health-endpoint: /actuator/health
dependencies:
kWebClient:
path: /kp-zk-server
loadBalancerType: ROUND_ROBIN
required: true
#ribbon:
# NIWSServerListFilterClassName: io.github.kprasad99.zk.KZoneAffinityServerFilter
Problem
Now the problem is my custom filter class is not being enabled/used, ribbon is still using the default zone filter, if I define the configuration using #RibbonClients
#RibbonClients(defaultConfiguration = DefaultRibbonConfig.class)
However, if I declare using ribbon.NIWSServerListFilterClassName the filter is not applied, but in this case I cannot set the zone property, need to hardcode the zone property.
As far as I know this isn't possible with Zookeeper out of the box.
However, you could achieve the same result by using spring-cloud-loadbalancer and a custom ServiceInstanceSupplier which extends DiscoveryClientServiceInstanceSupplier and filters the instances based on given metadata that has been set, or return the complete list of discovered instances if none matched the criteria to provide you some fallback.
This is a generic solution that could solve your question even if you're running in the same datacenter for example.
Hope this helps!
Related
Command side events are getting processed but query (projector) is not invoked.
Using axon kafka extension 4.0-RC2.
Please check below code reference.
AxonConfig
import org.springframework.context.annotation.Configuration;
#Configuration
public class AxonConfig {
}
application.yml
server:
port: 9001
spring:
application:
name: Query Application
datasource:
url: jdbc:postgresql://localhost:5441/orderdemo
username: orderdemo
password: secret
driver-class-name: org.postgresql.Driver
jpa:
properties:
hibernate:
dialect: org.hibernate.dialect.PostgreSQL95Dialect
jdbc:
lob:
non_contextual_creation: true
hbm2ddl.auto: update
implicit_naming_strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
physical_naming_strategy: org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
axon:
eventhandling:
processors:
query:
mode: tracking
source: kafkaMessageSource
kafka:
default-topic: axon-events
consumer:
group-id: query-group
bootstrap-servers: localhost:9092
For this configuration to work, the classes which contain the #EventHandler annotated functions you want to be called for handling the events from Kafka, needs to be part of the processing group query.
This requirement follows from the configuration pattern you've chosen, where "axon. eventhandling.processors.query" defines the Processing Group you want to configure. To specify the Processing Group, I think the easiest approach is to add the #ProcessingGroup annotation to your Event Handling Class. In the annotation, you have to provide the name of the Processing Group, which needs to correspond with what you've set int he configuration file.
Lastly, I would suggest to use a different name than query for your Processing Group. Something more specific to the query model that Event Handler updates would seem more in place to me.
Hope this helps!
I can't figure out if spring-cloud-gateway supports Route reading from consul registry, like it is with Zuul.
I added spring-cloud-starter-consul-discovery dependency and #EnableDiscoveryClient, and configured consul properties in application.yml, hovewer, /actuator/gateway/routes doesn't show any routes from consul
I also tried to set spring.cloud.gateway.discovery.locator.enabled: true but doesn't changed anything.
Sample excample below:
spring:
cloud:
consul:
discovery:
register: false
locator:
enabled: true
acl-token: d3ee84e2-c99a-5d84-e4bf-b2cefd7671ba
enabled: true
so the main question, is it even suppose to work?
EDIT: Probably should have mentioned it is version 2.0.0.M5., with Spring Boot 2.0.0.M7
Also I launched with --debug and there is this line:
GatewayDiscoveryClientAutoConfiguration#discoveryClientRouteDefinitionLocator:
Did not match:
- #ConditionalOnBean (types: org.springframework.cloud.client.discovery.DiscoveryClient; SearchStrategy: all) did not find any beans of type org.springframework.cloud.client.discovery.DiscoveryClient (OnBeanCondition)
Matched:
- #ConditionalOnProperty (spring.cloud.gateway.discovery.locator.enabled) matched (OnPropertyCondition)
I could solve it declaring the following bean: DiscoveryClientRouteDefinitionLocator (reference)
#Configuration
#EnableDiscoveryClient
public class AutoRouting {
#Bean
public DiscoveryClientRouteDefinitionLocator discoveryClientRouteDefinitionLocator(DiscoveryClient discoveryClient, DiscoveryLocatorProperties properties) {
return new DiscoveryClientRouteDefinitionLocator(discoveryClient, properties);
}
}
P.S: You need to include "spring-cloud-consul"
I am currently integrating Spring Cloud Vault Config into a Spring Boot application. From the home page:
Spring Cloud Vault Config reads config properties from Vaults using the application name and active profiles:
/secret/{application}/{profile}
/secret/{application}
/secret/{default-context}/{profile}
/secret/{default-context}
I would like to instead provide my own location from which to pull properties from Vault which does not start with /secret (e.g. /deployments/prod). I've been looking through the reference documentation but I haven't found anyway to specify this -- is it possible?
I was able to use the Generic Backend properties to massage the paths into what I was looking for. Something like:
spring.cloud.vault:
generic:
enabled: true
backend: deployments
profile-separator: '/'
default-context: prod
application-name: my-app
This will also unfortunately pickup Vault locations like deployments/my-app and deployments/prod/activeProfile so be careful not to have any properties in these locations that you don't want to be picked up.
It looks like there is a desire (and an implementation) to allow for these paths to be specified more programmatically.
It should be done this way.
Have a Configuration class
#Configuration
public class VaultConfiguration {
#Bean
public VaultConfigurer configurer() {
return new VaultConfigurer() {
#Override
public void addSecretBackends(SecretBackendConfigurer configurer) {
configurer.add("secret/my-app/path-1");
configurer.add("secret/my-app/path-2");
configurer.registerDefaultGenericSecretBackends(false);
}
};
}
}
This way you can scan your secrets placed in custom path
Regards
Arun
I solved the same problem in my Kotlin project. But it works in Java too.
Problem
I wanted to specify vault paths in yaml config, so i ended up with the following solution, that allows you to specify paths directly in bootstrap.yml using clear syntax, as:
spring:
cloud:
vault:
paths: "secret/your-app"
Solution:
Create VaultConfig class in your project, with the following content:
package com.your.app.configuration
import org.springframework.beans.factory.annotation.Value
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
import org.springframework.cloud.vault.config.VaultConfigurer
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
#Configuration
#ConditionalOnProperty(
prefix = "spring.cloud.vault", value = ["paths"],
matchIfMissing = false
)
class VaultConfig {
#Value("\${spring.cloud.vault.paths}")
private lateinit var paths: List<String>
#Bean
fun configurer(): VaultConfigurer {
return VaultConfigurer { configurer ->
paths.forEach {
configurer.add(it)
}
configurer.registerDefaultGenericSecretBackends(false)
configurer.registerDefaultDiscoveredSecretBackends(false)
}
}
}
Create spring.factories file in src/main/resources/META-INF/spring.factories with a content:
org.springframework.cloud.bootstrap.BootstrapConfiguration=com.your.app.configuration.VaultConfig
Don't forget to specify valid reference to your config instead of
com.your.app.configuration.VaultConfig
spring.factories allows your VaultConfig
happen in the bootstrap context, as documentation says.
Now you can specify desired paths in your bootstrap.yml, as follows:
spring:
cloud:
vault:
paths:
- "secret/application"
- "secret/your-app"
And it should work.
I have a service that has uses 3 feign clients. Each time I start my application, I get a TimeoutException on the first call to any feign client.
I have to trigger each feign client at least once before everything is stable. Looking around online, the problem is that something inside of feign or hystrix is lazy loaded and the solution was to make a configuration class that overrides the spring defaults. I've tried that wiith the below code and it is still not helping. I still see the same issue. Anyone know a fix for this? Is the only solution to call the feignclient twice via a hystrix callback?
#FeignClient(value = "SERVICE-NAME", configuration =ServiceFeignConfiguration.class)
#Configuration
public class ServiceFeignConfiguration {
#Value("${service.feign.connectTimeout:60000}")
private int connectTimeout;
#Value("${service.feign.readTimeOut:60000}")
private int readTimeout;
#Bean
public Request.Options options() {
return new Request.Options(connectTimeout, readTimeout);
}
}
Spring Cloud - Brixton.SR4
Spring Boot - 1.4.0.RELEASE
This is all running in docker
Ubuntu - 12.04
Docker - 1.12.1
Docker-Compose - 1.8
I found the solution to be that the default properties of Hystrix are not good. They have a very small timeout window and the request will always time out on the first try. I added these properties to my application.yml file in my config service and now all of my services can use feign with no problems and i dont have to code around the first time timeout
hystrix:
threadpool.default.coreSize: "20"
threadpool.default.maxQueueSize: "500000"
threadpool.default.keepAliveTimeMinutes: "2"
threadpool.default.queueSizeRejectionThreshold: "500000"
command:
default:
fallback.isolation.semaphore.maxConcurrentRequests: "20"
execution:
timeout:
enabled: "false"
isolation:
strategy: "THREAD"
thread:
timeoutInMilliseconds: "30000"
I suspect this is an issue, can anyone help to have a check?
In my sideCar application, I have application.yml:
server:
port: 5678
spring:
application:
name: nodeservice
sidecar:
port: ${nodeServer.instance.port:3000}
health-uri: http://localhost:${nodeServer.instance.port:3000}/app/health.json
eureka:
instance:
hostname: ${host.instance.name:localhost}
leaseRenewalIntervalInSeconds: 5 #default is 30, recommended to keep default
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
And in my main spring config app, I have:
String url_node = "";
try {
InstanceInfo instance = discoveryClient.getNextServerFromEureka("nodeservice", false);
// InstanceInfo instance = discoveryClient.getNextServerFromEureka("foo", false);
url_node = instance.getHomePageUrl();
} catch (Exception e) {
}
Now I start my nodeJS server, I have in spring app:
url for nodeService is: http://SJCC02MT0NUFD58.local:3000/
This is perfect, but after I shutdown my nodeJS server,
http://localhost:3000/app/health.json url is totally down, BUT, in the main java spring app, I still see the same output there.
So it seemed even if the NodeJS service is no longer available, eureka is still remembering that in memory.
Anything wrong for my configuration?
Another question is why the url being discovered by spring is http://SJCC02MT0NUFD58.local:3000/, not http://localhost:3000? I already configured Eureka.server.instance.host to be localhost.
Thanks
You are seeing the appropriate behavior. Eureka and ribbon are built to be very resilient (AP in CAP). In the case you described, a service had at least one instance, then there were none, the ribbon eureka client keeps the last know list of servers around as a last resort. You're just printing the names, if you try to connect to that service it will fail. This is where you use the Hystrix Circuit Breaker that can provide a fallback in the case that no instances are up.