When deploying Spring Cloud-enabled microservices on JBoss, we have to use a contextPath. Turbine doesn't seem to fetch the contextPath from eureka and eureka doesn't seem to ask it to the microservices.
Some more details:
Every microservice has its own Jboss, and its own contextPath defined in the war, as such:
hostname1:port1/products
hostname2:port2/users
hostname3:port3/orders
Endpoints (actuator and our own) are exposed after the contextPath (not using management.context-path):
hostname1:port1/products/env
hostname1:port1/products/info
hostname1:port1/products/books
hostname1:port1/products/books/123
Eureka only cares about the hostname and port. It allows us to define a different UrlPath to info and health although this is not part of the data Turbine needs to get to hystrix.stream:
eureka:
instance:
statusPageUrlPath: ${server.contextPath}/info
healthCheckUrlPath: ${server.contextPath}/health
Is there a way to tell Turbine which contextPath is used for which application? These are different per application so a cluster-wide fix described here wouldn't be sufficient I suppose.
You can only view one cluster at a time in normal turbine. With spring-cloud-netflix-hystrix-amqp we aggregate all statistics via rabbitmq (not http), so you can see everything at once.
Related
Context
I am deploying a set of services that are containerised using Docker into AWS. No matter which deployment solution is chosen (e.g. raw EC2/ECS/Elastic Beanstalk/Fargate) we will face the issue of "service discovery".
To name just a few of the options for service discovery that I've considered:
AWS Route 53 Service Registry
Kubernetes
Hashicorp Consul
Spring Cloud Netflix Eureka
Specifics Of My Stack
I am developing Java Spring Boot applications using Spring Cloud with the target deployment environment being AWS.
Given that my stack is Spring based, spring cloud eureka made sense to me while developing locally. It was easy to set up a single node, integrates well with the stack and ecosystem of choice and required very little set up.
Locally, we are using docker compose (not swarm) to deploy services - one of the containers deployed is a single node Eureka service discovery server.
However, when we progress outside of local development and into staging or production environment we are considering options like Kubernetes.
My Own Assessment Of Pros/Cons
AWS Route 53 Service Registry
Requires us to couple code specifically to AWS services. Not a problem per se, we are quite tied in anyway on other parts of the stack (SNS/SQS).
Makes running the stack locally slightly more difficult as it relies on Route 53, I suppose we could open up a certain hosted zone for local development.
AWS native, no managing service registries or extra "moving parts".
Spring Cloud Eureka
Downside is that thus requires us to deploy and manage a high availability service registry cluster and requires more resources. Another "moving part" to manage.
Advantages are that it fits into our stack well (spring ecosystem, spring boot, spring cloud, feign and zuul work well with this). Also can be run locally trivially.
I presume we need to configure the networks and registry zone to ensure that that clients publish their host address rather and docker container internal IP address. e.g. if service A is on host A and wants to talk to service B on host B, service B needs to advertise its EC2 address rather than some internal docker IP.
Questions
If we use Kubernetes for orchestration, are there any disadvantages to using something like Spring Cloud Eureka over the built in service discovery options described here https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services
Given Kube provides this, it seems suboptimal to then use eureka deployed using kube to perform discovery. I presume kube can make some optimisations that impact avaialbility and stability that might nit be possible using eureka. e.g kube would know when deploying a new service - eureka will have to rely on heartbeats/health checks and depending on how that is configured (e.g. frequency) this could result in stale records whereas i presume kube might not suffer from this for planned service shutdown/restarts. I guess it still does for unplanned failures such as a host failure or network partition.
Does anyone have any advice on this, do people use services like Kubernetes but use other mechanisms for service discovery rather than those provided by kube. Is there a good reason to do one or the other?
Possible Challenges I Anticipate
We could replace eureka, but relying on Kube to perform discovery will mean that we need to run kube locally to deploy whereas currently we have a simple tiny docker-compose file. Also, I'll have to look at how easy it'll be to ensure that ribbon, zuul and feign play nicely with this.
Currently we have ribbon configured with a eureka client so that service A can server to service B just as "service-b" for example and have ribbon resolve a healthy host via a eureka client. I guess we can configure ribbon to not use eureka and use an external Kube service name which will be resolved by Kube DNS at runtime...
Final Note
Thanks in advance for any contribution or advice. I know this might elicit a primarily opinion focused response. But I am hoping someone can provide objective guidance on when one solution might be preferable to another.
Service discovery is something you get out-of-the-box with Kubernetes. So having another external service in your platform will be another application to maintain, deploy and can be a point of failure. So I would stick with the the service discovery provided by Kubernetes.
Recently I started working with Microservices, I wrote a library for service discovery using Redis to store every service's url and port number, along with a TTL value for the entry. It turned out to be an expensive approach since for every cross service call to any other service required one call to Redis. Caching didn't seem to be a good idea, since the services won't be up all the times, there can be possible downtimes as well.
So I wanted to write a separate microservice which could take care of the orchestration part. For this I need to figure out a really low level network protocol to take care of the exchange of heartbeats(which would help me figure out if any of the service instance goes unavailable). How do applications like zookeeperClient, redisClient take care of heartbeats?
Moreover what is the industry's preferred protocol for cross service calls?
I have been calling REST Api's over HTTP and eliminated every possibility of Joins across different collections.
Is there a better way to do this?
Thanks.
I think the term "Orchestration" is not good for what you are asking. From what I've encountered so far in microservices world the term "Orchestration" is used when a complex business process is involved and not for service discovery. What you need is a Service registry combined with a Load balancer. You can find here all the information you need. Here are some relevant extras that great article:
There are two main service discovery patterns: client‑side discovery and server‑side discovery. Let’s first look at client‑side discovery.
The Client‑Side Discovery Pattern
When using client‑side discovery, the client is responsible for determining the network locations of available service instances and load balancing requests across them. The client queries a service registry, which is a database of available service instances. The client then uses a load‑balancing algorithm to select one of the available service instances and makes a request.
The network location of a service instance is registered with the service registry when it starts up. It is removed from the service registry when the instance terminates. The service instance’s registration is typically refreshed periodically using a heartbeat mechanism.
Netflix OSS provides a great example of the client‑side discovery pattern. Netflix Eureka is a service registry. It provides a REST API for managing service‑instance registration and for querying available instances. Netflix Ribbon is an IPC client that works with Eureka to load balance requests across the available service instances. We will discuss Eureka in more depth later in this article.
The client‑side discovery pattern has a variety of benefits and drawbacks. This pattern is relatively straightforward and, except for the service registry, there are no other moving parts. Also, since the client knows about the available services instances, it can make intelligent, application‑specific load‑balancing decisions such as using hashing consistently. One significant drawback of this pattern is that it couples the client with the service registry. You must implement client‑side service discovery logic for each programming language and framework used by your service clients.
The Server‑Side Discovery Pattern
The client makes a request to a service via a load balancer. The load balancer queries the service registry and routes each request to an available service instance. As with client‑side discovery, service instances are registered and deregistered with the service registry.
The AWS Elastic Load Balancer (ELB) is an example of a server-side discovery router. An ELB is commonly used to load balance external traffic from the Internet. However, you can also use an ELB to load balance traffic that is internal to a virtual private cloud (VPC). A client makes requests (HTTP or TCP) via the ELB using its DNS name. The ELB load balances the traffic among a set of registered Elastic Compute Cloud (EC2) instances or EC2 Container Service (ECS) containers. There isn’t a separate service registry. Instead, EC2 instances and ECS containers are registered with the ELB itself.
HTTP servers and load balancers such as NGINX Plus and NGINX can also be used as a server-side discovery load balancer. For example, this blog post describes using Consul Template to dynamically reconfigure NGINX reverse proxying. Consul Template is a tool that periodically regenerates arbitrary configuration files from configuration data stored in the Consul service registry. It runs an arbitrary shell command whenever the files change. In the example described by the blog post, Consul Template generates an nginx.conf file, which configures the reverse proxying, and then runs a command that tells NGINX to reload the configuration. A more sophisticated implementation could dynamically reconfigure NGINX Plus using either its HTTP API or DNS.
Some deployment environments such as Kubernetes and Marathon run a proxy on each host in the cluster. The proxy plays the role of a server‑side discovery load balancer. In order to make a request to a service, a client routes the request via the proxy using the host’s IP address and the service’s assigned port. The proxy then transparently forwards the request to an available service instance running somewhere in the cluster.
The server‑side discovery pattern has several benefits and drawbacks. One great benefit of this pattern is that details of discovery are abstracted away from the client. Clients simply make requests to the load balancer. This eliminates the need to implement discovery logic for each programming language and framework used by your service clients. Also, as mentioned above, some deployment environments provide this functionality for free. This pattern also has some drawbacks, however. Unless the load balancer is provided by the deployment environment, it is yet another highly available system component that you need to set up and manage.
As part of the move from Eureka to consul (Spring Cloud), we are looking at replicating certain behaviours.
One of them is that if a service disconnected to Eureka after a significant will reconnect when it can see Eureka again.
I see that a service will retry to consul and eventually stop. Is there a way to replicate the same behaviour on a service to consul past the max retries/backoff?
Just trying to support expected behaviours
It's a known issue https://github.com/spring-cloud/spring-cloud-consul/issues/197. That functionality you describe is built into the eureka thick client and something similar would need to be built for spring-cloud-consul.
I am working with microservices and netflix oss. I am using Eureka for service registry and discovery. I am trying out different options to understand behaviour of Eureka server. I still have some doubts with respect to below mentioned scenario.
I have couple of microservices, Service-A and Service-B. I also have two Eureka server instances viz. Eureka-1(port:8761) and Eureka-2(port:8762). I have registered both services with only Eureka-1. Also, service-A is invoking Service-B using eureka service instance name.
When I run Service-A, Service-B, Eureka-1 and Eureka-2 services, I am able to hit Service-A which internally hitting Service-B as expected. However, when I take Eureka-1 instance down and hit service-A, it still able to hit Service-B internally using eureka service instance name. My question is does Eureka-2 instance come to know about services registered with Eureka-1 instance. If yes, how and when does this communication happens between Eureka server instances?? Please help me understand this.
There is a good write up on the Netflix site (https://github.com/Netflix/eureka/wiki/Understanding-Eureka-Peer-to-Peer-Communication) that explains how eureka peer to peer happens.
If you have both eureka-1 and eureka-2 pointing at each other (i.e. their eureka client definition in the yml files), then when you register to eureka-1, eureka-2 will get the list of available services from eureka-1.
So once paired, eureka-2 will have the same list as eureka-2 and discovery will work.
You should probably put your eureka-1 and 2 configuration in your question so we have more detail
In some applications, I saw people are using #EnableEurekaClient. And some other example applications are using #EnableDiscoveryClient.
Is there any difference between these two?
There are multiple implementations of "Discovery Service" (eureka, consul, zookeeper). #EnableDiscoveryClient lives in spring-cloud-commons and picks the implementation on the classpath. #EnableEurekaClient lives in spring-cloud-netflix and only works for eureka. If eureka is on your classpath, they are effectively the same.
Discovery service concept in spring cloud is implemented in different ways like Eureka, consul, zookeeper etc. If you are using Eureka by Netflix then #EnableEurekaClient is specifically for that. But if you are using any other service discovery including Eureka you can use #EnableDiscoveryClient.
In terms of its practicality and simplicity, if the registered center is eureka, then #EnableEurekaClient is recommended. If it is another registration center, #EnableDiscoveryClient is recommended.
#EnableEurekaCient will initilize the service discovery using eureka server, its like explicitly specifying to use EUREKA
While #EnableDiscovery Client will register dicovery service using the jar available in classpath like consul, Eureka, Kubernetes.
#EnableEurekaClient works only for Eureka if we use #EnableDiscoveryClient we can use any discovery client like Consul other than Eureka