If I have an ECS service, which can scale out, or always runs more than one task, is load balancing handled by the built-in service discovery?
I mean, if my service A is running 3 tasks, and another service B is making requests using the domain name, generated for A by service discovery, how will it decide where a request goes?
From this Medium article, it appears that route 53 is going to use its own catalog of healthy instances to determine which service to route to. So yes, load balancing will occur in that way.
Related
According to AWS documentation:
You can configure Service Discovery for an ECS Service that is behind
a Load Balancer, but Service Discovery traffic is always routed to the
Task and not the Load Balancer.
If this the case, how does the load balancing happens here?
Also, without the Load Balancer, how does the service discovery works, will the traffic routed to a random Container Instances?
TL;DR Yes, the traffic will be sent to random instances.
When you use ECS Service Discovery, you have two options for discovering your services. One is via Route 53 DNS, which in case of ECS Service Discovery leverages Multivalue Routing Policy, so that your client application receives up to eight healthy endpoints, selected at random.
The other option is to use Cloud Map DiscoverInstances API, which returns up to 100 endpoints for a given service name, selected at random.
Can someone explain the load balancing mechanism in AWS ECS for me? I clearly understand how inter service communication is handled within a kubernetes cluster, there is an automatic load balancer applied when accessing a defined internal service. This means Container/Pod scalability is simply predefined:
when a Pod-1A from within the service-A is accessing another Pod-1B
from within a different Service-B (Service to Service communication)
this call is automatically load balanced to this Pod-1B from
service-B.
So with service Registry in kubernetes we simply need to define Services and communication is automatically load balanced to the available Pods within the services.
So assuming that Pods are equal to Tasks and Services are equal to Services in AWS ECS, how is this load balancing mechanism handled wihtin ECS? Do we really need to apply an Elastic Load balancer at the task/pod level manually compared to kubernetes? (So that we need to define manually a load balancer for every service, to make this service and its tasks with its container scalable?)
Edit:
What is the reason in AWS ECS, to define a service which instantiates
multiple replicas of a Task, when no load balancer has been defined?
Will the traffic be routed only to the same Task replica (Container)
all the time? (No scaling at all?)
Please note, this is not about access from external ip addresses, where an ingress controller is needed. I am talking about microservices where each service exposes its own http api to communicate with other services within the cluster (internal microservice Application), typically there is an API Gateway handling external traffic (ingress controller).
Recently I started working with Microservices, I wrote a library for service discovery using Redis to store every service's url and port number, along with a TTL value for the entry. It turned out to be an expensive approach since for every cross service call to any other service required one call to Redis. Caching didn't seem to be a good idea, since the services won't be up all the times, there can be possible downtimes as well.
So I wanted to write a separate microservice which could take care of the orchestration part. For this I need to figure out a really low level network protocol to take care of the exchange of heartbeats(which would help me figure out if any of the service instance goes unavailable). How do applications like zookeeperClient, redisClient take care of heartbeats?
Moreover what is the industry's preferred protocol for cross service calls?
I have been calling REST Api's over HTTP and eliminated every possibility of Joins across different collections.
Is there a better way to do this?
Thanks.
I think the term "Orchestration" is not good for what you are asking. From what I've encountered so far in microservices world the term "Orchestration" is used when a complex business process is involved and not for service discovery. What you need is a Service registry combined with a Load balancer. You can find here all the information you need. Here are some relevant extras that great article:
There are two main service discovery patterns: client‑side discovery and server‑side discovery. Let’s first look at client‑side discovery.
The Client‑Side Discovery Pattern
When using client‑side discovery, the client is responsible for determining the network locations of available service instances and load balancing requests across them. The client queries a service registry, which is a database of available service instances. The client then uses a load‑balancing algorithm to select one of the available service instances and makes a request.
The network location of a service instance is registered with the service registry when it starts up. It is removed from the service registry when the instance terminates. The service instance’s registration is typically refreshed periodically using a heartbeat mechanism.
Netflix OSS provides a great example of the client‑side discovery pattern. Netflix Eureka is a service registry. It provides a REST API for managing service‑instance registration and for querying available instances. Netflix Ribbon is an IPC client that works with Eureka to load balance requests across the available service instances. We will discuss Eureka in more depth later in this article.
The client‑side discovery pattern has a variety of benefits and drawbacks. This pattern is relatively straightforward and, except for the service registry, there are no other moving parts. Also, since the client knows about the available services instances, it can make intelligent, application‑specific load‑balancing decisions such as using hashing consistently. One significant drawback of this pattern is that it couples the client with the service registry. You must implement client‑side service discovery logic for each programming language and framework used by your service clients.
The Server‑Side Discovery Pattern
The client makes a request to a service via a load balancer. The load balancer queries the service registry and routes each request to an available service instance. As with client‑side discovery, service instances are registered and deregistered with the service registry.
The AWS Elastic Load Balancer (ELB) is an example of a server-side discovery router. An ELB is commonly used to load balance external traffic from the Internet. However, you can also use an ELB to load balance traffic that is internal to a virtual private cloud (VPC). A client makes requests (HTTP or TCP) via the ELB using its DNS name. The ELB load balances the traffic among a set of registered Elastic Compute Cloud (EC2) instances or EC2 Container Service (ECS) containers. There isn’t a separate service registry. Instead, EC2 instances and ECS containers are registered with the ELB itself.
HTTP servers and load balancers such as NGINX Plus and NGINX can also be used as a server-side discovery load balancer. For example, this blog post describes using Consul Template to dynamically reconfigure NGINX reverse proxying. Consul Template is a tool that periodically regenerates arbitrary configuration files from configuration data stored in the Consul service registry. It runs an arbitrary shell command whenever the files change. In the example described by the blog post, Consul Template generates an nginx.conf file, which configures the reverse proxying, and then runs a command that tells NGINX to reload the configuration. A more sophisticated implementation could dynamically reconfigure NGINX Plus using either its HTTP API or DNS.
Some deployment environments such as Kubernetes and Marathon run a proxy on each host in the cluster. The proxy plays the role of a server‑side discovery load balancer. In order to make a request to a service, a client routes the request via the proxy using the host’s IP address and the service’s assigned port. The proxy then transparently forwards the request to an available service instance running somewhere in the cluster.
The server‑side discovery pattern has several benefits and drawbacks. One great benefit of this pattern is that details of discovery are abstracted away from the client. Clients simply make requests to the load balancer. This eliminates the need to implement discovery logic for each programming language and framework used by your service clients. Also, as mentioned above, some deployment environments provide this functionality for free. This pattern also has some drawbacks, however. Unless the load balancer is provided by the deployment environment, it is yet another highly available system component that you need to set up and manage.
say, I have a service fabric cluster with one node type which has 5 nodes, and deployed two stateless services on those 5 nodes, but I would deploy service_1 on 3 nodes and service_2 on 2 other nodes.
I understand that a node type is actually a VMSS and azure will create a load balancer on top of that VMSS. It works perfectly on REST stateless service which spread to all of nodes of certain node type.
But in my case, the service deployed on part of nodes, can I still leverage the load balancer to route to instances of two separate services?
You have multiple options:
Use the built in HTTP gateway service from Service Fabric. Unfortunately, I haven't found any documentation about this yet, so I don't know about the advantages and disadvantages of this solution. See this comment in the service-fabric-issues project for some details.
Implement your own stateless gateway service which has an InstanceCount of -1 (which means it gets placed on every node). This service will act like an internal load balancer and route every request to the correct service. See weidazhao's (a Microsoft employee - this project might become part of the SDK) or our project for existing gateway projects.
Keep using one load balancer which points to the whole scale set and use probe methods to let the load balancer disable nodes on which the service is not running. However, in case of placement changes, this results in failing requests until the load balancer figures it out.
Create separate node types (VM Scale Sets) for every service and create a separate load balancer for each node type. However, this results in management overhead and might not be ideal in terms of resource usage.
There is an open feature suggestion for this topic on the Azure UserVoice site - your votes would be welcome.
I have a scenario where one of our services exposes WCF hosts that receive callbacks from an external service.
These hosts are dynamically created and there may be hundreds of them. I need to ensure that they are all up and running on the node before the node starts receiving requests so they don't receive failures, this is critical.
Is there a way to ensure that the service doesn't receive requests until I say it's ready? In cloud services I could do this by containing all this code within the OnStart method.
My initial thought is that I might be able to bootstrap this before I open the communication listener - in the hope that the fabric manager only sends requests once this has been done, but I can't find any information on how this lifetime is handled.
There's no "fabric manager" that controls network traffic between your services within the cluster. If your service is up, clients or other services inside the cluster can choose to try to connect to it if they know its address. With that in mind, there are two things you have control over here:
The first is whether or not your service's endpoint is discoverable by other services or clients. This is the point at which your service endpoint is registered with Service Fabric's Naming Service, which occurs when your ICommunicationListener.OpenAsync method returns. At that point, the service endpoint is registered and others can discover it and attempt to connect to it. Of course you don't have to use the Naming Service or the ICommunicationListener pattern if you don't want to; your service can open up an endpoint whenever it feels like it, but if you don't register it with the Naming Service, you'll have to come up with your own service discovery mechanism.
The second is whether or not the node on which your service is running is receiving traffic from the Azure Load Balancer (or any load balancer if you're not hosting in Azure). This has less to do with Service Fabric and more to do with the load balancer itself. In Azure, you can use a load balancer probe to determine whether or not traffic should be sent to nodes.
EDIT:
I added some info about the Azure Load Balancer to our documentation, hope this helps: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-connect-and-communicate-with-services/