For joining a new agent into a group of consul members, one needs to know the IP of at least one existing member.
I have to support a scenario in which I do not know an IP of any existing group members. Human intervention is not possible in this situation as a the service discovery has to run completely automatically.
My idea to solve this: I could send a message via broadcast or multicast to other computers in the network. Every computer hosts a custom service that I will implement. That service replies to this special broadcast/multicast message (e.g.: "Looking for existing cluster members") with the IP address to which the consul is listening. The service will then get one or more replies from the network and use one of the IP addresses to join the cluster.
Does anyone have a better idea to solve this problem? Are there any existing libraries to solve this?
Hashicorp provides a free service as part of Atlas to do just that: https://www.consul.io/docs/guides/atlas.html
The other easier ways to do it without a custom service:
Expose Consul servers in a third-party DNS that the servers can access before they join the Consul cluster. Alternatively, have a http service that is exposed that returns an IP to join to and use curl or something to get the IP. Keep the IP updated using Consul.
If you're running in AWS or a similar service that provides metadata about machines, use that service to tag your Consul boxes and then use the service's query functionality to do something like "find servers that have tag Consul and are running" and pass them to -join.
Related
I have a couple of microservices that I want to register in Consul, so that they can find each other and communicate.
Everything runs on docker compose.
I am wondering how that would work if one of the two services has multiple replicas. How does Consul (or docker compose) deal with that? Is there some sort of internal load balancing or what?
Consul supports registering multiple instances/replicas of a service. When a consumer queries the Consul catalog, Consul will return information for each of the registered service instances.
If the consumer/client is querying Consul via DNS, the client's DNS resolver will ultimately be responsible for choosing the endpoint to connect to from the list of IP's in the DNS response.
If the client is querying Consul via the HTTP API (e.g., /v1/agent/health/service/:service), the client must implement its own logic to select an upstream instance from the list of instances returned in the API response.
See the query services section of the Register a Service with Consul Service Discovery tutorial for more info.
I have followed the AWS getting started guide to provision an EKS cluster (3 public subnets and 3 private subnets). After creating it, I get the following API server endpoint https://XXXXXXXXXXXXXXXXXXXX.gr7.us-east-2.eks.amazonaws.com (replaced the URL with X's for privacy reasons).
Accessing the URL in the browser I get the expected output from the cluster endpoint.
Question: How do I point my registered domain in Route 53 to my cluster endpoint?
I can't use a cname record because my domain is a root domain and will receive an apex domain error.
I don' have access to a static ip, and I don't believe my EKS cluster has a public IP address I can directly used. This would mean I can't use an A record (as I need an IP address).
Can I please get help/instructions as to how I can point my domain straight to my cluster?
Below is my AWS VPC architecture:
Don't try and assign a pretty name to the API endpoint. Your cluster endpoint is the address that's used to talk to the control plane. When you configure your kubectl tool, the api endpoint is what kubectl talks to.
Once you've got an application running on your EKS cluster, and have a load balancer, or Ingress, or something for incoming connections, that's when you worry about creating pretty names.
And yes, If you're dealing with AWS load balancers, you don't get the option of A records, so you can't use the apex of the domain, unless you're hosting DNS in route 53, in which case, you can use "alias" records to point the apex of a domain at a load balancer.
Kubernetes is a massively complex thing to try understand and get running. Given that this is the type of question you're asking, it sounds like you don't have the full picture yet. I recommend (1) joining the Kubenetes slack channel. It'll be a much faster way to get help than SO, and (2) take in Jeff Geerling's excellent Kubernetes 101 course on youtube.
I created my kubernetes cluster with specified security group for each ec2 server type, for example for backend server I have backend-sg associated with and a node-sg which is created with the cluster.
Now I try to restrict access to my backend ec2 and open only port 8090 as an inbound and port 8080 as an outbound to a specific security group (lets call it frontend-sg).
I was manage to do so but when changing the inbound port to 8081 in order to check that those restrictions actually worked I was still able to acess port 8080 from the frontend-sg ec2.
I think I am missing something...
Any help would be appreciated
Any help would be appriciated
I will try to illustrate situation in this answer to make it more clear. If I'm understanding your case correctly, this is what you have so far:
Now if you try ports from Frontend EC2 instance to Backend EC2 instance, and they are in same security group (node-sg) you will have traffic there. If you want to check group isolation then you should have one instance outside of node-sg and only in frontend-sg targetting any instance in backend-sg (supposing that both node-sg and backend-sg are not permitting said ports for inbound traffic)...
Finally, a small note... Kubernetes is by default closing all traffic (and you need ingress, loadbalancer, upstream proxy, nodePort or some other means to actually expose your front-facing services) so traditional fine graining of backend/frontend instances and security groups is not that "clearcut" when using k8s, especially since you don't really want to schedule manually (or by labels for that matter) which instances pods will actually run (but instead leave that to k8s scheduler for better unitilization of resources).
We are running a SaaS service that we are looking to migrate to Kubernetes, preferably at one of the hyperscalars. One specific issue I have not yet found a clean solution for is the need for Egress IP address selection from within the application.
We deal with a large amount of upstream providers that have access control and rate limiting based on source IP adres. Also a partition of our customers are using their own accounts with some of the upstream providers. To access the upstream providers in the context of their account we need to control the source IP used for the connection from within the application.
We are running currently our services in a DMZ behind a load balancer, so direct network interface selection is already impossible. We use some iptables rules on our load balancers/gateways to do address selection based on mapped port numbers. (e.g. egress connections to port 1081 are mapped to source address B and target port 80, port 1082 to source address C port 80)
This however is quite a fragile setup that also does not map nicely when trying to migrate to more standardized *aaS offerings.
Looking for suggestions for a better setup.
One of the things that could help you solve it is Istio Egress Gateway so I suggest you look into it.
Otherwise, it is still dependent on particular platform and way to deploy your cluster. For example on AWS you can make sure your egress traffic always leaves from predefined, known set of IPs by using instances with Elastic IPs assigned to forward your traffic (be it regular EC2s or AWS NAT Gateways). Even with Egress above, you need some way to define a fixed IP for this, so AWS ElasticIP (or equivalent) is a must.
I have an HPC cluster application where I am looking to replace MPI and our internal cluster management software with a combination of Kubernetes and some middleware, most likely ZMQ or RabbitMQ.
I'm trying to design how best to do peer discovery on this system using Kubernetes' service discovery.
I know Kubernetes can provide a DNS name for a given service, and that's great, but is there a way to also dynamically discover ports?
For example, assuming I replaced the MPI middleware with ZeroMQ, I would need a way for ranks (processes on the cluster) to find each other. I know I could simply have the ranks issue service creation messages to the Kubernetes discovery mechanism and get a hostname like myapp_mypid_rank_42 fairly easily, but how would I handle the port?
If possible, it would be great if I could just do:
zmqSocket.connect("tcp://myapp_mypid_rank_42");
but I don't think that would work since I have no port number information from DNS.
How can I have Kubernetes service discovery also provide a port in as simple a manner as possible to allow ranks in the cluster to discover each other?
Note: The registering process knows its port and can register it with the K8s service discovery daemon. The problem is a quick and easy way to get that port number back for the processes that want it. The question I'm asking is whether or not there is a mechanism as simple as a DNS host name, or will I need to explicitly query both hostname and port number from the k8s daemon rather than simply building a hostname based on some agreed upon rule (like building a string from myapp_mypid_myrank)?
Turns out the best way to do this is with a DNS SRV record:
https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services
https://en.wikipedia.org/wiki/SRV_record
A DNS SRV record provides both a hostname/IP and a port for a given request.
Luckily, Kubernetes service discovery supports SRV records and provides them on the cluster's DNS.
I think in the most usual case you should know the port number to access your services.
But if it is useful, Kubernetes add some environment variables to every pod to ease autodiscovery of all services. For example {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT. Docs here