I want to use Apache Zookeeper (or Curator) as a replicated naming service. Let's say I run 3 zookeeper servers and I have a dozen of computers with different applications which can connect to these servers.
How should I communicate zookeeper IP addresses to clients? A configuration file which should be distributed manually to each machine?
Corba Naming service had an option of UDP broadcast discovery in which case no configuration file is needed. Is there a similar possibility in Zookeeper?
It depends where/how you are deploying. If this is at AWS you can use Route 53 or elastic IPs. In general, the solution is some kind of DNS. i.e. a well known hostname for each of the ZK instances.
If you use something like Exhibitor (disclaimer, I wrote it) it's easier in that Exhibitor can work with Apache Curator to provide up-to-date cluster information.
Related
We are running an ActiveMQ Artemis cluster in a Kubernetes cluster. All of our applications (Java/Springboot/JMS) running in the Kubernetes cluster take advantage of connecting directly to the broker instances.
However, the IP addresses from the Kubernetes Pod network are unavailable outside of the cluster. Exposing the broker instances to the public network is possible — but with different IP addresses. This is similar to hiding the Artemis cluster behind a NAT configuration. When connecting to the brokers through the public IP addresses, client applications receive cluster topology information containing IP addresses (or hostnames?) that are unreachable outside of the cluster.
Is there any way to deal with “internal” and “external” IP addresses and/or hostnames and make topology discovery work for cluster-external applications?
And, related (I am not a Java developer): Is there any way to log received topology information for JMS applications?
ActiveMQ Artemis CORE client provides the useTopologyForLoadBalancing url parameter to disable the use of the cluster topology information for load balancing, i.e.
tcp://localhost:61616?useTopologyForLoadBalancing=false
The log of the cluster topology information can be enabled setting the TRACE for the org.apache.activemq.artemis.core.protocol.core logger in the logging.properties file, see the documentation, i.e.
loggers=...,org.apache.activemq.artemis.core.protocol.core
logger.org.apache.activemq.audit.message.level=TRACE
handler.CONSOLE.level=TRACE
handler.FILE.level=TRACE
You can't rely on topology discovery from outside clients. What you can do is either provide a list of the external ips or have a router / load-balancer in front of your cluster.
Most of the examples I come across to use Kafka in Kubernetes is to deploy it as a headless service but I am not able to get the answer yet on why it should be headless and not Cluster IP? In my opinion cluster, IP provides the load balancing in which we ensure out of the box that not only one of the broker gets loaded always with its resources as I see with headless the Kafka clients be it sarma or java client tries to pick always the first IP from the DNS lookup and connects to it, will this not be a bottleneck if there are around 100+ clients trying to do the same and open connection to the first IP? or Kafka handles this inbuilt already which I am still trying to understand how it really happens.
When there is no differentiation between various instances of a services(replicas of a pod serving a stateless application), you can expose them under a ClusterIP service as connecting to any of the replica to serve the current request is okay. This is not the case with stateful services(like Kafka, databases etc). Each instance is responsible for it's own data. Each instance might be owning a different partition/topic etc. The instances of the service are not exact "replicas". Solutions for running such stateful services on Kubernetes usually use headless services and/or statefulsets so that each instance of the service has a unique identity. Such stateful applications usually have their own clustering technology that rely on each instance in the cluster having a unique identity.
Now that you know why stable identities are required for stateful applications and how statefulsets with headless services provide stable identities, you can check how your Kafka distributions might using them to run Kafka on kubernetes.
This blog post explains how strimzi does it:
For StatefulSets – which Strimzi is using to run the Kafka brokers –
you can use the Kubernetes headless service to give each of the pods a
stable DNS name. Strimzi is using these DNS names as the advertised
addresses for the Kafka brokers. So with Strimzi:
The initial connection is done using a regular Kubernetes service to
get the metadata.
The subsequent connections are opened using the DNS
names given to the pods by another headless Kubernetes service.
It's used in cases where communication to specific Pods is needed.
For example, A monitoring service must be able to reach all pods behind a service, to check their status, so it needs the addresses of all Pods and not just any one of them. This would be a use case of headless service.
Or when there is a cluster of Pods being set up, it's important to coordinate with the Pods to keep the cluster working for consumers. In Kafka, this work is done by Zookeeper. thus a headless service is needed by Zookeeper
Stateful:
Kafka streaming platform maintain replicas of partition across kafka brokers based on RELICATION_FACTOR. It maintains it data across persistent storage. When it comes to K8s ; stateful type is suggested; Pods in StatefulSets are not interchangeable: each Pod has a unique identifier that is maintained no matter where it is scheduled.
Headless:
To maintain internal communication between PODS. Lets not forget Zookeeper orchestrates kafka brokers.
Thanks
Within POD they should know eachother who is running and who stopped
I have an HPC cluster application where I am looking to replace MPI and our internal cluster management software with a combination of Kubernetes and some middleware, most likely ZMQ or RabbitMQ.
I'm trying to design how best to do peer discovery on this system using Kubernetes' service discovery.
I know Kubernetes can provide a DNS name for a given service, and that's great, but is there a way to also dynamically discover ports?
For example, assuming I replaced the MPI middleware with ZeroMQ, I would need a way for ranks (processes on the cluster) to find each other. I know I could simply have the ranks issue service creation messages to the Kubernetes discovery mechanism and get a hostname like myapp_mypid_rank_42 fairly easily, but how would I handle the port?
If possible, it would be great if I could just do:
zmqSocket.connect("tcp://myapp_mypid_rank_42");
but I don't think that would work since I have no port number information from DNS.
How can I have Kubernetes service discovery also provide a port in as simple a manner as possible to allow ranks in the cluster to discover each other?
Note: The registering process knows its port and can register it with the K8s service discovery daemon. The problem is a quick and easy way to get that port number back for the processes that want it. The question I'm asking is whether or not there is a mechanism as simple as a DNS host name, or will I need to explicitly query both hostname and port number from the k8s daemon rather than simply building a hostname based on some agreed upon rule (like building a string from myapp_mypid_myrank)?
Turns out the best way to do this is with a DNS SRV record:
https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services
https://en.wikipedia.org/wiki/SRV_record
A DNS SRV record provides both a hostname/IP and a port for a given request.
Luckily, Kubernetes service discovery supports SRV records and provides them on the cluster's DNS.
I think in the most usual case you should know the port number to access your services.
But if it is useful, Kubernetes add some environment variables to every pod to ease autodiscovery of all services. For example {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT. Docs here
Is there option to connect to external database cluster from POD? I need to connect to elastic search, zookeeeper, Kafka and couchbase, each of them has its own cluster. Per my understanding the documentation, I can define multi external IPs, but I cannot find how will k8s behave if one of them is down. I am working with pure k8s 1.6 now, and we will migrate to 1.7 soon. Information about OpenShift 3.7 will be also welcome because I cannot find anything specific in its documentation.
The k8s doc on your link has more info on exposing services running on k8s but not externally
You generally want to expose your service using a DNS entry and manage the HA for that service separately.
For example you can a single DNS entry mykafka.mydomain.com and then assign IP addresses to that entry:
kafka1 ip
kafka2 ip
kafka3 ip
You can see that approach on the Openshift docs in the USING AN EXTERNAL DOMAIN NAME section. Yes, its not clear from the docs whether k8s/openshift does a round robin on the multiple IPs for an external service and if automatically fails over.
Hope it helps.
I have many services connected to zookeeper, and I want that service A can get service B's IP, when service B connected to zookeeper, is there any API can do that? Or I have to use other config file to write down all services's IP?
Take a look if this solves your problem:
http://curator.apache.org/curator-x-discovery/
Zookeeper doesn't provide service discovery out of the box, but it is easy to implement it yourself.
You won't be able to get the IP addresses of other connected clients (services, in your case) straight from the Zookeeper API. In order to get other services connected to the cluster, each service has to individually create an ephemeral znode under a specific path, e.g. /services, and set the necessary addressing and naming info as znode's data (IP, port, etc). This way, you can list that path and discover active services, or watch the /services path for any changes in your service configuration.
Since services are creating ephemeral nodes, they will automatically be removed once they are disconnected and their session expires. Of course, once you start doing something like this, you will see that there are many small details and design decisions you have to make, ergo the already mentioned Curator recipe.