I'm having the same problem with kafka-streams and spring-kafka applications. The first one is using kafka-clients:1.0.0 library while another one version 1.0.2
There is just one broker instance running in kubernetes (KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${POD_IP}:9092"). It's stateful set and it's accessed from the client app via headless service internal endpoint (although I've tried cluster ip and the issue is the same)
Once I delete this kafka pod and it's recreated, my client application can't reconnect. Pod is indeed recreated with another ip address but since I'm accessing it via service internal endpoint I'm expecting my client app to resolve this new address but it's not happening.
The kafka-clients library is logging "Found least loaded node [old_ip]:9092 (id: 0 rack: null)" while there is nothing anymore running on this address
JVM TTL cache is not a problem as I've set it to periodically refresh.
Restarting client application solves the problem
If providing {POD_IP} in KAFKA_ADVERTISED_LISTENERS causes this problem, would providing a pod's hostname solve this problem? Or is there a way to direct my client to resolve this new address?
It seems that it has something to do with KAFKA-7755. Updating client version to 2.2.0 / 2.1.1 should help.
Related
I am currently using 1.12.1 of flink
Recently, I added the HA feature using K8 in our Flink project and seeing some issue.
When we deploy the application for the first time with 2JM and 1TM it works fine. The config map in K8 gets updated with the right leader IP address.
However, if i re-deploy again with 2JM and increase TM to say 2, it doesn't work correctly. One of the TM connects to the wrong JM and stays as it is. I checked the config map and they seem to have different leader IP address. Jobmanager leader CM seems to have the right IP address however the Resource Manager CM seems to have the IP address of the standby JM.
recently I have encountered a problem with Kafka (running on our company's K8S system). Every thing was running fine then suddenly all of my kafka and zookeeper pods could not connect to their headless services (the pods are still in running state), which results in timeout exception everytime I pushlish a message into a topic. Below is an image from the log of a zookeeper pod:
The same things happen to all of my broker pods.
Have anyone faced with this problem and solved it? Please let me know.
Thanks in advance! By the way I'm sorry for my bad English.
this seems like networking problem, "no route to host"
I am using kakfa orderer service for hyperledger fabric 1.4. while updating chaincode or making any puutState call i am getting error message stated as Failed to send the transaction successfully to the order status: SERVICE UNAVAILABLE. while checking zoopeker and kafka node it seems like kafka nodes are not able to talk to each other.
kakfa & zookeeper logs
Could you provide more info about the topology of the zookeeper-kafka cluster?
Did you use docker to deploy the zk cluster? If so, you can refer to this file.
https://github.com/whchengaa/hyperledger-fabric-technical-tutorial/blob/cross-machine-hosts-file/balance-transfer/artifacts/docker-compose-kafka.yaml
Remember to specify the IP address of the other zookeeper nodes in the hosts file which is mounted to /etc/hosts of that zookeeper node.
Make sure the port number of zookeeper nodes listed in ZOO_SERVERS environment variables are correct.
We have created a statefulset & headless service. There are 2 ways by which we can define peer ips in application:
Use 'cassandra-headless-service-name' in contactPoints
Fetch the peers ip from headless-service & externalize the peers ip and read these ips when initializing the connection.
SO far so good.
Above will work if one/some pods are restarted, not all. In this case, driver will updated the new ips automatically.
But, how this will work in case of complete outage ? If all pods are down & when they come back, if all pods ip are changed (IP can change in Kubernetes), how do application will connect to Cassandra?
In a complete outage, you're right, the application will not have any valid endpoints for the cluster. Those will need to be refreshed (and the app restarted) before the app will connect to Cassandra.
We actually wrote a RESTful API that we can use query current, valid endpoints by cluster. That way, the app teams can find the current IPs for their cluster at any time. I recommend doing something similar.
I'm trying to deploy Apache Flink 1.6 on kubernetes. With following the tutorial at job manager high availabilty
page. I already have a working Zookeeper 3.10 cluster from its logs I can see that it's healthy and doesn't configured to Kerberos or SASL.All ACL rules are let's every client to write and read znodes. When I start the cluster everything works as expected every JobManager and TaskManager pods are successfully getting into Running state and I can see the connected TaskManager instances from the master JobManager's web-ui. But when I delete the master JobManager's pod, the other JobManager pod's cannot elect a leader with following error message on any JobManager-UI in the cluster.
{
"errors": [
"Service temporarily unavailable due to an ongoing leader election. Please refresh."
]
}
Even if I restart this page nothing changes. It stucks at this error message.
My suspicion is, the problem is related with high-availability.storageDir option. I already have a working (tested with CloudExplorer) minio s3 deployment to my k8s cluster. But flink cannot write anything to the s3 server. Here you can find every config from github-gist.
According to the logs it looks as if the TaskManager cannot connect to the new leader. I assume that this is the same for the web ui. The logs say that it tries to connect to flink-job-manager-0.flink-job-svc.flink.svc.cluster.local/10.244.3.166:44013. I cannot say from the logs whether flink-job-manager-1 binds to this IP. But my suspicion is that the headless service might return multiple IPs and Flink picks the wrong/old one. Could you log into the flink-job-manager-1 pod and check what its IP address is?
I think you should be able to resolve this problem by defining for each JobManager a dedicated service or if you use the pod hostname instead.