We are using spring-session to store user sessions in redis cluster. We want to receive session-created and session-destroyed events. However it seems, redis cluster does not publish events properly. We receive session events if we use standalone redis server.
We have enabled all notifications on all redis nodes in the cluster.
Is event publishing/subscription supported with redis clusters?
It seems while using cluster, it tries to subscribe to only one node in the cluster. Should it subscribe to all nodes?
It was a Jedis issue which has been fixed in https://github.com/xetorthio/jedis/issues/1258.
Also go through the replies of issue reported on git-hub for more information :
https://github.com/spring-projects/spring-session/issues/478
Related
I am using the Artemis Cloud operator for deploying ActiveMQ Artemis in k8s cluster. Number of replicas configured is 3. Queues are expected to be created by client applications. Operator creates a headless service and service for each pod in the cluster setup.
Whenever client connects to a pod ,it creates a queue in that broker pod.So if client connects to 3 brokers at random time, three queues gets created in pods, one in each pod respectively.So when a producer sends message to the queue, it is sent to the connected pod. It wont be present in each broker pod.(No replication of messages).
My question is what service name should client applications use inorder to connect to artemis pods and also maintain session affinity? In other words, what should be done in order to make a client connect to same broker whenever it tries a connection?(and avoid duplicate queue creation)
What I currently use is a kubernetes clusterip service I created that splits traffics to pods.And queues are created via stomp producer.
Not sure which helm it will be using in the background but you will be using the service name inside the kubernetes app for connection instead of the ClusterIP.
Service name could be starting the with helm chart name. i was checking and referring this chart for ref : https://github.com/deviceinsight/activemq-artemis-helm could be different.
Looks like it's creating the service with Cluster IP:none (Headless service) so connecting to the existing service is not working, i doubt currently it might be returning all the PODs IP.
Option : 2 if above one not work give try to this
In another case, you can create the new service type clusterIP with a different name, everything else will be the same, like port and all.
In service you can notice amqp, mqtt and other port, so you app will connect to new service with port config as per requirement. For example : active-mq-service:61613
If everything working fine for you and you are just looking for session affinity you can add this config to your service and it will start managing the session affinity.
SessionAffinity: ClientIP
If you want to make sure that connections from a particular client are
passed to the same Pod each time, you can select the session affinity
based on the client's IP addresses by setting
service.spec.sessionAffinity to "ClientIP" (the default is "None").
You can also set the maximum session sticky time by setting
service.spec.sessionAffinityConfig.clientIP.timeoutSeconds
appropriately. (the default value is 10800, which works out to be 3
hours).
Ref doc : https://kubernetes.io/docs/concepts/services-networking/service/
We have created a statefulset & headless service. There are 2 ways by which we can define peer ips in application:
Use 'cassandra-headless-service-name' in contactPoints
Fetch the peers ip from headless-service & externalize the peers ip and read these ips when initializing the connection.
SO far so good.
Above will work if one/some pods are restarted, not all. In this case, driver will updated the new ips automatically.
But, how this will work in case of complete outage ? If all pods are down & when they come back, if all pods ip are changed (IP can change in Kubernetes), how do application will connect to Cassandra?
In a complete outage, you're right, the application will not have any valid endpoints for the cluster. Those will need to be refreshed (and the app restarted) before the app will connect to Cassandra.
We actually wrote a RESTful API that we can use query current, valid endpoints by cluster. That way, the app teams can find the current IPs for their cluster at any time. I recommend doing something similar.
Most of the examples I come across to use Kafka in Kubernetes is to deploy it as a headless service but I am not able to get the answer yet on why it should be headless and not Cluster IP? In my opinion cluster, IP provides the load balancing in which we ensure out of the box that not only one of the broker gets loaded always with its resources as I see with headless the Kafka clients be it sarma or java client tries to pick always the first IP from the DNS lookup and connects to it, will this not be a bottleneck if there are around 100+ clients trying to do the same and open connection to the first IP? or Kafka handles this inbuilt already which I am still trying to understand how it really happens.
When there is no differentiation between various instances of a services(replicas of a pod serving a stateless application), you can expose them under a ClusterIP service as connecting to any of the replica to serve the current request is okay. This is not the case with stateful services(like Kafka, databases etc). Each instance is responsible for it's own data. Each instance might be owning a different partition/topic etc. The instances of the service are not exact "replicas". Solutions for running such stateful services on Kubernetes usually use headless services and/or statefulsets so that each instance of the service has a unique identity. Such stateful applications usually have their own clustering technology that rely on each instance in the cluster having a unique identity.
Now that you know why stable identities are required for stateful applications and how statefulsets with headless services provide stable identities, you can check how your Kafka distributions might using them to run Kafka on kubernetes.
This blog post explains how strimzi does it:
For StatefulSets – which Strimzi is using to run the Kafka brokers –
you can use the Kubernetes headless service to give each of the pods a
stable DNS name. Strimzi is using these DNS names as the advertised
addresses for the Kafka brokers. So with Strimzi:
The initial connection is done using a regular Kubernetes service to
get the metadata.
The subsequent connections are opened using the DNS
names given to the pods by another headless Kubernetes service.
It's used in cases where communication to specific Pods is needed.
For example, A monitoring service must be able to reach all pods behind a service, to check their status, so it needs the addresses of all Pods and not just any one of them. This would be a use case of headless service.
Or when there is a cluster of Pods being set up, it's important to coordinate with the Pods to keep the cluster working for consumers. In Kafka, this work is done by Zookeeper. thus a headless service is needed by Zookeeper
Stateful:
Kafka streaming platform maintain replicas of partition across kafka brokers based on RELICATION_FACTOR. It maintains it data across persistent storage. When it comes to K8s ; stateful type is suggested; Pods in StatefulSets are not interchangeable: each Pod has a unique identifier that is maintained no matter where it is scheduled.
Headless:
To maintain internal communication between PODS. Lets not forget Zookeeper orchestrates kafka brokers.
Thanks
Within POD they should know eachother who is running and who stopped
I'm in the middle of creating an K8s app that doesn't expose any HTTP endpoints, is just a background app that pulls messages from a message bus and takes some action based on the incoming message. No other apps will interact directly with this background app, only thru posting messages into the message bus.
Scaling is a requirement and most likely will always need to run more than one replica. What is the recommended Service type in Kubernetes to handle this type of workload ?
No service required... just create a Deployment, which will result in a ReplicaSet, which will keep n replicas of your app running.
I have a single server instance in the cluster which has got some special duties and we can have only one such server in the cluster. I would like to monitor the health of this server and bring up another instance if this node fails and make the new node join the cluster. Do you think Apache zookeeper can do the job? Could you point me to an example?