I have multiple k8 pods where my service is running where the logic to make a connection to zookeeper (running as a cluster) and then participate in leader election on path /election is present. Now, because there are multiple apps : App1, App2, App3 and all of them are running my service thus all pods under them connect to zk and participate in leader election. But i want only the pods of App1 to connect to this zk path and participate in leader election.
The number of apps can increase/decrease.
I have a way to get the app Id of the pod thus can make code changes to connect only specific app pods to the zk path. But with this, other app pods need to have latest code changes which cannot be guaranteed.
Also is there a way to keep a config in zk itself, where we can have a script to identify whether a server can connect with the specific zk path or not?
One way I could think of is, you can protect the /election path with an ACL and make it accessible only to App1. You can give only App1 the create permission under the path and protect it with a password and thus only App1 will be able to create ephemeral nodes under /election path and other apps that do not have access to this password, will not be able to create nodes under this path.
One caveat in this approach is that, the apps other than App1 will start seeing Unauthorized Error from the moment you create the ACL for this path.
Related
I am using the Artemis Cloud operator for deploying ActiveMQ Artemis in k8s cluster. Number of replicas configured is 3. Queues are expected to be created by client applications. Operator creates a headless service and service for each pod in the cluster setup.
Whenever client connects to a pod ,it creates a queue in that broker pod.So if client connects to 3 brokers at random time, three queues gets created in pods, one in each pod respectively.So when a producer sends message to the queue, it is sent to the connected pod. It wont be present in each broker pod.(No replication of messages).
My question is what service name should client applications use inorder to connect to artemis pods and also maintain session affinity? In other words, what should be done in order to make a client connect to same broker whenever it tries a connection?(and avoid duplicate queue creation)
What I currently use is a kubernetes clusterip service I created that splits traffics to pods.And queues are created via stomp producer.
Not sure which helm it will be using in the background but you will be using the service name inside the kubernetes app for connection instead of the ClusterIP.
Service name could be starting the with helm chart name. i was checking and referring this chart for ref : https://github.com/deviceinsight/activemq-artemis-helm could be different.
Looks like it's creating the service with Cluster IP:none (Headless service) so connecting to the existing service is not working, i doubt currently it might be returning all the PODs IP.
Option : 2 if above one not work give try to this
In another case, you can create the new service type clusterIP with a different name, everything else will be the same, like port and all.
In service you can notice amqp, mqtt and other port, so you app will connect to new service with port config as per requirement. For example : active-mq-service:61613
If everything working fine for you and you are just looking for session affinity you can add this config to your service and it will start managing the session affinity.
SessionAffinity: ClientIP
If you want to make sure that connections from a particular client are
passed to the same Pod each time, you can select the session affinity
based on the client's IP addresses by setting
service.spec.sessionAffinity to "ClientIP" (the default is "None").
You can also set the maximum session sticky time by setting
service.spec.sessionAffinityConfig.clientIP.timeoutSeconds
appropriately. (the default value is 10800, which works out to be 3
hours).
Ref doc : https://kubernetes.io/docs/concepts/services-networking/service/
We have created a statefulset & headless service. There are 2 ways by which we can define peer ips in application:
Use 'cassandra-headless-service-name' in contactPoints
Fetch the peers ip from headless-service & externalize the peers ip and read these ips when initializing the connection.
SO far so good.
Above will work if one/some pods are restarted, not all. In this case, driver will updated the new ips automatically.
But, how this will work in case of complete outage ? If all pods are down & when they come back, if all pods ip are changed (IP can change in Kubernetes), how do application will connect to Cassandra?
In a complete outage, you're right, the application will not have any valid endpoints for the cluster. Those will need to be refreshed (and the app restarted) before the app will connect to Cassandra.
We actually wrote a RESTful API that we can use query current, valid endpoints by cluster. That way, the app teams can find the current IPs for their cluster at any time. I recommend doing something similar.
I want to have a publicly routable url to a user specific pod (with one container exposing port 80) on kubernetes.
i.e every end user will get a pod when they are active in the system, when they become inactive the pod is killed, when they become active again they get a new pod with a new url.
I am looking for a solution where I can goto
.example.com -> which would proxy to pod1 running for a user.
I have tried to have create one service per user with one pod running. And I add the rules to nginx ingress manually to route to a user specific service. This seems to work. However, I am not sure if this is a scalable option where I can have 10k+ users which would mean 10k+ services and 10k+ rules in ingress.
Any help is appreciated! Thanks.
It is not viable design to have one pod for each user. for 10000 of users, you are going to need 200 node cluster ( assuming 50 pods per node ). And if your website attracts more users the cluster would grow further.
User specific logic and functions should be better handled in application service itself.
I have an app with 3 nodes running with a service, which uses zookeeper to elect a leader. And I want the requests to the service to be redirected to a certain pod, depending if it's leader or not. I have a http method in my app that returns if it's leader or not (if this helps in any way).
I'm not aware of a k8s feature for doing this.
What I would do:
write a little application which looks for the endpoints of the existing service, searches the current leader, and creates/updates a 2nd service without selector (https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors) with an endpoint pointing to the leader. Then you can use that new service for your dashboard.
To do that, use a StatefulSet, for example, following the example here.
I have many services connected to zookeeper, and I want that service A can get service B's IP, when service B connected to zookeeper, is there any API can do that? Or I have to use other config file to write down all services's IP?
Take a look if this solves your problem:
http://curator.apache.org/curator-x-discovery/
Zookeeper doesn't provide service discovery out of the box, but it is easy to implement it yourself.
You won't be able to get the IP addresses of other connected clients (services, in your case) straight from the Zookeeper API. In order to get other services connected to the cluster, each service has to individually create an ephemeral znode under a specific path, e.g. /services, and set the necessary addressing and naming info as znode's data (IP, port, etc). This way, you can list that path and discover active services, or watch the /services path for any changes in your service configuration.
Since services are creating ephemeral nodes, they will automatically be removed once they are disconnected and their session expires. Of course, once you start doing something like this, you will see that there are many small details and design decisions you have to make, ergo the already mentioned Curator recipe.