I have an elk cluster , whenever cluster goes to unhealthy (due to node down or un assigned Shards) , how can I get this alerts to email ?
Related
I deployed my frontend and backend application in my kops cluster on AWS ec2 with master size of t2 medium , when I increase the load on my applications, my both worker node goes to not ready state and the pods changes their state to pending state,
how can I resolve this issue my cluster is in production at moment.
You should firstly run kubectl get events -n default to see why the nodes go into NotReady.
Usually your cluster is overloaded. Try using cluster autoscaler to dynamically manage your cluster capacity. Also ensure you have proper resource requests on your Pods.
I am running RabbitMQ on kubernetes as service, and I have multiple microservices which get register and consumes message from rabbitmq.
Services are running in Multi Node cluster
I have install RabbitMQ using Bitnami helm chart 8.6.1 with single replica
I have 2 replicas for each service (Consumers) so there must be atmost 2 consumer for that event on rabbitmq.
After deploying new container (consumer) on K8s old container (Consumer) which already got terminated still visible on RabbitMQ connection and that cosumer is still accepting messages, and because of that old pod code gets executed (That old container was registered at 2nd April 2021).
I have verified that new deployment has all updated images with new hash.
I have checked all IP addess on K8S pod but that unwated IP is not visible.
Does anyone knows how can I trace that old pod IP on k8s, because that pod is no longer visible in k8s get pods.
Thanks
I am trying to deploy pods on the EKS cluster. Below are some screen shots which shows that AWS EKS cluster is created and is active, group nodes are also active, now when i try to deploy any pod like nginx, wordpress or something else, these are not in the ready state. I tried deploying kubernetes dashboard and its in ready state, but why others are not in ready state do not know and that's why their URLs are not reachable.
also, while checking logs it says as below:
Error from server (NotFound): pods "deployment-2048-64549f6964-87d59" not found
Pods are in pending state. If a Pod is stuck in Pending it means that it can not be scheduled onto a node. It can happen because there are insufficient resources of one type or another that prevent pods scheduling.
You can look at the output by kubectl describe <deployment/pod_name>. There will be messages from the scheduler about why it can not schedule your pod.
I have a plan to use Kubernetes in Google Cloud as a production stage.
However, I found this document explaining that there is a limitation at 250 worker nodes in a group.
In a GKE cluster, an internal forwarding rule points to all the nodes in the cluster. Each node in the cluster is a backend VM for the ILB. The maximum number of backend VMs for an ILB is 250, regardless of how the VMs are associated with instance groups. So the maximum number of nodes in a GKE cluster with an ILB is 250. If you have autoscaling enabled for your cluster, you must ensure that autoscaling does not scale your cluster beyond 250 nodes.
Could you explain the above to me?
Should I create a second and third cluster, in case our worker nodes reach ~250.
I will be using the load balancer in the front of our services.
In my kubernetes cluster one node (node1)is ubuntu and the another node(node2) is a virtual machine running inside node1. I am able to get prometheus metric from node1, but not from node 2. While i am issuing get pods command i am able to see my pods running on node 2, but i coudn't get the metric information.
Is this my Vmware issue / Cluster /Prometheus issue. how to resolve it?
Thanks,