Say I have multiple namespaces on a kube cluster and each namespace has a RMQ container, and these RMQ containers are spread throughout a few nodes, is there a way to deploy one metricbeat pod (preferably as a daemonset) per node to monitor these RMQ containers? What's the best practice regarding using Metricbeat for monitoring, do you need one metricbeat per container? Thanks.
Metricbeat is installed as a daemonset to ensure that it can get all the node stats across all namespaces. So, you just need one instance of Metricbeat on every node in your k8s cluster
More details: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html
Related
I have a EKS cluster with cluster autoscaler setup, spanning across three availability zones. I have deployed a Redis Cluster using helm and it works fine. Basically it is a statefulset of 6 replicas with dynamic PVC.
Currently, my EKS cluster has two worker nodes, which I will name as Worker-1A and Worker-1B in AZ 1A and 1B respectively, and has no worker node on AZ 1C. I am doing some testing to make sure the Redis Cluster can always spin up and attach the volume properly. All the Redis Cluster pods are created in Worker-1B. In my testing, I kill all the pods in the Redis Cluster, and before it spins new pods up, I deploy some other deployments to use all the resources in Worker-1A and Worker-1B. Now since that the worker nodes have no resource to create new pods, the cluster autoscaler will create a worker node in AZ 1C (to balance nodes across AZ). Now the problem comes, when the Redis Cluster statefulset trying to recreate the pods, it cannot create in Worker-1B because there is no resource, and it will try to create in Worker-1C instead, and the pods will hit the following error: node(s) had volume node affinity conflict.
I know this situation might be rare but how do I fix this issue if it ever happens? I am hoping if there is an automated way to solve this instead of fixing it manually.
I have been reading for several days about how to deploy a Kubernetes cluster from scratch. It's all ok until it comes to etcd.
I want to deploy the etcd nodes inside the Kubernetes cluster. It looks there are many options, like etcd-operator (https://github.com/coreos/etcd-operator).
But, to my knowledge, a StatefulSet or a ReplicaSet makes use of a etcd.
So, what is the right way to deploy such a cluster?
My first thought: start with a single member etcd, either as a pod or a local service in the master node and, when the Kubernetes cluster is up, deploy the etcd StatefulSet and move/change/migate the initial etcd to the new cluster.
The last part sounds weird to me: "and move/change/migate the initial etcd to the new cluster."
Am I wrong with this approach?
I don't find useful information on this topic.
Kubernetes has 3 components: master components, node components and addons.
Master components
kube-apiserver
etcd
kube-scheduler
kube-controller-manager/cloud-controller-manager
Node components
kubelet
kube-proxy
Container Runtime
While implementing Kubernetes yu have to implement etcd as part of it. If it is multi node architecture you can use independent node or along with master node as per your requirement. You can find more details here. If you are looking for step by step guide follow this document if you need multi node architecture. If you need single node Kubernetes go for minikube.
In Kelsey Hightower's Kubernetes Up and Running, he gives two commands :
kubectl get daemonSets --namespace=kube-system kube-proxy
and
kubectl get deployments --namespace=kube-system kube-dns
Why does one use daemonSets and the other deployments?
And what's the difference?
Kubernetes deployments manage stateless services running on your cluster (as opposed to for example StatefulSets which manage stateful services). Their purpose is to keep a set of identical pods running and upgrade them in a controlled way. For example, you define how many replicas(pods) of your app you want to run in the deployment definition and kubernetes will make that many replicas of your application spread over nodes. If you say 5 replica's over 3 nodes, then some nodes will have more than one replica of your app running.
DaemonSets manage groups of replicated Pods. However, DaemonSets attempt to adhere to a one-Pod-per-node model, either across the entire cluster or a subset of nodes. A Daemonset will not run more than one replica per node. Another advantage of using a Daemonset is that, if you add a node to the cluster, then the Daemonset will automatically spawn a pod on that node, which a deployment will not do.
DaemonSets are useful for deploying ongoing background tasks that you need to run on all or certain nodes, and which do not require user intervention. Examples of such tasks include storage daemons like ceph, log collection daemons like fluentd, and node monitoring daemons like collectd
Lets take the example you mentioned in your question: why iskube-dns a deployment andkube-proxy a daemonset?
The reason behind that is that kube-proxy is needed on every node in the cluster to run IP tables, so that every node can access every pod no matter on which node it resides. Hence, when we make kube-proxy a daemonset and another node is added to the cluster at a later time, kube-proxy is automatically spawned on that node.
Kube-dns responsibility is to discover a service IP using its name and only one replica of kube-dns is enough to resolve the service name to its IP. Hence we make kube-dns a deployment, because we don't need kube-dns on every node.
I have a Kubernetes deployment that has 3 replicas. It starts 3 pods which are distributed across a given cluster. I would like to know how to reliably get one pod to contact another pod within the same ReplicaSet.
The deployment above is already wrapped up in a Kubernetes Service. But Services do not cover my use case. I need each instance of my container (each Pod) to start-up a local in memory cache and have these cache communicate/sync with other cache instances running on other Pods. This is how I see a simple distributed cache working on for my service. Pod to pod communication within the same cluster is allowed as per the Kubernetes Network Model but I cannot see a reliable way to address each a pod from another pod.
I believe I can use a StatefulSet, however, I don't want to lose the ClusterIP assigned to the service which is required by Ingress for load balancing.
Ofcourse you can use statefulset, and ingress doesn't need ClusterIP that assigned to the service, since it uses the endpoints, so 'headless service' is ok.
i am running ipyparallel in an kube cluster. I have several pods running on one node which is fine. But for my computation i want to help ipyparallel in loadbalancing by choosing pods evenly over all nodes.
Is there a way to get this information from inside the pods/docker?
You could use a Kubernetes Service which does round-robin loadbalancing.
If you need the IP addresses, you could do a DNS A- or SRV-Records lookup and get all IPs of all running instances: http://kubernetes.io/docs/admin/dns/