In my kubernetes cluster one node (node1)is ubuntu and the another node(node2) is a virtual machine running inside node1. I am able to get prometheus metric from node1, but not from node 2. While i am issuing get pods command i am able to see my pods running on node 2, but i coudn't get the metric information.
Is this my Vmware issue / Cluster /Prometheus issue. how to resolve it?
Thanks,
Related
I'm trying to launch a prometheus pod in order to scrape the etcd metrics from within our kubernetes cluster.
I was trying to reproduce the solution proposed here: Access etcd metrics for Prometheus
Unfortunately, the etcd containers seem to be unavailable from the cluster.
# nc -vz etcd1 2379
nc: getaddrinfo for host "etcd1" port 2379: Name or service not known
In a way, this seems logical since no etcd container appear in the cluster:
kubectl get pods -A | grep -i etcd does not return anything.
However, when I connect onto the machine hosting the master nodes, I can find the containers using the docker ps command.
The cluster has been deployed using Kubespray.
Do you know if there is a way to reach the etcd containers from the cluster pods?
Duh… the etcd container is configured with the host network. Therefore, the metrics endpoint is directly accessible on the node.
I have followed this https://www.gojek.io/blog/diy-set-up-telegraf-influxdb-grafana-on-kubernetes
and able to create the setup.
Now grafana is only showing metrics for the pod running telegraf.
Is it possible to monitor the pod using telegraf running another application on same node?
can someone please help?
I am monitoring Kubernetes cluster ( which is deployed using kubeadm ) with grafana and Prometheus.
I am facing some difficulty with kube-state-metrics that is when I up the only master node then I am seeing kube-state-metrics down in Prometheus targets but when I up the one node then kube-state-metrics is up in Prometheus target.
Another interesting part here is when I up only master node I see one kube-state-meterics pod is up and running in kube-system namespace but I am unable to access the metrics.
Am I missing anything in understanding kube-state-metrics?
Please help me out.
I have an AKS Cluster with two nodepools. Node pool 1 has 3 nodes, and nodepool 2 has 1 node - all Linux VMs. I noticed that after stopping the VMs and then doing kubectl get pods, the Pods status shows "running" though the VMs are not actually running. How is this possible?
This is the command I tried: kubectl get pods -n development -o=wide
The screenshot is given below. Though VMs are not running, the Pod status shows "running". However, trying to access the app using the Public IP of the service resulted in
ERR_CONNECTION_TIMED_OUT
Here is a full thread (https://github.com/kubernetes/kubernetes/issues/55713) on this issue. The problem here is by default the pod waits for 5 minutes before evicting to another node when the current node becomes notReady, but in this case none of the worker nodes are ready and hence pods are not getting evicted. Refer the git issue, there are some suggestions and solutions provided.
What is actually going is related to the kubelet processes running on the nodes cannot provide their status to the Kubernetes API server. Kubernetes will always assume that your PODs are running when the nodes associated with the POD are offline. The fact that all nodes are offline, will in fact cause your POD to not be running hence not being accessible, causing the ERR_CONNECTION_TIMED_OUT
You can run kubectl get nodes to get the status of the nodes, they should show NotReady. Please check and let me know.
Also, can you please provide the output for kubectl get pods -A
i am running ipyparallel in an kube cluster. I have several pods running on one node which is fine. But for my computation i want to help ipyparallel in loadbalancing by choosing pods evenly over all nodes.
Is there a way to get this information from inside the pods/docker?
You could use a Kubernetes Service which does round-robin loadbalancing.
If you need the IP addresses, you could do a DNS A- or SRV-Records lookup and get all IPs of all running instances: http://kubernetes.io/docs/admin/dns/