Hoping for some pointers, I have a question: what role/permissions do I need to add to a user that has Clusterrole:view in order to let that user use 'top node' and 'top pod' command in kubectl? We're using kubectl version 1.15.x (client) on a 1.15.7 cluster (AKS), although I'm not sure if that's relevant information.
I tried to find this in the k8s documentation, but could not find the permission-reference descrption, only some examples and guidelines.
Thanks in advance!
Regards, Ludo
There is no specific permission for top node and top pod. The permission is on whether you can do get nodes and get pods. If permission is there to do get nodes and get pods then you should be able to top nodes and top pods as well. Permission to perform get nodes or get pods is defined via RBAC in kubernetes.
The metric server just exposes metrics of different resources such as pods, nodes by collecting and aggregating it from different sources such as kubelet. That's the reason there is no permission for the metrics itself rather the permission is on the resources of which you want to see the metrics.
The API that is being invoked when you do kubectl top pods is this
GET https://API-SERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods
So you can only see metrics for pods in the default namespace when you are targeted to default namespace. Now whether you can get pods in default namespace is governed by RBAC.
Related
Is it possible to see nodes names via kubectl get nodes command in AWS EKS?
When I'm typing kubectl get nodes I'm getting list of nodes in format ip-111-22-3-444.region.compute.internal. It is much harder to understand what nodes are provided for. It is true that you can get information about nodegroup via describe command but it would be much nicer to see it on the list, like it is in GKE and AKS. In EC2 console I can see them named as expected but not in kubectl get nodes. Is it possible in EKS?
I have deployed a k8s service, however its not showing any pods. This is what I see
kubectl get deployments
It should create on the default namespace
kubectl get nodes (this shows me nothing)
How do I troubleshoot a failed deployment. The test-control-plane is the one deployed by kind this is the k8s one I'm using.
kubectl get nodes
If above command is not showing anything which mean there is no Nodes in your cluster so where your workload will run ?
You need to have at least one worker node in K8s cluster so deployment can schedule the POD on it and run the application.
You can check worker node using same command
kubectl get nodes
You can debug more and check the reason of issue further using
kubectl describe deployment <name of your deployment>
To find out what really went wrong, first follow the steps described in Harsh Manvar in his answer. Perhaps obtaining that information can help you find the problem. If not, check the logs of your deployment. Try to list your pods and see which ones did not boot properly, then check their logs.
You can also use the kubectl describe on pods to see in more detail what went wrong. Since you are using kind, I include a list of known errors for you.
You can also see this visual guide on troubleshooting Kubernetes deployments and 5 Tips for Troubleshooting Kubernetes Deployments.
I am surprised that nobody has yet asked this, but what exactly is deployment.apps?
I see it often in commands e.g
kubectl rollout pause deployment.apps/nginx-deployment
or even used interchangably for the deployments keyword:
kubectl get deployments= kubectl get deployment.apps
I do not understand what it indicates though. Even in K8s official docs, they just take for granted that the reader understands the term.
Could someone please explain it to me?
Kubernetes API has its different resources (e.g. Pods, Deployments, Ingress) grouped in what they call "api groups" and in the notation deployment.apps - "deployment" is the resource name and the "apps" is the api group name.
Also see the motivation for API groups
I am scraping the kubernetes metrics from prometheus and would need to extract the number of running pods.
I can see container_last_seen metrics but how should i get no of pods running. Can someone help on this?
If you need to get number of running pods, you can use a metric from the list of pods metrics https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md for that (To get the info purely on pods, it'd make sens to use pod-specific metrics).
For example if you need to get the number of pods per namespace, it'll be:
count(kube_pod_info{namespace="$namespace_name"}) by (namespace)
To get the number of all pods running on the cluster, then just do:
count(kube_pod_info)
Assuming you want to display that in Grafana according to your question tags, from this Kubernetes App Metrics dashboard for example:
count(count(container_memory_usage_bytes{container_name="$container", namespace="$namespace"}) by (pod_name))
You can just import the dashboard and play with the queries.
Depending on your configuration/deployment, you can adjust the variables container_name and namespace, grouping by (pod_name) and count'ing it does the trick. Some other label than pod_name can be used as long as it's shared between the pods you want to count.
If you want to see only the number of "deployed" pods in some namespace, you can use the solutions in previous answers.
My use case was to see the current running pods in some namespace and below is my solution:
'min_over_time(sum(group(kube_pod_container_status_ready{namespace="BC_NAME"}) by (pod,uid)) [5m:1m]) OR on() vector(0)'
Please replace BC_NAME with your namespace name.
The timespan provides you fine the data.
If no data found - no pod currently running it returns '0'
I want to have full control of what I do with my single node cluster (savings...lol), but somehow I can't do this even if I delete the deployment it respawns ..
As mentioned in another answer, you cannot delete them directly via the Kubernetes API; however, you can delete them indirectly via the Google Container Engine API.
To remove the dashboard, run gcloud container clusters update $CLUSTER_NAME --update-addons=KubernetesDashboard=DISABLED.
To disable heapster you need to disable monitoring using gcloud container clusters update $CLUSTER_NAME --monitoring-service=none (it may actually require disabling another add-on too, I can't recall at the moment).
See https://cloud.google.com/sdk/gcloud/reference/container/clusters/update for the commands referenced above.
Heapster is configured as a cluster addon. The addon manager is going to reconcile it to it's preconfigured state if you change or delete it.
You are stuck with it.
Even if you delete heapster pod; it restart automatically. I can made it with scaling it down to zero as shown below
kubectl scale --replicas=0 deployment/heapster-v1.6.0-beta.1 --namespace=kube-system
And you can find the exact name of the heapster pod within result of the command below
kubectl get deployments --namespace=kube-system
By the way you can find more options to reduce resource usage here:
https://cloud.google.com/kubernetes-engine/docs/how-to/small-cluster-tuning