Get kubernetes apiserver prometheus metrics with kubectl? - kubernetes

I'm trying to get kube-apiserver prometheus metrics with kubectl command.
I can get it from kubectl proxy [which proxy the api in localhost:8001(by default)]. then go to http://localhost:8001/metrics
I want to get this metrics directly through kubectl command, without proxying.
My kubernetes version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.24", GitCommit:"2ce02ef1754a457ba464ab87dba9090d90cf0468", GitTreeState:"clean", BuildDate:"2019-08-12T22:05:28Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}


Got the answer
$ kubectl get --raw /metrics
This command directly communicate with kube apiserver and get the prometheus metrics of kube apiserver.


You can install helm chart present here -
https://github.com/helm/charts/tree/master/stable/prometheus-operator
It will provide you all metrics, you can run the prometheus service locally by port-forwarding and can get all metrics at its dashboard

Related

Grafana showing k8s pods down for a minute

while using grafana for monitoring with Prometheus, we saw that sometimes grafana showed no pods for a service but when I checked in the cluster, all pods are running without any issue.
This issue is not continuous. Now I have to find out why grafana is alerting? But I don't know where to start.
Pls, ask if any info needed and pls show me the path, where I can start investigating.
Other info
This cluster is AWS EKS. Using prometheus:v2.22.1. Deployment of Prometheus & EKS cluster is done by Terraform.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c49e2", GitCommit:"8c49e2efc3cfbb7788a58025e679787daed22018", GitTreeState:"clean", BuildDate:"2021-10-17T05:13:46Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.18) exceeds the supported minor version skew of +/-1

Is there a way to rollback edited kubernetes manifest file?

Pod with some values was deployed, then I edited it kubectl edit pod <pod>, and now wanted to get back to the previous state (no longer have the values as someone else deployed it some time ago). Is it possible?
And second question.
If someone deployed to GKE cluster with helm, is it possible (even though I have access to cluster and can see all kubectl get all) that I don't see those deployments with helm list but see the kubernetes pods ? - rephrasing it. Is it possible someone deployed to cluster with helm and I only see pods, no helm config with helm list ?
ps: helm and kubernetes works fine with other clusters or minikube:
helm version
Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T11:51:44Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}```

Pods does not have rollback feature that's why you should use deployment which provides rollback functionality. Also good practice for production is to version control your yamls for easy rollback and audit.

"kubectl describe ingress ..." could not find the requested resource

I am trying to execute describe on ingress but does not work. Get command works fine but not describe. Is anything that I am doing wrong? I am running this against AKS.
usr#test:/mnt/c/Repos/user/charts/canary$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-route xyz.westus.cloudapp.azure.com 80 6h
usr#test:/mnt/c/Repos/user/charts/canary$ kubectl describe ingress ingress-route
Error from server (NotFound): the server could not find the requested resource
Version seems fine:
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", ..}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10"...}

This could be caused by the incompatibility between the Kubernetes cluster version and the kubectl version.
Run kubectl version to print the client and server versions for the current context, example:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-gke.0", GitCommit:"569511c9540f78a94cc6a41d895c382d0946c11a", GitTreeState:"clean", BuildDate:"2019-08-21T23:28:44Z", GoVersion:"go1.11.13b4", Compiler:"gc", Platform:"linux/amd64"}
You might want to upgrade your cluster version or downgrade your kubectl version. See more details in https://github.com/kubernetes/kubectl/issues/675

Kubectl Unable to describe on HPA

When I'm trying to describe on hpa following error is thrown:
kubectl describe hpa go-auth
Error from server (NotFound): the server could not find the requested resource
My kubectl version is :
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.7", GitCommit:"b80664a77d3bce5b4701bc881d972b1a702290bf", GitTreeState:"clean", BuildDate:"2019-04-04T03:12:09Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

Beware of kubectl version skew. Running kubectl v1.14 with kube-apiserver v1.12 is not supported.
As per kubectl docs:
You must use a kubectl version that is within one minor version
difference of your cluster. For example, a v1.2 client should work
with v1.1, v1.2, and v1.3 master. Using the latest version of kubectl
helps avoid unforeseen issues.
Give it another try using kubectl v1.12.x and you probably will get rid of this problem. Also, take a look at the #568 issue (especially this comment), which addresses the same problem that you have.
If you are wondering on how to manage multiple kubectl versions, I recommend this read: Using different kubectl versions with multiple Kubernetes clusters.

The kubernetes "AVAILABLE" column indicates "0", but the former steps(in Kubernetes guide) are OK

I need to deploy some docker images, and manage them with the Kubernetes.
I followed the tutorial"Interactive Tutorial - Deploying an App"(https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive/).
But after I typing the command kuberctl get deployments, in the result table, the deployment column shows 0 instead of 1, it's confusing me.
If there is anyone kindly guides me what's going wrong and what shall I do?
The OS is Ubuntu16.04;
The kuberctl version command shows the server and client version informations well.
The docker image is tagged already(a mysql:5.7 image).
devserver:~$ kubectl version　　　　
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}　　
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
devserver:~$ kubectl get deployments
NAME 　DESIRED　CURRENT　UP-TO-DATE　AVAILABLE　AGE
ap-mysql 　　１　　　　１　　　　１　　　　　　　０　　　　 １
hello-node　　１　　　　１　　　　１　　　　　　　０ 　　　　１
I expect the answer about the phenomenon and the resolution. And I need to deploy my image on the minikube.

Katacoda uses hosted VM's so sometimes it may be slow to respond to the terminal input.
To verify if any deployment is present you may run kubectl get deployments --all-namespaces.To see what's going on with your deployment you can run kubectl describe DEPLOYMENT_NAME -n NAMESPACE.To inspect a pod you can do the same kubectl describe POD_NAME -n NAMESPACE.