I know that i can use kubectl get componentstatus
command to check the health status of the k8 cluster but some how the output i am receiving do not show the health. Below is the output from master server.
I can do deployments, can create pods and services which means everything is working fine but not sure how to check the health status.
Solved in kube-apiserver v1.17.0, also you should use command below in your older apiserver.
kubectl get cs -o=go-template='{{printf "NAME\t\t\tHEALTH_STATUS\tMESSAGE\t\n"}}{{range .items}}{{$name := .metadata.name}}{{range .conditions}}{{printf "%-24s%-16s%-20s\n" $name .status .message}}{{end}}{{end}}'
enjoy
can you try with this command
kubectl get componentstatus -o jsonpath="{.items[*].conditions[*].status}"
I know both commands are same but outputting it as yaml worked for me
try this command
kubectl cluster-info
Related
As the question shows, I have very low knowledge about kubernetes. Following a tutorial, I made a Kubernetes cluster to run a web app on a local server using Minikube. I have applied the kubernetes components and they are running but the Web-Server does not respond to HTTP requests. My problem is that all the system that I have created is like a black box for me and I have literally no idea how to open it and see where the problem is. Can you explain how I can debug such implementaions in a wise way. Thanks.
use a tool like https://github.com/kubernetes/kubernetes-dashboard
You can install kubectl and kubernetes-dashboard in a k8s cluster (https://kubernetes.io/docs/tasks/tools/install-kubectl/), and then use the kubectl command to query information about a pod or container, or use the kubernetes-dashboard web UI to query information about the cluster.
For more information, please refer to https://kubernetes.io/
kubectl get pods
will show you all your pods and their status. A quick check to make sure that all is at least running.
If there are pods that are unhealthy, then
kubectl describe pod <pod name>
will give some more information.. eg image not found etc
kubectl log <pod name> --all
is often the next step , use -f to follow the logs as you exercise your api.
It is possible to link up images running in a pod with most ide debuggers, but instructions will differ depending on language and ide used...
I have a Kubernetes cluster, in which different pods are running in different namespaces. How do I know if any pod failed?
Is there any single command to check the failed pod list or restated pod list?
And reason for the restart(logs)?
Depends if you want to have detailed information or you just want to check a few last failed pods.
I would recommend you to read about Logging Architecture.
In case you would like to have this detailed information you should use 3rd party software, as its described in Kubernetes Documentation - Logging Using Elasticsearch and Kibana or another one FluentD.
If you are using Cloud environment you can use Integrated with Cloud Logging tools (i.e. in Google Cloud Platform you can use Stackdriver).
In case you want to check logs to find reason why pod failed, it's good described in K8s docs Debug Running Pods.
If you want to get logs from specific pod
$ kubectl logs ${POD_NAME} -n {NAMESPACE}
First, look at the logs of the affected container:
$ kubectl logs ${POD_NAME} ${CONTAINER_NAME}
If your container has previously crashed, you can access the previous container's crash log with:
$ kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
Additional information you can obtain using
$ kubectl get events -o wide --all-namespaces | grep <your condition>
Similar question was posted in this SO thread, you can check if for more details.
This'll work: kubectl get pods --all-namespaces | | grep -Ev '([0-9]+)/\1'
Also, Lens is pretty good in these situations.
Most of the times, the reason for app failure is printed in the lasting logs of the previous pod. You can see them by simply putting --previous flag along with your kubectl logs ... cmd.
If I look at my logs in GCP logs, I see for instance that I got a request that gave 500
log_message: "Method: some_cloud_goo.Endpoint failed: INTERNAL_SERVER_ERROR"
I would like to quickly go to that pod and do a kubectl logs on it. But I did not find a way to do this.
I am fairly new to k8s and GKE, any way to traceback the pod that handled that request?
You could run command "kubectl get pods " on each node to check the status of all pods and could figure out accordingly by running for detail description of an error " kubectl describe pod pod-name"
As mentioned in #Neelam answer, you can can get the pod names with the command kubectl get pods -A and log into all your pods to find the error.
Or, alternatively, you could deploy a custom monitoring system like Elastic GKE Logging available in GCP github Click-to-deploy.
See here to install from MarketPlace with few clicks.
It is a free alternative to have a complete monitoring system and you can filter your logs in Kibana dashboard after deployed.
There is a documentation article here explaining on how one can reserve resources on a node for system use.
What I did not manage to figure out is how can one get these values? If I understand things correctly kubectl top nodes will return available resources, but I would like to see kube-reserved, system-reserved and eviction-threshold as well.
Is it possible?
by checking the kubelet's flag, we can get the values of kube-reserved, system-reserved and eviction-threshold.
ssh into the $NODE and ps aufx | grep kubelet will list out the running kubelet and its flag.
kube-reserved and system-reserved values are only useful for scheduling as scheduler can see the allocatable resources.
To see your eviction-threshold (evictionHard or systemReserved) after login on master node first start the kubectl proxy in the background using the following command:
kubectl proxy --port=8001 &
After that run the following command to see your desired node config (replace your node name in variable.eg VAR="worker-2")
VAR="NODE_NAME"; curl -sSL "http://localhost:8001/api/v1/nodes/$VAR/proxy/configz"
You shoul see a result look like:
"evictionHard":{"imagefs.available":"15%","memory.available":"100Mi","nodefs.available":"10%","nodefs.inodesFree":"5%"},
"systemReserved":{"cpu":"600m","memory":"0.5Gi"}
Enjoy ;)
Assuming kubectl is not setup locally, you having full access to SSH into nodes.
Is there any local commands that can be issued from the slaves nodes to edit the label ?
Can I use kubernetes patch but not being triggered from kubectl ?
or should I edit kubelet file(label section) and restart kubelet service ?
You can do a patch API call directly. Pass --v=8 to kubectl to see the API calls it is making for any command.