How to grep older logs from all pods in K8 (-l and --since=10m not working simultaneously) - kubernetes

I have been trying to find a solution to this among the previously asked questions, but I can't find one that works for my use case (which to me seems like a general use case)
So I have a load balancer service and 5 pods in a namespace that share a label app=abc_application. So when I want to follow logs in all pods simultaneously, I use this
kubectl logs -f -l app=abc_application -c abc_application_container
Now my use case looks like this. I have a request that failed an hour back and I want to check the logs. I wanted to use the --since=60m argument but that doesn't work with the above command.
Is there any alternative than getting logs of individual pods? Can this command not be integrated?

I tried this with kubectl tail and got it working
To install kubectl tail -> kubectl krew install tail
kubectl tail -n <namespace> -l app=abc_application --since=2h

you can also do the same with logs
kubectl logs <POD name> -n <Namespace name> --since-time='2021-09-21T10:00:00Z'
Using simple since with logs
kubectl logs <POD name> -n <Namespace name> --since=60h (5s, 2m, or 3h)
If you want to tail logs by a few line
kubectl logs <POD name> -n <Namespace name> --tail=200
If want to grep anything from logs
kubectl logs <POD name> -n <Namespace name> | grep <string>
With the above command, you can pass the container name with -c & -l for label.
Reference : https://jamesdefabia.github.io/docs/user-guide/kubectl/kubectl_logs/

Related

determine the Reason for Pod Failure

I have a broken pod in default namespace. I want to store associated error events to a file error.txt. I need to use the -o wide output specifier with my command.
I saw command like this
- kubectl get pod termination-demo -o go-template="{{range
.status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"
but i need to use -o wide command.
If you want to achieve the pod state :
kubectl get pod -n <NAMESPACE> <POD> --output=wide -o go-template="{{range.status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" > pod.err
Verify by :
$ cat pod.err
not sure what you looking to achieve:
if your goal is to get the content into a file you can redirect the output to a file:
kubectl get pod termination-demo -o go-template="{{range.status.containerStatuses}}{{.lastState.terminated.message}}{{end}} > error.txt

How to access a kubernetes pod by its partial name?

I often run tasks like:
Read the log of the service X
or
Attach a shell inside the service Y
I always use something in my history like:
kubectl logs `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep <partial_name>`
or
kubectl exec -it `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep <partial_name>` bash
Do you know if kubectl has already something in place for this? Or should I create my own set of aliases?
Kubernetes instances are loosely coupled by the means of labels (key-value pairs). Because of that Kubernetes provides various functionalities that can help you to operate on sets of objects based on labels.
In case you have several pods of the same service good chances that they are managed by some ReplicaSet with the use of some specific label. You should see it if you run:
kubectl get pods --show-labels
Now for aggregating logs for instance you could use label selector like:
kubectl logs -l key=value
For more info please see: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ .
added to my .zshconfig
sshpod () {
kubectl exec --stdin --tty `kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep ${1} | head -n 1` -- /bin/bash
}
usage
sshpod podname
this
finds all pods
greps needed name
picks the first
sshs into the pod
You can go access a pod by its deployment/service/etc:
kubectl exec -it svc/foo -- bash
kubectl exec -it deployment/bar -- bash
Kubernetes will pick a pod that matches the criteria and send you to it.
You can enable shell autocompletion. Kubectl provides this support for Bash and Zsh which will save you a lot of typing (you will use TAB to get the suggestion/completion).
Kuberentes documentations has a great set of information about how to enable autocompletion under Optional kubectl configurations. It covers Bash on Linux, Bash on MacOS and Zsh.

kubernetes list all running pods name

I looking for the option to list all pods name
How to do without awk (or cut). Now i'm using this command
kubectl get --no-headers=true pods -o name | awk -F "/" '{print $2}'
Personally I prefer this method because it relies only on kubectl, is not very verbose and we don't get the pod/ prefix in the output:
kubectl get pods --no-headers -o custom-columns=":metadata.name"
You can use the go templating option built into kubectl to format the output to just show the names for each pod:
kubectl get pods --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
Get Names of pods using -o=name Refer this cheatsheet for more.
kubectl get pods -o=name
Example output:
pod/kube-xyz-53kg5
pod/kube-xyz-jh7d2
pod/kube-xyz-subt9
To remove trailing pod/ you can use standard bash sed command
kubectl get pods -o=name | sed "s/^.\{4\}//"
Example output:
kube-xyz-53kg5
kube-pqr-jh7d2
kube-abc-s2bt9
To get podname with particular string, standard linux grep command
kubectl get pods -o=name | grep kube-pqr | sed "s/^.\{4\}//"
Example output:
kube-pqr-jh7d2
With this name, you can do things, like adding alias to get shell to running container:
alias bashkubepqr='kubectl exec -it $(kubectl get pods -o=name | grep kube-pqr | sed "s/^.\{4\}//") bash'
You can use custom-columns in output option to get the name and --no-headers option
kubectl get --no-headers=true pods -l app=external-dns -o custom-columns=:metadata.name
You can use -o=name to display only pod names. For example to list proxy pods you can use:
kubectl get pods -o=name --all-namespaces | grep kube-proxy
The result is:
pod/kube-proxy-95rlj
pod/kube-proxy-bm77b
pod/kube-proxy-clc25
There is also this solution:
kubectl get pods -o jsonpath={..metadata.name}
Here is another way to do it:
kubectl get pods -o=name --field-selector=status.phase=Running
The --field-selector=status.phase=Running is needed as the question mention all the running pod names. If the all in the question is for all the namespaces, just add the --all-namespaces option.
Note that this command is very convenient when one want a quick way to access something from the running pod(s), such as logs :
kubectl logs -f $(kubectl get pods -o=name --field-selector=status.phase=Running)
Get all running pods in the namespace
kubectl get pods --field-selector=status.phase=Running --no-headers -o custom-columns=":metadata.name"
From viewing, finding resources.
You could also specify namespace with -n <namespace name>.
jsonpath alternative
kubectl get po -o jsonpath="{range .items[*]}{#.metadata.name}{end}" -l app=nginx-ingress,component=controller
see also:
more examples of kubectl output options
If you want to extract specific container's pod name then
A simple command can do all the hard work
kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' --selector=app=<CONTAINER-NAME>
Just replace <CONTAINER-NAME> with your service container-name
Well, In our case we have kept pods inside different namespace, here to identify the specific pod or list of pods we ran following command-
Approach 1:
To get the list of namespaces
kubectl get ns -A
To get all the pods inside one namespaces kubectl get pods -n <namespace>
Approach 2:
Use this command-
kubectl get pods --all-namespaces
kubectl exec -it $(kubectl get pods | grep mess | awk '{print $1}') /bin/bash
kubectl get po --all-namespaces | awk '{if ($4 != "Running") system ("kubectl -n " $1 " delete pods " $2 " --grace-period=0 " " --force ")}'

How do you cleanly list all the containers in a kubernetes pod?

I am looking to list all the containers in a pod in a script that gather's logs after running a test. kubectl describe pods -l k8s-app=kube-dns returns a lot of info, but I am just looking for a return like:
etcd
kube2sky
skydns
I don't see a simple way to format the describe output. Is there another command? (and I guess worst case there is always parsing the output of describe).
Answer
kubectl get pods POD_NAME_HERE -o jsonpath='{.spec.containers[*].name}'
Explanation
This gets the JSON object representing the pod. It then uses kubectl's JSONpath to extract the name of each container from the pod.
You can use get and choose one of the supported output template with the --output (-o) flag.
Take jsonpath for example,
kubectl get pods -l k8s-app=kube-dns -o jsonpath={.items[*].spec.containers[*].name} gives you etcd kube2sky skydns.
Other supported output output templates are go-template, go-template-file, jsonpath-file. See http://kubernetes.io/docs/user-guide/jsonpath/ for how to use jsonpath template. See https://golang.org/pkg/text/template/#pkg-overview for how to use go template.
Update: Check this doc for other example commands to list container images: https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/
Quick hack to avoid constructing the JSONpath query for a single pod:
$ kubectl logs mypod-123
a container name must be specified for pod mypod-123, choose one of: [etcd kubesky skydns]
I put some ideas together into the following:
Simple line:
kubectl get po -o jsonpath='{range .items[*]}{"pod: "}{.metadata.name}{"\n"}{range .spec.containers[*]}{"\tname: "}{.name}{"\n\timage: "}{.image}{"\n"}{end}'
Split (for readability):
kubectl get po -o jsonpath='
{range .items[*]}
{"pod: "}
{.metadata.name}
{"\n"}{range .spec.containers[*]}
{"\tname: "}
{.name}
{"\n\timage: "}
{.image}
{"\n"}
{end}'
How to list BOTH init and non-init containers for all pods
kubectl get pod -o="custom-columns=NAME:.metadata.name,INIT-CONTAINERS:.spec.initContainers[*].name,CONTAINERS:.spec.containers[*].name"
Output looks like this:
NAME INIT-CONTAINERS CONTAINERS
helm-install-traefik-sjts9 <none> helm
metrics-server-86cbb8457f-dkpqm <none> metrics-server
local-path-provisioner-5ff76fc89d-vjs6l <none> local-path-provisioner
coredns-6488c6fcc6-zp9gv <none> coredns
svclb-traefik-f5wwh <none> lb-port-80,lb-port-443
traefik-6f9cbd9bd4-pcbmz <none> traefik
dc-postgresql-0 init-chmod-data dc-postgresql
backend-5c4bf48d6f-7c8c6 wait-for-db backend
if you want a clear output of which containers are from each Pod
kubectl get po -l k8s-app=kube-dns \
-o=custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name
To get the output in the separate lines:
kubectl get pods POD_NAME_HERE -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
Output:
base-container
sidecar-0
sidecar-1
sidecar-2
If you use json as output format of kubectl get you get plenty details of a pod. With json processors like jq it is easy to select or filter for certain parts you are interested in.
To list the containers of a pod the jq query looks like this:
kubectl get --all-namespaces --selector k8s-app=kube-dns --output json pods \
| jq --raw-output '.items[].spec.containers[].name'
If you want to see all details regarding one specific container try something like this:
kubectl get --all-namespaces --selector k8s-app=kube-dns --output json pods \
| jq '.items[].spec.containers[] | select(.name=="etcd")'
Use below command:
kubectl get pods -o=custom-columns=PodName:.metadata.name,Containers:.spec.containers[*].name,Image:.spec.containers[*].image
To see verbose information along with configmaps of all containers in a particular pod, use this command:
kubectl describe pod/<pod name> -n <namespace name>
Use below command to see all the information of a particular pod
kubectl get pod <pod name> -n <namespace name> -o yaml
For overall details about the pod try following command to get the container details as well
kubectl describe pod <podname>
I use this to display image versions on the pods.
kubectl get pods -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{end}{end}' && printf '\n'
It's just a small modification of script from here, with adding new line to start next console command on the new line, removed commas at the end of each line and listing only my pods, without service pods (e.g. --all-namespaces option is removed).
There are enough answers here but sometimes you want to see a deployment object pods' containers and initContainers. To do that;
1- Retrieve the deployment name
kubectl get deployment
2- Retrieve containers' names
kubectl get deployment <deployment-name> -o jsonpath='{.spec.template.spec.containers[*].name}'
3- Retrieve initContainers' names
kubectl get deployment <deployment-name> -o jsonpath='{.spec.template.spec.initContainers[*].name}'
Easiest way to know the containers in a pod:
kubectl logs -c -n

Command to delete all pods in all kubernetes namespaces

Upon looking at the docs, there is an API call to delete a single pod, but is there a way to delete all pods in all namespaces?
There is no command to do exactly what you asked.
Here are some close matches.
Be careful before running any of these commands. Make sure you are connected to the right cluster, if you use multiple clusters. Consider running. kubectl config view first.
You can delete all the pods in a single namespace with this command:
kubectl delete --all pods --namespace=foo
You can also delete all deployments in namespace which will delete all pods attached with the deployments corresponding to the namespace
kubectl delete --all deployments --namespace=foo
You can delete all namespaces and every object in every namespace (but not un-namespaced objects, like nodes and some events) with this command:
kubectl delete --all namespaces
However, the latter command is probably not something you want to do, since it will delete things in the kube-system namespace, which will make your cluster not usable.
This command will delete all the namespaces except kube-system, which might be useful:
for each in $(kubectl get ns -o jsonpath="{.items[*].metadata.name}" | grep -v kube-system);
do
kubectl delete ns $each
done
kubectl delete daemonsets,replicasets,services,deployments,pods,rc,ingress --all --all-namespaces
to get rid of them pesky replication controllers too.
You can simply run
kubectl delete all --all --all-namespaces
The first all means the common resource kinds (pods, replicasets, deployments, ...)
kubectl get all == kubectl get pods,rs,deployments, ...
The second --all means to select all resources of the selected kinds
Note that all does not include:
non namespaced resourced (e.g., clusterrolebindings, clusterroles, ...)
configmaps
rolebindings
roles
secrets
...
In order to clean up perfectly,
you could use other tools (e.g., Helm, Kustomize, ...)
you could use a namespace.
you could use labels when you create resources.
You just need sed to do this:
kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete pod \2/e'
Explains:
use command kubectl get pods --all-namespaces to get the list of all pods in all namespaces.
use --no-headers=true option to hide the headers.
use s command of sed to fetch the first two words, which represent namespace and pod's name respectively, then assemble the delete command using them.
the final delete command is just like:
kubectl --namespace kube-system delete pod heapster-eq3yw.
use the e modifier of s command to execute the command assembled above, which will do the actual delete works.
To avoid delete pods in kube-system namespace, just need to add grep -v kube-system to exclude kube-system namespace before the sed command.
I tried commands from listed answers here but pods were stuck in terminating state.
I found below command to delete all pods from particular namespace if stuck in terminating state or you are not able to delete it then you can delete pods forcefully.
kubectl delete pods --all --grace-period=0 --force --namespace namespace
Hope it might be useful to someone.
K8s completely works on the fundamental of the namespace. if you like to release all the resource related to specified namespace.
you can use the below mentioned :
kubectl delete namespace k8sdemo-app
steps to delete pv:
delete all deployment and pods or resources related to that PV
kubectl delete --all deployment -n namespace
kubectl delete --all pod -n namespace
edit pv
kubectl edit pv pv_name -n namespace
remove kubernetes.io/pv-protection
delete pv
kubectl delete pv pv_name -n namespace
Delete all PODs in all Namespace only (restart deployment)
kubectl get pod -A -o yaml | kubectl delete -f -
You can use kubectl delete pods -l dev-lead!=carisa or what label you have.
Here is a one-liner that can be extended with grep to filter by name.
kubectl get pods -o jsonpath="{.items[*].metadata.name}" | \
tr " " "\n" | \
xargs -i -P 0 kubectl delete pods {}
One line command to delete all pods in all namespaces.
kubectl get ns -o=custom-columns=Namespace:.metadata.name --no-headers | xargs -n1 kubectl delete pods --all -n
kubectl delete po,ing,svc,pv,pvc,sc,ep,rc,deploy,replicaset,daemonset --all -A
If you already have pods which are recreated, think to delete all deployments first
kubectl delete -n *NAMESPACE deployment *DEPLOYMENT
Just replace the NAMSPACE and the DEPLOYMENT to corresponding ones, you can get all deployments information by the following command
kubectl get deployments --all-namespaces
Kubectl bulk (bulk-action on krew) plugin may be useful for you, it gives you bulk operations on selected resources. This is the command for deleting pods
' kubectl bulk pods -n namespace delete '
You could check details in this
I create a python code to delete all in namespace
delall.py
import json,sys,os;
obj=json.load(sys.stdin);
for item in obj["items"]:
os.system("kubectl delete " + item["kind"] + "/" +item["metadata"]["name"] + " -n yournamespace")
and then
kubectl get all -n kong -o json | python delall.py
If you have multiple pod which are crashing or error and you want to delete them
kubectl delete pods --all -n | gep
It was hinted at above, but I just thought I would helpfully point out that the shortcut for "--all-namespaces" is "-A" that's with a capital A. HTH somebody. I've opened a PR to have this helpful hint added to the official Kubernetes Cheat Sheet.
If you want to delete pods in all namespaces just to have them restarted and you are aware that some of them will be recreated, I like the following for loop:
for i in $(kubectl get pods -A | awk '{print $1}' | uniq | grep -V NAMESPACE); do kubectl delete --all pods -n $i; done
if you have hpa, then scale down.