Is there a way to use kubectl to list only the pods belonging to a deployment?
Currently, I do this to get pods:
kubectl get pods| grep hello
But it seems an overkill to get ALL the pods when I am interested to know only the pods for a given deployment. I use the output of this command to see the status of all pods, and then possibly exec into one of them.
I also tried kc get -o wide deployments hellodeployment, but it does not print the Pod names.
There's a label in the pod for the selector in the deployment. That's how a deployment manages its pods. For example for the label or selector app=http-svc you can do something like that this and avoid using grep and listing all the pods (this becomes useful as your number of pods becomes very large)
here are some examples command line:
# single label
kubectl get pods -l=app=http-svc
kubectl get pods --selector=app=http-svc
# multiple labels
kubectl get pods --selector key1=value1,key2=value2
K8s components are linked to each other by labels and selectors. There are just no built-in attributes of My-List-of-ReplicaSets or My-List-Of-Pods for a deployment. You can't get them from kubectl describe or kubectl get
As #Rico suggested above, you have to use label filters. But you can't simply use the labels that you specify in the deployment metafile because deployment will generate a random hash and use it as an additional label.
For example, I have a deployment and a standalone pod that share the same label app=http-svc. While the first two are managed by the deployment, the 3rd one is not and shouldn't be in the result.
ma.chi#~/k8s/deployments % kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
http-9c89b5578-6cqbp 1/1 Running 0 7s app=http-svc,pod-template-hash=574561134
http-9c89b5578-vwqbx 1/1 Running 0 7s app=http-svc,pod-template-hash=574561134
nginx-standalone 1/1 Running 0 7s app=http-svc
ma.chi#~/k8s/deployments %
The source file is
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: http-svc
name: http
spec:
replicas: 2
selector:
matchLabels:
app: http-svc
strategy: {}
template:
metadata:
labels:
app: http-svc
spec:
containers:
- image: nginx:1.9.1
name: nginx1
---
apiVersion: v1
kind: Pod
metadata:
labels:
app: http-svc
name: nginx-standalone
spec:
containers:
- image: nginx:1.9.1
name: nginx1-standalone
To exact spot the containers created and managed by your deployment, you can use the script below(which is ugly, but this is the best I can do)
DEPLOY_NAME=http
RS_NAME=`kubectl describe deployment $DEPLOY_NAME|grep "^NewReplicaSet"|awk '{print $2}'`; echo $RS_NAME
POD_HASH_LABEL=`kubectl get rs $RS_NAME -o jsonpath="{.metadata.labels.pod-template-hash}"` ; echo $POD_HASH_LABEL
POD_NAMES=`kubectl get pods -l pod-template-hash=$POD_HASH_LABEL --show-labels | tail -n +2 | awk '{print $1}'`; echo $POD_NAMES
Here's a tidier shell alias (based on this code by #kekaoyunfuwu) that only lists the pods of a deployment (no interim results are shown):
k_list_pods_in_deployment() (
test $# -eq 0 && {
echo "Missing deployment name" && kubectl get deployments
return 1
}
deployment="$1"; shift
replicaSet="$(kubectl describe deployment $deployment \
| grep '^NewReplicaSet' \
| awk '{print $2}'
)"
podHashLabel="$(kubectl get rs $replicaSet \
-o jsonpath='{.metadata.labels.pod-template-hash}'
)"
kubectl get pods -l pod-template-hash=$podHashLabel --show-labels \
| tail -n +2 | awk '{print $1}'
)
alias k.list-pods-in-deployment=k_list_pods_in_deployment
Related
Let's say I've a deployment. For some reason it's not responding after sometime. Is there any way to tell Kubernetes to rollback to previous version automatically on failure?
You mentioned that:
I've a deployment. For some reason it's not responding after sometime.
In this case, you can use liveness and readiness probes:
The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
The above probes may prevent you from deploying a corrupted version, however liveness and readiness probes aren't able to rollback your Deployment to the previous version. There was a similar issue on Github, but I am not sure there will be any progress on this matter in the near future.
If you really want to automate the rollback process, below I will describe a solution that you may find helpful.
This solution requires running kubectl commands from within the Pod.
In short, you can use a script to continuously monitor your Deployments, and when errors occur you can run kubectl rollout undo deployment DEPLOYMENT_NAME.
First, you need to decide how to find failed Deployments. As an example, I'll check Deployments that perform the update for more than 10s with the following command:
NOTE: You can use a different command depending on your need.
kubectl rollout status deployment ${deployment} --timeout=10s
To constantly monitor all Deployments in the default Namespace, we can create a Bash script:
#!/bin/bash
while true; do
sleep 60
deployments=$(kubectl get deployments --no-headers -o custom-columns=":metadata.name" | grep -v "deployment-checker")
echo "====== $(date) ======"
for deployment in ${deployments}; do
if ! kubectl rollout status deployment ${deployment} --timeout=10s 1>/dev/null 2>&1; then
echo "Error: ${deployment} - rolling back!"
kubectl rollout undo deployment ${deployment}
else
echo "Ok: ${deployment}"
fi
done
done
We want to run this script from inside the Pod, so I converted it to ConfigMap which will allow us to mount this script in a volume (see: Using ConfigMaps as files from a Pod):
$ cat check-script-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: check-script
data:
checkScript.sh: |
#!/bin/bash
while true; do
sleep 60
deployments=$(kubectl get deployments --no-headers -o custom-columns=":metadata.name" | grep -v "deployment-checker")
echo "====== $(date) ======"
for deployment in ${deployments}; do
if ! kubectl rollout status deployment ${deployment} --timeout=10s 1>/dev/null 2>&1; then
echo "Error: ${deployment} - rolling back!"
kubectl rollout undo deployment ${deployment}
else
echo "Ok: ${deployment}"
fi
done
done
$ kubectl apply -f check-script-configmap.yml
configmap/check-script created
I've created a separate deployment-checker ServiceAccount with the edit Role assigned and our Pod will run under this ServiceAccount:
NOTE: I've created a Deployment instead of a single Pod.
$ cat all-in-one.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployment-checker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: deployment-checker-binding
subjects:
- kind: ServiceAccount
name: deployment-checker
namespace: default
roleRef:
kind: ClusterRole
name: edit
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: deployment-checker
name: deployment-checker
spec:
selector:
matchLabels:
app: deployment-checker
template:
metadata:
labels:
app: deployment-checker
spec:
serviceAccountName: deployment-checker
volumes:
- name: check-script
configMap:
name: check-script
containers:
- image: bitnami/kubectl
name: test
command: ["bash", "/mnt/checkScript.sh"]
volumeMounts:
- name: check-script
mountPath: /mnt
After applying the above manifest, the deployment-checker Deployment was created and started monitoring Deployment resources in the default Namespace:
$ kubectl apply -f all-in-one.yaml
serviceaccount/deployment-checker created
clusterrolebinding.rbac.authorization.k8s.io/deployment-checker-binding created
deployment.apps/deployment-checker created
$ kubectl get deploy,pod | grep "deployment-checker"
deployment.apps/deployment-checker 1/1 1
pod/deployment-checker-69c8896676-pqg9h 1/1 Running
Finally, we can check how it works. I've created three Deployments (app-1, app-2, app-3):
$ kubectl create deploy app-1 --image=nginx
deployment.apps/app-1 created
$ kubectl create deploy app-2 --image=nginx
deployment.apps/app-2 created
$ kubectl create deploy app-3 --image=nginx
deployment.apps/app-3 created
Then I changed the image for the app-1 to the incorrect one (nnnginx):
$ kubectl set image deployment/app-1 nginx=nnnginx
deployment.apps/app-1 image updated
In the deployment-checker logs we can see that the app-1 has been rolled back to the previous version:
$ kubectl logs -f deployment-checker-69c8896676-pqg9h
...
====== Thu Oct 7 09:20:15 UTC 2021 ======
Ok: app-1
Ok: app-2
Ok: app-3
====== Thu Oct 7 09:21:16 UTC 2021 ======
Error: app-1 - rolling back!
deployment.apps/app-1 rolled back
Ok: app-2
Ok: app-3
I stumbled upon Argo Rollout which addresses this non automatic rollback and many other deployment related things.
https://argoproj.github.io/argo-rollouts/
I have created a replicaset with wrong container image with below configuration.
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: rs-d33393
namespace: default
spec:
replicas: 4
selector:
matchLabels:
name: busybox-pod
template:
metadata:
labels:
name: busybox-pod
spec:
containers:
- command:
- sh
- -c
- echo Hello Kubernetes! && sleep 3600
image: busyboxXXXXXXX
name: busybox-container
Pods Information:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
rs-d33393-5hnfx 0/1 InvalidImageName 0 11m
rs-d33393-5rt5m 0/1 InvalidImageName 0 11m
rs-d33393-ngw78 0/1 InvalidImageName 0 11m
rs-d33393-vnpdh 0/1 InvalidImageName 0 11m
After this, i try to edit the image inside replicaset using kubectl edit replicasets.extensions rs-d33393 and update image as busybox.
Now, i am expecting pods to be recreated with proper image as part of replicaset.
This has not been the exact result.
Can someone please explain, why it is so?
Thanks :)
With ReplicaSets directly you have to kill the old pod, so the new ones will be created with the right image.
If you would be using a Deployment, and you should, changing the image would force the pod to be re-created.
Replicaset does not support updates. As long as required number of pods exist matching the selector labels, replicaset's jobs is done. You should use Deployment instead.
https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
From the docs:
To update Pods to a new spec in a controlled way, use a Deployment, as
ReplicaSets do not support a rolling update directly.
Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods. Therefore, it is recommend to use Deployments instead of directly using ReplicaSets unless you don’t require updates at all. ( i.e. one may never need to manipulate ReplicaSet objects when using a Deployment)
Its easy to perform rolling updates and rollbacks when deployed using deployments.
$ kubectl create deployment busybox --image=busyboxxxxxxx --dry-run -o yaml > busybox.yaml
$ cat busybox.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: busybox
name: busybox
spec:
replicas: 1
selector:
matchLabels:
app: busybox
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: busybox
spec:
containers:
- image: busyboxxxxxxx
name: busyboxxxxxxx
ubuntu#dlv-k8s-cluster-master:~$ kubectl create -f busybox.yaml --record=true
deployment.apps/busybox created
Check rollout history
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
Update image on deployment
ubuntu#dlv-k8s-cluster-master:~$ kubectl set image deployment.app/busybox *=busybox --record
deployment.apps/busybox image updated
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
2 kubectl set image deployment.app/busybox *=busybox --record=true
Rollback Deployment
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout undo deployment busybox
deployment.apps/busybox rolled back
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
2 kubectl set image deployment.app/busybox *=busybox --record=true
3 kubectl create --filename=busybox.yaml --record=true
You could use
k scale rs new-replica-set --replicas=0
and then
k scale rs new-replica-set --replicas=<Your number of replicas>
Edit the replicaset(assuming its called replicaset.yaml) file with command:
kubectl edit rs replicaset
edit the image name in the editor
save the file
exit the editor
Now , you will need to either delete the replica sets or delete the existing pods:
kubectl delete rs new-replica-set
kubectl delete pod pod_1 pod_2 pod_2 pod_4
replicaset should spin up new pods with new image.
I am trying to define the namespace name when executing the kubectl create deployment command?
This is what I tried:
kubectl create deployment test --image=banu/image1 namespace=test
and this doesn't work.
And I want to expose this deployment using a ClusterIP service within the cluster itself for that given namespace How can I do that using kubectl command line?
You can specify either -n or --namespace options.
kubectl create deployment test --image=nginx --namespace default --dry-run -o yaml and see result deployment yaml.
Using kubectl run
kubectl run test --namespace test --image nginx --port 9090 --dry-run -o yaml
You need to create a namespace like this
kubectl create ns test
ns stands for namespace, so with kubectl you say you want to create namespace with name test
Then while you creating the deployment you add the namespace you want
kubectl create deployment test --image=banu/image1 -n test
Where flag -n stands for namespace, that way you say to Kubernetes that all resources related to that deployment will be under the test namespace
In order to see all the resources under a specific namespace
kubectl get all -n test
--namespace and -n is the same things
Use -n test instead of namespace=test
Sample with nginx image:
$ kubectl create deployment nginx --image=nginx -n test
deployment.apps/nginx created
$ kubectl get deploy -n test
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 8s
In second case you need to create service and define labels from deployment.
You can find correct labels by runnig something like:
kubectl -n test describe deploy test |grep Labels:
and apply service like:
apiVersion: v1
kind: Service
metadata:
name: test-svc
namespace: test
spec:
ports:
- name: test
port: 80 # Change this port
protocol: TCP
type: ClusterIP
selector:
# Here you need to define output from previous step
We're using a >1.8 version of k8s on gcloud. Unfortunately EventStore stops pushing data until it is rebooted. Thus we'd like to run kubectl --namespace=$NAMESPACE delete pod eventstore-0 every 6 hours. Thus we have a cron job like:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: eventstore-restart
spec:
# Run every full hour, 15 past, 30 past, 45 past every other time-unit.
schedule: "0,15,30,45 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
containers:
- name: eventstore-restart
image: eu.gcr.io/$PROJECT_ID/kubectl:latest
imagePullPolicy: Always
command: [ "/bin/sh", "-c" ]
args:
- 'set -x; kubectl --namespace=$NAMESPACE get pods
| grep -ho "eventstore-\d+"
| xargs -n 1 -I {} kubectl --namespace=$NAMESPACE delete pod {}'
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccount: restart-eventstore
However, this seems to expand to kubectl get pods ..., piped with | { ... }, which causes "/bin/sh: syntax error: unexpected end of file (expecting "}") to fail the script.
How do I write the command to delete a pod on a schedule?
I would do this:
kubectl delete po $(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep eventstore) -n $NAMESPACE
or (your way)
kubectl get pods -n $NAMESPACE -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep eventstore | xargs -n 1 -I {} kubectl delete po {}
Now, if you know you want to delete pod "eventstore-0", why to not do directly kubectl delete pod eventstore-0?
I suggest to use label selectors to filter results of kubectl get, and jsonpath output to get just the name of the pod.
Assuming that your pod is labeled with app=eventstore and that you want to delete every pod with this label, you could use the following command:
k get po --namespace=$NAMESPACE --selector app=eventstore -o jsonpath="{.items[*].metadata.name}" | xargs -n 1 -I {} kubectl --namespace=$NAMESPACE delete po {}
If you want to delete just the first pod, use jsonpath="{.items[0].metadata.name}"
#suren answer is good, but it won't work in all the cases when you want to delete multiple specific pods. For example, if you have: pod1, pod2, pod3, pod4, and you want to delete only pod2 and pod4, you can't do it with grep (grep pod will catch all of them).
In that case, you can delete them like that:
kubectl delete pods -n $NAMESPACE $(echo -e 'pod2\npod4')
I'm setting up a Kubernetes cluster and am testing a small container. This is my YAML file for the pod:
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
restartPolicy: Never
containers:
- name: node
image: 'node:5'
command: ['node']
args: ['-e', 'console.log(1234)']
I deploy it with kubectl create -f example.yml and sure enough it runs as expected:
$ kubectl logs example
1234
However, the pod's status is "Error":
$ kubectl get po example
NAME READY STATUS RESTARTS AGE
example 0/1 Error 0 16m
How can I investigate why the status is "Error"?
kubectl describe pod example will give you more info on what's going on
also
kubectl get events can get you more details too although not dedicated to the given pod.