I would like to perform rolling back a deployment in my environment.
Command:
kubectl rollout undo deployment/foo
Steps which are perform:
create pods with old configurations
delete old pods
Is there a way to not perform last step - for example - developer would like to check why init command fail and debug.
I didn't find information about that in documentation.
Yes it is possible, before doing rollout, first you need to remove labels (corresponding to replica-set controlling that pod) from unhealthy pod. This way pod won't belong anymore to the deployment and even if you do rollout, it will still be there. Example:
$kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
sleeper 1/1 1 1 47h
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 5m46s pod-template-hash=d75b55fc9,run=sleeper
$kubectl label pod sleeper-d75b55fc9-87k5k pod-template-hash- run-
pod/sleeper-d75b55fc9-87k5k labeled
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 6m34s <none>
sleeper-d75b55fc9-swkj9 1/1 Running 0 3s pod-template-hash=d75b55fc9,run=sleeper
So what happens here, we have a pod sleeper-d75b55fc9-87k5k which belongs to sleeper deployment, we remove all labels from it, deployment detects that pod "has gone" so it creates a new one sleeper-d75b55fc9-swkj9, but the old one is still there and ready for debugging. Only pod sleeper-d75b55fc9-swkj9 will be affected by rollout.
Related
We have a data visualization server hosted in Kubernetes pods. The dashboards in that data viz are displayed in the browser of different monitors/terminals for near-real time operational reporting. Sometimes the pods fail, and when they come alive again, the browser redirects to Single Sign-On page instead of going to the dashboard the URL is originally configured to.
The server are hosted in I would presume a replica set. There are two pods that exist as far as I can tell.
I was granted privilege on using kubectl to solve this problem, but still quite new with the whole Kubernetes thing. Using kubectl, how do I simulate pod failure/restart for testing purposes? Since the pods are in duplicate, shutting one of them will only redirect the traffic to the other pod. How to make both pods fail/restart at the same time? (I guess doing kubectl delete pod on both pods will do, but I want to make sure k8s will respawn the pods automatically, and not delete them forever).
If I understand the use case correctly, you might want to use kubectl scale command. This will give you the flexibility to make the replica count to zero to N by running a simple kubectl scale command. See examples. Also, if you are using deployment, you can just do the kubectl delete pod, the deployment controller will spawn a new one to satisfy the replica count.
kubectl scale deployment/<DEPLOYMENT-NAME> --replicas=<DESIRED-NUMBER-OF-REPLICA>
short example:
kubectl scale deployment/deployment-web --replicas=0
deployment.apps/deployment-web scaled
Long Example:
// create a deployment called, deployment-web with two replicas.
kubectl create deployment deployment-web --image=nginx --replicas 2
deployment.apps/deployment-web created
// verify that both replicas are up
kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
deployment-web 2/2 2 2 13s
// expose the deployment with a service [OPTIONAL-STEP, ONLY FOR EXPLANATION]
kubectl expose deployment deployment-web --port 80
service/deployment-web exposed
//verify that the service is created
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deployment-web ClusterIP 10.233.24.174 <none> 80/TCP 5s
// dump the list of end-points for that service, there would be one for each replica. Notice the two IPs in the 2nd column.
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.111.6:80,10.233.115.9:80 12s
//scale down to 1 replica for the deployment
kubectl scale --current-replicas=2 --replicas=1 deployment/deployment-web
deployment.apps/deployment-web scaled
// Notice the endpoint is reduced from 2 to 1.
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.115.9:80 43s
// also note that there is only one pod remaining
kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-web-64c769b44-qh2qf 1/1 Running 0 105s
// scale down to zero replica
kubectl scale --current-replicas=1 --replicas=0 deployment/deployment-web
deployment.apps/deployment-web scaled
// The endpoint list is empty
kubectl get ep
NAME ENDPOINTS AGE
deployment-web <none> 9m4s
//Also, both pods are gone
kubectl get pod
No resources found in default namespace.
// When you are done with testing. restore the replicas
kubectl scale --current-replicas=0 --replicas=2 deployment/deployment-web
deployment.apps/deployment-web scaled
//endpoints and pods are restored back
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.111.8:80,10.233.115.11:80 10m
foo-svc 10.233.115.6:80 50m
kubernetes 192.168.22.9:6443 6d23h
kubectl get pod -l app=deployment-web
NAME READY STATUS RESTARTS AGE
deployment-web-64c769b44-b72k5 1/1 Running 0 8s
deployment-web-64c769b44-mt2dd 1/1 Running 0 8s
I want to remove zk and kafka from my k8s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka1-mvzch 1/1 Running 1 25s
kafka2-m292k 0/1 CrashLoopBackOff 8 20m
zookeeper1-qhmnf 1/1 Running 0 20m
zookeeper2-t7r8w 1/1 Running 0 20m
$kubectl delete pod kafka1-mvzch kafka2-m292k zookeeper1-qhmnf zookeeper2-t7r8w
pod "kafka1-mvzch" deleted
pod "kafka1-m292k" deleted
pod "zookeeper1-qhmnf" deleted
pod "zookeeper2-t7r8w" deleted
but when I run get pods, it still shows the pods.
And I got no service and deployment
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 7h1m
$ kubectl get deployment
No resources found in default namespace.
You are removing the pods, and they will be deleted.
But there is some other construct that re-creates pods to replace the (now deleted) previous pods.
In fact, the names of the pods with the random-looking suffix suggest that there is another controller operating the pods.
When looking at the linked tutorial, you notice that a ReplicationController is created. This ensures the pods.
If you want to remove it, remove the replication controller; the pods will be deleted as well.
You can use kubectl get pod -ojsonpath='{.metadata.ownerReferences}' to identify the owner object of the pods. The owner might be a Deployment, StatefulSet, etc.
Looking at the medium.com guide that you mentioned, I see that they suggest to create ReplicationControllers.
You can cleanup your namespace by running kubectl delete replicationcontroller --all.
I have 2 pods running on default namespace as shown below
NAMESPACE NAME READY STATUS RESTARTS AGE
default alpaca-prod 1/1 Running 0 36m
default alpaca-test 1/1 Running 0 4m26s
kube-system coredns-78fcd69978-xd7jw 1/1 Running 0 23h
But when I try to get deployments I do not see any
kubectl get deployments
No resources found in default namespace.
Can someone explain this behavior ?
I am running k8 on Minikube.
I think these are pods which were spawned without Deployment, StatefulSet or DaemonSet.
You can run pod like this using the command, e.g.:
kubectl run nginx-test --image=nginx -n default
pods created via DaemonSet usually end with -xxxxx
pods created via Deployment usually end with -xxxxxxxxxx-xxxxx
pods created via StatefulSet usually end with -0, -1 etc.
pods created without upper resource, usually have exact name as you specified e.g. nginx-test, nginx, etc.
So my guess that is a standalone Pod resource (last option)
I'm trying to get information for a cron job so I can grab the current release of service.
So when I run kubectl get pods I get:
NAME READY STATUS RESTARTS AGE
cron-backfill-1573451940-jlwwj 0/1 Completed 0 33h
test-pod-66df8ccd5f-jvmkp 1/1 Running 0 16h
When I run kubectl get pods --selector=job-name=cron-backfill I get:
No resources found in test namespace.
But when I run kubectl get pods --selector=app=test-pod I get:
NAME READY STATUS RESTARTS AGE
test-pod-66df8ccd5f-jvmkp 1/1 Running 0 16h
which is what I want. I figured since the first pod is a cron job there must be some other command used to check for those, but no luck.
I tried looking through the k8s docs here https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ but can't find something that seems to work.
You need to
kubectl describe pods cron-backfill-1573451940-jlwwj
And then you can see the Labels: part
EX:
Labels: app=<app-name>
controller-uid=<xxxxxxxxxx>
job-name=cron-backfill-1573451940-jlwwj
release=<release-name>
Final you can use following command to get your pods:
kubectl get pods --selector=job-name=cron-backfill-1573451940-jlwwj
Hope this may help you, Guy!
If I run this:
kubectl get pods -n kube-system
I get this output:
NAME READY STATUS RESTARTS AGE
coredns-6fdd4f6856-6bl64 0/1 Pending 0 1h
coredns-6fdd4f6856-xgrbm 0/1 Pending 0 1h
kubernetes-dashboard-65c76f6c97-c69jg 0/1 Pending 0 13m
supposedly I need a kubernetes scheduler in order to actually launch containers? Does anyone know how to initiate a kube-scheduler?
More than a Kubernetes scheduler issue, it looks like it's more about not having enough resources on your nodes (or no nodes at all) in your cluster to schedule any workloads. You can check your nodes with:
$ kubectl get nodes
Also, you are not likely able to see any control plane resource on the kube-system namespace because you may be using managed services like EKS or GKE.