I'd need to force moving a Pod from one Openshift Node to another, in order to do some performance tests. From the documentation it seems that setting the nodeSelector in the Deployment config is the way to go, but it doesn't work, according to my tests.
Here is what I have tried as a test:
Create nginx Pod
oc new-app -f https://raw.githubusercontent.com/sclorg/nginx-ex/master/openshift/templates/nginx.json
The Pod is running on "ip-10-0-121-229.us-east-2.compute.internal" Node.
Now I patch the node selector, setting a target Node:
oc patch dc nginx-example -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname": "ip-10-0-169-74.us-east-2.compute.internal"}}}}}'
However the Pod is still running on the same Node. Even by killing the Pod it re-attaches there.
Any suggestion?
Thanjs
you can try this -
kubectl get pod -o wide
this will give you the VM on which your pod is running
then execute
kubectl cordon {name_of_that_node_in_which_POD_is_running}
then delete the pods, those you want to get aligned to other node
then run
kubectl uncordon {the_node_that_was_cordoned_above}
You can use nodeName field in your pod spec to schedule the pod yourself, rather than scheduler do it.
oc explain pod.spec.nodeName
FIELD: nodeName <string>
DESCRIPTION:
NodeName is a request to schedule this pod onto a specific node. If it is
non-empty, the scheduler simply schedules this pod onto that node, assuming
that it fits resource requirements.
you can patch it in similar way. Don't forget to delete nodeSelctor fields
oc patch dc nginx-example -p '{"spec":{"template":{"spec":{"nodeName": "ip-10-0-169-74.us-east-2.compute.internal"}}}}'
Related
I have kubernets cluster in gcp with docker container runtime. I am trying to change docker container runtime into containerd. Following steps shows what I did.
New node pool added ( nodes with containerd )
drained old nodes
Once I perform above steps I am getting " Pod is blocking scale down because it has local storage " warning message.
You need to add the once annotation to POD so that cluster autoscaler can remove that POD from POD safe to evict.
cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
above annotation, you have to add in into POD.
You can read more at : https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#cluster-not-scalingdown
NoScaleDown example: You found a noScaleDown event that contains a
per-node reason for your node. The message ID is
"no.scale.down.node.pod.has.local.storage" and there is a single
parameter: "test-single-pod". After consulting the list of error
messages, you discover this means that the "Pod is blocking scale down
because it requests local storage". You consult the Kubernetes Cluster
Autoscaler FAQ and find out that the solution is to add a
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to
the Pod. After applying the annotation, cluster autoscaler scales down
the cluster correctly.
For further clarification, you can use this command to update the pod's annotation:
kubectl annotate pod <podname> -n <namespace> "cluster-autoscaler.kubernetes.io/safe-to-evict=true"
Had the same error when using Gitlab + Autodevops + GoogleCloud.
The issue is the cm_acme pods's that are spun up to answer the letsencrypt challenges.
e.g. we have pods like this
cm-acme-http-solver-d2tak
hanging around in our cluster so the cluster won't downsize until these pods are destroyed.
A simple
kubectl get pods -A | grep cm-acme
will list all the pods that need to be destroyed with
kubectl delete pod -n {namespace} {pod name}
I have 3 nodes in kubernetes cluster. I create a daemonset and deployed it in all the 3 devices. This daemonset created 3 pods and they were successfully running. But for some reasons, one of the pod failed.
I need to know how can we restart this pod without affecting other pods in the daemon set, also without creating any other daemon set deployment?
Thanks
kubectl delete pod <podname> it will delete this one pod and Deployment/StatefulSet/ReplicaSet/DaemonSet will reschedule a new one in its place
There are other possibilities to acheive what you want:
Just use rollout command
kubectl rollout restart deployment mydeploy
You can set some environment variable which will force your deployment pods to restart:
kubectl set env deployment mydeploy DEPLOY_DATE="$(date)"
You can scale your deployment to zero, and then back to some positive value
kubectl scale deployment mydeploy --replicas=0
kubectl scale deployment mydeploy --replicas=1
Just for others reading this...
A better solution (IMHO) is to implement a liveness probe that will force the pod to restart the container if it fails the probe test.
This is a great feature K8s offers out of the box. This is auto healing.
Also look into the pod lifecycle docs.
kubectl -n <namespace> delete pods --field-selector=status.phase=Failed
I think the above command is quite useful when you want to restart 1 or more failed pods :D
And we don't need to care about name of the failed pod.
I am starting exploring runnign docker containers with Kubernetes. I did the following
Docker run etcd
docker run master
docker run service proxy
kubectl run web --image=nginx
To cleanup the state, I first stopped all the containers and cleared the downloaded images. However I still see pods running.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-3476088249-w66jr 1/1 Running 0 16m
How can I remove this?
To delete the pod:
kubectl delete pods web-3476088249-w66jr
If this pod is started via some replicaSet or deployment or anything that is creating replicas then find that and delete that first.
kubectl get all
This will list all the resources that have been created in your k8s cluster. To get information with respect to resources created in your namespace kubectl get all --namespace=<your_namespace>
To get info about the resource that is controlling this pod, you can do
kubectl describe web-3476088249-w66jr
There will be a field "Controlled By", or some owner field using which you can identify which resource created it.
When you do kubectl run ..., that's a deployment you create, not a pod directly. You can check this with kubectl get deploy. If you want to delete the pod, you need to delete the deployment with kubectl delete deploy DEPLOYMENT.
I would recommend you to create a namespace for testing when doing this kind of things. You just do kubectl create ns test, then you do all your tests in this namespace (by adding -n test). Once you have finished, you just do kubectl delete ns test, and you are done.
If you defined your object as Pod then
kubectl delete pod <--all | pod name>
will remove all of the generated Pod. But, If wrapped your Pod to Deployment object then running the command above only will trigger a re-creation of them.
In that case, you need to run
kubectl delete deployment <--all | deployment name>
That will also remove the Service object that is related to the deleted Deployment
I'm unable to delete the kubernetes pod, it keeps recreating it.
There's no service or deployment associated with the pod. There's a label on the pod thou, is that the root cause?
If I edit the label out with kubectl edit pod podname it removes the label from the pod, but creates a new pod with the same label at the same time. ¿?
Pod can be created by ReplicationControllers or ReplicaSets. The latter one might be created by an Deployment. The described behavior strongly indicates, that the Pod is managed by either of these two.
You can check for these with this commands:
kubectl get rs
kubectl get rc
I have installed Kubernetes in Ubuntu server using instructions here. I am trying to create pods using kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --hostport=8000 --port=8080 as listed in the example. However, when I do kubectl get pod I get the status of the container as pending. I further did kubectl describe pod for debugging and I see the message:
FailedScheduling pod (hello-minikube-3383150820-1r4f7) failed to fit in any node fit failure on node (minikubevm): PodFitsHostPorts.
I am further trying to delete this pod by kubectl delete pod hello-minikube-3383150820-1r4f7 but when I further do kubectl get pod I see another pod with prefix "hello-minikube-3383150820-" that I havent created. Does anyone know how to fix this problem? Thank you in advance.
The PodFitsHostPorts predicate is failing because you have something else on your nodes using port 8000. You might be able to find what it is by running kubectl describe svc.
kubectl run creates a deployment object (you can see it with kubectl describe deployments) which makes sure that you always keep the intended number of replicas of the pod running (in this case 1). When you delete the pod, the deployment controller automatically creates another for you. If you want to delete the deployment and the pods it keeps creating, you can run kubectl delete deployments hello-minikube.