I accidentally deleted the kubernetes svc:
service "kubernetes" deleted
using:
kubectl delete svc --all
what should I do? I was just trying to remove services so I could launch new ones.
A bit theory first ;)
Whenever you delete kubernetes svc, you also delete endpoint and this is where
Reconciler comes in. It is actually a controller manager for the core bootstrap Kubernetes controller loops, which manage creating the "kubernetes" service, the
"default", "kube-system" and "kube-public" namespaces, and provide the IP repair check on service IPs.
So, in healthy clusters default.kubernetes service should be automatically recreated by controller manager.
If it's not, I'd recommend to:
Check api-server logs
kubectl logs -f kube-apiserver-master -n kube-system
You should see something like:
Resetting endpoints for master service "kubernetes" to [10.156.0.3]
If you don't see it, try to manually remove etcd key for this service
Because the current state of the cluster is stored in etcd, it may happen that the key remain when you deleted a service:
a. exec to etcd-master pods
kubectl exec -it etcd-master -n kube-system sh
b. get the etcd key value
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt get /registry/services/endpoints/default/kubernetes
c. if you get any value like:
v1 Endpointst
O
kubernetesdefault"*$eafc04cf-90f3-11e9-a75e-42010a9c00032����z!
10.156.0.3
https�2TCP"
just remove it by
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt rm /registry/services/endpoints/default/kubernetes
After you did it, check the api-server logs once again.
ubuntu#master:~$ kubectl delete all --all
service "kubernetes" deleted
ubuntu#master:~$ kubectl get svc
Command 'kubernetes' not found, but can be installed with:
sudo apt install kubernetes
ubuntu#master:~$ sudo apt install kubernetes
ran the installation and it was restored.
Related
I have a k3s cluster with following pods:
kube-system pod/calico-node-xxxx
kube-system pod/calico-kube-controllers-xxxxxx
kube-system pod/metrics-server-xxxxx
kube-system pod/local-path-provisioner-xxxxx
kube-system pod/coredns-xxxxx
How can I reset (stop and start the pods again) the pods either with command (kubectl maybe) or any script?
To reset a pod, you can just delete it. If it's managed by deployment (pods in your question should be), they should be recreated automatically.
kubectl delete pod <pod-name> <pod2-name> ... -n <namespace>
If the pods you want to reset, have common label, you can filter them with --selector flag
kubectl delete pods --selector=<label-name>=<label-value> -n <namespace>
However, if you changed the deployments somehow, you will need to apply the unmodified manifest.
kubectl apply -f <yaml-file>
Warning: - This will reset your whole cluster and delete all running data.
This is not the exact answer but best answer. take 1 min only.
Just uninstall by running below command
sudo /usr/local/bin/k3s-uninstall.sh
Then install a fresh cluster with below command
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --disable=traefik" sh -
Then export var using below command
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
Also it may complain about some k3s config file access so
sudo chmod 444 /etc/rancher/k3s/k3s.yaml
I have removed and delete a node from k8s cluster using the following commands:
kubectl drain worker1 --ignore-daemonsets
kubectl delete worker1
After that, I saw the kube-proxy and the weave daemonset(both for worker1) still existed (it is expected since I ignored the daemonset)even the nodes is drained and deleted.
How can I remove these pods if the node(worker1) is drained and deleted.
Thank you
Find out the name of the pod which is scheduled on that deleted node and delete the pod using kubectl delete pods <pod_name> --grace-period=0 --force -n <namespace>
Use below command to display more details about pod including the node on which the pod is scheduled
kubectl get pods -n <namespace> -o wide
You could also use kubeadm reset on that node. Please note this will uninstall and remove all Kubernetes related software from that node.
I am starting exploring runnign docker containers with Kubernetes. I did the following
Docker run etcd
docker run master
docker run service proxy
kubectl run web --image=nginx
To cleanup the state, I first stopped all the containers and cleared the downloaded images. However I still see pods running.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-3476088249-w66jr 1/1 Running 0 16m
How can I remove this?
To delete the pod:
kubectl delete pods web-3476088249-w66jr
If this pod is started via some replicaSet or deployment or anything that is creating replicas then find that and delete that first.
kubectl get all
This will list all the resources that have been created in your k8s cluster. To get information with respect to resources created in your namespace kubectl get all --namespace=<your_namespace>
To get info about the resource that is controlling this pod, you can do
kubectl describe web-3476088249-w66jr
There will be a field "Controlled By", or some owner field using which you can identify which resource created it.
When you do kubectl run ..., that's a deployment you create, not a pod directly. You can check this with kubectl get deploy. If you want to delete the pod, you need to delete the deployment with kubectl delete deploy DEPLOYMENT.
I would recommend you to create a namespace for testing when doing this kind of things. You just do kubectl create ns test, then you do all your tests in this namespace (by adding -n test). Once you have finished, you just do kubectl delete ns test, and you are done.
If you defined your object as Pod then
kubectl delete pod <--all | pod name>
will remove all of the generated Pod. But, If wrapped your Pod to Deployment object then running the command above only will trigger a re-creation of them.
In that case, you need to run
kubectl delete deployment <--all | deployment name>
That will also remove the Service object that is related to the deleted Deployment
I've been trying to shut down kubernetes cluster , but I couldn't managed to do it.
When I type
kubectl cluster-info
I can see that my cluster is still running.
I tried commands like running script
kube-down.sh
but it didn't work.
I deleted all pods. How can I shut it down ?
The tear down section of the official documentation says:
To undo what kubeadm did, you should first drain the node and make sure that the node is empty before shutting it down.
Talking to the master with the appropriate credentials, run:
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
Then, on the node being removed, reset all kubeadm installed state:
kubeadm reset
You cannot use kubectl stop command as it has been deprecated. If you have created pods using a yaml file, I suggest you use
kubectl delete -f <filename>.yml to stop any running pod.
You can also delete service associated with running pods by using the following command:
# Delete pods and services with same names "baz" and "foo"
kubectl delete pod,service baz foo
When using kube-down.sh you've to make sure that all the environment variables which were adjusted for the kube-up.sh are also used during the shut down. See also
Is there a way to get some details about Kubernetes pod that was deleted (stopped, replaced by new version).
I am investigating bug. I have logs with my pod name. That pod does not exist anymore, it was replaced by another one (with different configuration). New pod resides in same namespace, replication controller and service as old one.
Commands like
kubectl get pods
kubectl get pod <pod-name>
work only with current pods (live or stopped).
How I could get more details about old pods? I would like to see
when they were created
which environment variables they had when created
why and when they were stopped
As of today, kubectl get pods -a is deprecated, and as a result you cannot get deleted pods.
What you can do though, is to get a list of recently deleted pod names - up to 1 hour in the past unless you changed the ttl for kubernetes events - by running:
kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1
You can then investigate further issues within your logging pipeline if you have one in place.
As far as I know you cannot get the Pod details once the Pod is deleted. Can I know what is the usecase?
Example:
if a Pod is created using kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/false
you will have a Pod with status terminated:error
if a Pod is created using kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/true
you will have a Pod with status terminated:Completed
if a container in a Pod restarts: the Pod will be alive and you can get the logs of previous container (only the previous container) using
kubectl logs --container <container name> --previous=true <pod name>
if you doing an upgrade of you app and you are creating Pods using Deployments. If the update deployment "say a new image", the Pod will be terminated and new Pod will be created. You can get the Pod details from the Deployment's YAML. if you want to get details of previous Pod you have see "spec" section of previous Deployment's YAML
You can try kubectl logs --previous to list the logs of a previously stopped pod
http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/
You may also want to check out these debugging tips
http://kubernetes.io/docs/user-guide/debugging-pods-and-replication-controllers/
There is a way to find out why pods were deleted and who deleted them.
The only way to find out something is to set the ttl for k8s to be greater than the default 1h and search through the events:
kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1
If your container has previously crashed, you can access the previous container’s crash log with:
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
There is this flag:
-a, --show-all=false: When printing, show all resources (default hide terminated pods.)
But this may not help in all cases of old pods.
kubectl get pods -a
you will get the list of running pods and the terminated pods in case you are searching for this
If you want to see all the previously deleted pods and you are trying to fetch the previous pods.
Command line:
kubectl get pods
in which you will get all the pod details, because every service has one or more pods and they have unique ip address
Here you can check the lifecycle of pods and what phases of pod has.
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle
and you can see the previous pod logs by typing a command:
kubectl logs --previous