I have one master and two worker nodes (worker-1 and worker-2). All the Nodes are up and running without any issue. when i was planned to installed istio service mesh i tried to deploy sample book info deployment.
After deploying bookinfo i verified pod status running below command
root#master:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
details-v1-79c697d759-9k98l 2/2 Running 0 11h 10.200.226.104 worker-1 <none> <none>
productpage-v1-65576bb7bf-zsf6f 2/2 Running 0 11h 10.200.226.107 worker-1 <none> <none>
ratings-v1-7d99676f7f-zxrtq 2/2 Running 0 11h 10.200.226.105 worker-1 <none> <none>
reviews-v1-987d495c-hsnmc 1/2 Running 0 21m 10.200.133.194 worker-2 <none> <none>
reviews-v2-6c5bf657cf-jmbkr 1/2 Running 0 11h 10.200.133.252 worker-2 <none> <none>
reviews-v3-5f7b9f4f77-g2s6p 2/2 Running 0 11h 10.200.226.106 worker-1 <none> <none>
I have noticed that two pod are not running here status shows 1/2 (which is in worker-2 node), almost i spent two days but not able to find anything to fix the above issue. here the describe pod status
Warning Unhealthy 63s (x14 over 89s) kubelet Readiness probe failed: Get "http://10.244.133.194:15021/healthz/ready":
dial tcp 10.200.133.194:15021: connect: connection refused
Then today morning i realized something issue with worker-2 node when the pod is not running with status of 1/2, i planned cordon node like below
kubectl cordon worker-2
kubectl delete pod <worker-2 pod>
kubectl get pod -o wide
After cordon worker-2 node i could see all the pod are up with status of 2/2 in worker-1 node without any issue.
root#master:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
details-v1-79c697d759-9k98l 2/2 Running 0 11h 10.200.226.104 worker-1 <none> <none>
productpage-v1-65576bb7bf-zsf6f 2/2 Running 0 11h 10.200.226.107 worker-1 <none> <none>
ratings-v1-7d99676f7f-zxrtq 2/2 Running 0 11h 10.200.226.105 worker-1 <none> <none>
reviews-v1-987d495c-2n4d9 2/2 Running 0 17s 10.200.226.113 worker-1 <none> <none>
reviews-v2-6c5bf657cf-wzqpt 2/2 Running 0 17s 10.200.226.112 worker-1 <none> <none>
reviews-v3-5f7b9f4f77-g2s6p 2/2 Running 0 11h 10.200.226.106 worker-1 <none> <none>
could you please someone help me how to fix this issue to schedule (pending pods) pods in worker-2 node as well.
Note: when i am trying to re-deploy all the nodes (worker-1 and worker-2) again pod status going back to 1/2 status
oot#master:~/istio-1.9.1/samples# kubectl logs -f ratings-v1-b6994bb9-wfckn -c istio-proxy
ates: 0 successful, 0 rejected
2021-04-21T07:12:19.941679Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-21T07:12:21.942096Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
Related
I have followed the instructions on this blog to create a simple container image and deploy it in a k8s cluster.
However, in my case the pods do not run:
student#master:~$ k get pod -o wide -l app=hello-python --field-selector spec.nodeName=master
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python-58547cf485-7l8dg 0/1 ErrImageNeverPull 0 2m26s 192.168.219.126 master <none> <none>
hello-python-598c594dc5-4c9zd 0/1 ErrImageNeverPull 0 2m26s 192.168.219.67 master <none> <none>
student#master:~$ sudo podman images hello-python
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/hello-python latest 11cf1e5a86b1 50 minutes ago 941 MB
student#master:~$ hostname
master
student#master:~$
I understand why it may not work on the worker node, but why it does not work on the same node where the image is cached - the master node?
student#master:~$ k describe pod hello-python-58547cf485-7l8dg | grep -A 10 'Events:'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned default/hello-python-58547cf485-7l8dg to master
Warning Failed 8m7s (x12 over 10m) kubelet Error: ErrImageNeverPull
Warning ErrImageNeverPull 4m59s (x27 over 10m) kubelet Container image "localhost/hello-python:latest" is not present with pull policy of Never
student#master:~$
My question is: how to make the pod run on the master node with the imagePullPolicy = never given that the image in question is available on the master node as the podman images attests?
EDIT 1
I am using a k8s cluster running on two VMs deployed in GCE. It was setup with a script provided in the context of the Linux Foundation Kubernetes Developer course LFD0259.
EDIT 2
The master node is allowed to run workloads - this is how the LFD259 course sets it up. For example:
student#master:~$ k create deployment xyz --image=httpd
deployment.apps/xyz created
student#master:~$ k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
xyz-6c6bd4cd89-qn4zr 1/1 Running 0 5m37s 192.168.171.66 worker <none> <none>
student#master:~$
student#master:~$ k scale deployment xyz --replicas=10
deployment.apps/xyz scaled
student#master:~$ k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
xyz-6c6bd4cd89-c2xv4 1/1 Running 0 73s 192.168.219.71 master <none> <none>
xyz-6c6bd4cd89-g89k2 0/1 ContainerCreating 0 73s <none> master <none> <none>
xyz-6c6bd4cd89-jfftl 0/1 ContainerCreating 0 73s <none> worker <none> <none>
xyz-6c6bd4cd89-kbdnq 1/1 Running 0 73s 192.168.219.106 master <none> <none>
xyz-6c6bd4cd89-nm6rt 0/1 ContainerCreating 0 73s <none> worker <none> <none>
xyz-6c6bd4cd89-qn4zr 1/1 Running 0 7m22s 192.168.171.66 worker <none> <none>
xyz-6c6bd4cd89-vts6x 1/1 Running 0 73s 192.168.171.84 worker <none> <none>
xyz-6c6bd4cd89-wd2ls 1/1 Running 0 73s 192.168.171.127 worker <none> <none>
xyz-6c6bd4cd89-wv4jn 0/1 ContainerCreating 0 73s <none> worker <none> <none>
xyz-6c6bd4cd89-xvtlm 0/1 ContainerCreating 0 73s <none> master <none> <none>
student#master:~$
It depends how you've set up your Kubernetes Cluster. I assume you've installed it with kubeadm. However, by default the Master is not scheduleable for workloads. And by my understanding the image you're talking about only exists on the master node right? If that's the case you can't start a pod with that Image as it only exists on the master node, which doesn't allow workloads by default.
If you were to copy the Image to the worker node, your given command should work.
However if you want to make your Master-Node scheduleable just taint it with (maybe you need to amend the last bit if it differs from yours):
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
I have a daemonset configuration that runs on all nodes.
every pod listens on port 34567. I want from other pod on different node to communicate with this pod. how can I achieve that?
Find the target Pod's IP address as shown below
controlplane $ k get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-42pq8 1/1 Running 1 5m43s 10.88.0.4 node01 <none> <none>
coredns-fb8b8dccf-f9n5x 1/1 Running 1 5m43s 10.88.0.3 node01 <none> <none>
etcd-controlplane 1/1 Running 0 4m38s 172.17.0.23 controlplane <none> <none>
katacoda-cloud-provider-74dc75cf99-2jrpt 1/1 Running 3 5m42s 10.88.0.2 node01 <none> <none>
kube-apiserver-controlplane 1/1 Running 0 4m33s 172.17.0.23 controlplane <none> <none>
kube-controller-manager-controlplane 1/1 Running 0 4m45s 172.17.0.23 controlplane <none> <none>
kube-keepalived-vip-smkdc 1/1 Running 0 5m27s 172.17.0.26 node01 <none> <none>
kube-proxy-8sxkt 1/1 Running 0 5m27s 172.17.0.26 node01 <none> <none>
kube-proxy-jdcqc 1/1 Running 0 5m43s 172.17.0.23 controlplane <none> <none>
kube-scheduler-controlplane 1/1 Running 0 4m47s 172.17.0.23 controlplane <none> <none>
weave-net-8cxqg 2/2 Running 1 5m27s 172.17.0.26 node01 <none> <none>
weave-net-s4tcj 2/2 Running 1 5m43s 172.17.0.23 controlplane <none> <none>
Next "exec" into the originating pod - kube-proxy-8sxkt in my example
kubectl -n kube-system exec -it kube-proxy-8sxkt sh
Next, you will use the destination pod's IP and port (10256 - my example) number to connect. Please note that you may have to install curl/telnet if your originating container's image does not include the application
# curl telnet://172.17.0.23:10256
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close
You can call via pod's IP.
Note: This IP can only be used in the k8s cluster.
POD address (IP) is a good option you can use it, unless you know the POD IP which might get changed from time to time due to deployment and scaling changes.
i would suggest trying out the Daemon set by exposing it using the service type Node port if you have a fix amount of Node and not much autoscaling there.
If you want to connect your POD with a specific POD you can use the Node IP on which POD is scheduled and use the Node port service.
Node IP:Node port
Read more at : https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
If you don't want to connect to a specific POD and just any of the Daemon sets replica will work to connect with you can use the service name to connect PODs with each other.
my-svc.my-namespace.svc.cluster-domain.example
Read more about the service and POD DNS
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
I deployed nginx ingress by kubespray. I have 3 masters and 2 workers and 5 ingress-nginx-controller. I tried to shutdown one worker and now I see still 5 nginx ingress on all hosts.
[root#node1 ~]# kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-controller-5828c 1/1 Running 0 7m4s 10.233.96.9 node2 <none> <none>
ingress-nginx-controller-h5zzl 1/1 Running 0 7m42s 10.233.92.7 node3 <none> <none>
ingress-nginx-controller-wrvv6 1/1 Running 0 6m11s 10.233.90.17 node1 <none> <none>
ingress-nginx-controller-xdkrx 1/1 Running 0 5m44s 10.233.105.25 node4 <none> <none>
ingress-nginx-controller-xgpn2 1/1 Running 0 6m38s 10.233.70.32 node5 <none> <none>
The problem is I am getting 503 error with app after one node was power off. Is some option disconnect not working ingress-nginx-controller or possibility to use round robin, please? Or could I catch non working ingress-nginx-controller and redirect traffic to correct one, please?
I shutdown the node where the app was running. Now is everything working.
I am trying to install Kubernetes 1.14.3 on IPV6 environment.
I don't have any IPV4 interface on this environment , only IPV6.
I tried with p-lain kubeadm config file and it seems to work but when I try to apply the calico cni the calico-node keeps failing.
2019-07-28 07:15:26.714 [INFO][9] startup.go 331: Hit error connecting to datastore - retry error=Get https://[fd20::4001]:443/api/v1/nodes/foo: dial tcp [fd20::4001]:443: connect: network is unreachable
this is the status og the pods at the moment:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-6894d6f4f4-hwsmc 0/1 ContainerCreating 0 79s <none> master-eran <none> <none>
kube-system calico-node-fj8q7 0/1 Running 1 79s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system coredns-fb8b8dccf-8b995 0/1 ContainerCreating 0 5m53s <none> master-eran <none> <none>
kube-system coredns-fb8b8dccf-fbpwq 0/1 ContainerCreating 0 5m53s <none> master-eran <none> <none>
kube-system etcd-master-eran 1/1 Running 0 4m56s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-apiserver-master-eran 1/1 Running 0 4m53s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-controller-manager-master-eran 1/1 Running 0 5m7s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-proxy-4qzb8 1/1 Running 0 5m53s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-scheduler-master-eran 1/1 Running 0 4m50s 2001:df0:8800:4::7 master-eran <none> <none>
I guess that the codedns and controller will start only after the calico-node will run but it keeps failing on the error I pasted earlier.
in the kubeadm config file I chose ipvs in proxy configurations.
does any one have any idea on how to solve this?
thanks
NEW STATUS:
I was able to resOlve the calico-node issue but now I am failing on calico-controller :
7-30 07:58:22.979 [ERROR][1] client.go 255: Error getting cluster information config ClusterInformation="default" error=Get https://[fd20::4001]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp [fd20::4001]:443: connect: permission denied
2019-07-30 07:58:22.979 [FATAL][1] main.go 118: Failed to initialize Calico datastore error=Get https://[fd20::4001]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp [fd20::4001]:443: connect: permission denied
According to the Calico documentation you may need to perform a few additional steps before you can start using it with ipv6 only. About enabling IPv6 with Kubernetes you can read here.
I'm trying to setup istio on Google container engine, istio has been installed successfully but booking sample has been failed to load.
Is there something I have configured in wrong way?
Help me, please!
Thanks in advance!
Here's what's I have tried:
kubectl get pods
details-v1-3121678156-3h2wx 2/2 Running 0 58m
grafana-1395297218-h0tjv 1/1 Running 0 5h
istio-ca-4001657623-n00zx 1/1 Running 0 5h
istio-egress-2038322175-0jtf5 1/1 Running 0 5h
istio-ingress-2247081378-fvr33 1/1 Running 0 5h
istio-mixer-2450814972-jrrm4 1/1 Running 0 5h
istio-pilot-1836659236-kw7cr 2/2 Running 0 5h
productpage-v1-1440812148-gqrgl 0/2 Pending 0 57m
prometheus-3067433533-fqcfw 1/1 Running 0 5h
ratings-v1-3755476866-jbh80 2/2 Running 0 58m
reviews-v1-3728017321-0m7mk 0/2 Pending 0 58m
reviews-v2-196544427-6ftf5 0/2 Pending 0 58m
reviews-v3-959055789-079xz 0/2 Pending 0 57m
servicegraph-3127588006-03b93 1/1 Running 0 5h
zipkin-4057566570-0cb86 1/1 Running 0 5h
kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
details 10.11.249.214 <none> 9080/TCP
grafana 10.11.247.226 104.199.211.175 3000:31036/TCP
istio-egress 10.11.246.60 <none> 80/TCP
istio-ingress 10.11.242.178 35.189.165.119 80:31622/TCP,443:31241/TCP
istio-mixer 10.11.242.104 <none> 9091/TCP,9094/TCP,42422/TCP
istio-pilot 10.11.251.240 <none> 8080/TCP,8081/TCP
kubernetes 10.11.240.1 <none> 443/TCP
productpage 10.11.255.53 <none> 9080/TCP
prometheus 10.11.248.237 130.211.249.66 9090:32056/TCP
ratings 10.11.252.40 <none> 9080/TCP
reviews 10.11.242.168 <none> 9080/TCP
servicegraph 10.11.252.60 35.185.161.219 8088:32709/TCP
zipkin 10.11.245.4 35.185.144.62 9411:31677/TCP
get ingress IP and export env variable then curl
NAME HOSTS ADDRESS PORTS AGE
gateway * 35.189.165.119 80 1h
Abduls-MacBook-Pro:~ abdul$ export GATEWAY_URL=35.189.165.119:80
Abduls-MacBook-Pro:~ abdul$ curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage
000
I ran into a similar issue ("upstream connect error or disconnect/reset before headers") when i deployed the istio sample app on GKE. Try to delete all pods (and wait for all of to come up again). Then restart your proxy...
kubectl delete pods --all