I am facing same issue in my lab setup on my laptop.
Environment:
Istio version installed 1.7. Pods are up and working
vagrant#master-1:~$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-75b5cddb4d-5t5lq 1/1 Running 1 16h
istio-egressgateway-695f5944d8-s7mbg 1/1 Running 1 16h
istio-ingressgateway-5c697d4cd7-vpd68 1/1 Running 1 16h
istiod-76fdcdd945-tscgc 1/1 Running 0 17m
kiali-6c49c7d566-8wbnw 1/1 Running 1 16h
prometheus-9d5676d95-zxbnk 2/2 Running 2 14h
Kubernetes Cluster information:-
Cluster is deployed by hard way
1 LB in front of master with IP 192.168.5.30 and HA proxy running , 1 Master node 192.168.5.11, 2 worker nodes in setup deployed on VMbox Ubuntu VMs . I am using weavenet as the CNI for my cluster.
Worker Node in cluster:-
vagrant#loadbalancer:~$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
worker-3 Ready <none> 62d v1.18.0 192.168.5.24 <none> Ubuntu 18.04.4 LTS 4.15.0-112-generic docker://19.3.12
worker-4 Ready <none> 61d v1.18.0 192.168.5.25 <none> Ubuntu 18.04.4 LTS 4.15.0-112-generic docker://19.3.12
vagrant#loadbalancer:~$
Kube-Apisever config
--ExecStart=/usr/local/bin/kube-apiserver \\
--advertise-address=192.168.5.11 \\
--allow-privileged=true \\
--apiserver-count=3 \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/log/audit.log \\
--authorization-mode=Node,RBAC \\
--bind-address=0.0.0.0 \\
--client-ca-file=/var/lib/kubernetes/ca.crt \\
--enable-admission-plugins=NodeRestriction,ServiceAccount \\
--enable-swagger-ui=true \\
--enable-bootstrap-token-auth=true \\
--etcd-cafile=/var/lib/kubernetes/ca.crt \\
--etcd-certfile=/var/lib/kubernetes/etcd-server.crt \\
--etcd-keyfile=/var/lib/kubernetes/etcd-server.key \\
--etcd-servers=https://192.168.5.11:2379 \\
--event-ttl=1h \\
--encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
--kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \\
--kubelet-client-certificate=/var/lib/kubernetes/kube-apiserver.crt \\
--kubelet-client-key=/var/lib/kubernetes/kube-apiserver.key \\
--kubelet-https=true \\
--service-account-key-file=/var/lib/kubernetes/service-account.crt \\
--service-cluster-ip-range=10.96.0.0/24 \\
--service-node-port-range=30000-32767 \\
--tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \\
--tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \\
--v=2
istio pod svc
vagrant#master-1:~$ kubectl describe svc istiod -n istio-system
Name: istiod
Namespace: istio-system
Labels: app=istiod
install.operator.istio.io/owning-resource=installed-state
install.operator.istio.io/owning-resource-namespace=istio-system
istio=pilot
istio.io/rev=default
operator.istio.io/component=Pilot
operator.istio.io/managed=Reconcile
operator.istio.io/version=1.7.0
release=istio
Annotations: Selector: app=istiod,istio=pilot
Type: ClusterIP
IP: 10.96.0.197
Port: grpc-xds 15010/TCP
TargetPort: 15010/TCP
Endpoints: 10.44.0.7:15010
Port: https-dns 15012/TCP
TargetPort: 15012/TCP
Endpoints: 10.44.0.7:15012
Port: https-webhook 443/TCP
TargetPort: 15017/TCP
Endpoints: 10.44.0.7:15017
Port: http-monitoring 15014/TCP
TargetPort: 15014/TCP
Endpoints: 10.44.0.7:15014
Port: dns-tls 853/TCP
TargetPort: 15053/TCP
Endpoints: 10.44.0.7:15053
Session Affinity: None
Events: <none>
basic troubleshooting
vagrant#loadbalancer:~$ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | grep policy:
policy: enabled
vagrant#loadbalancer:~$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep
> istio-injection: enabled
bjectSelector: {}
reinvocationPolicy:
> Never
Error from Kube API server
Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521377 1750 trace.go:116] Trace[51800791]: “Call mutating webhook” configuration:istio-sidecar-injector,webhook:sidecar-injector.istio.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:9b96e1b2-3bbe-41d6-a727-0e19cdd9fbd1 (started: 2020-08-31 02:47:52.521061627 +0000 UTC m=+1080.518695497) (total time:30.000277923s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[51800791]: [30.000277923s] [30.000277923s] END
Aug 31 02:48:22 master-1 kube-apiserver[1750]: W0831 02:48:22.521529 1750 dispatcher.go:181] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook “sidecar-injector.istio.io”: Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521814 1750 trace.go:116] Trace[491776795]: “Create” url:/api/v1/namespaces/dev/pods,user-agent## ##:kubectl/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.5.30 (started: 2020-08-31 02:47:52.510910326 +0000 UTC m=+1080.508544152) (total time: 30.010883231s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[491776795]: [30.010883231s] [30.003030474s] END
As I already mentioned in comments if you're using VM you should follow this guide to deploy Istio and connect a virtual machine to it.
Just note that VM support is still an alpha feature.
Quoted from 1.6 upgrade notes
Better Virtual Machine support
Expanding our support for workloads not running in Kubernetes was one of the our major areas of investment for 2020, and we’re excited to announce some great progress here.
For those of you who are adding non-Kubernetes workloads to meshes (for example, workloads deployed on VMs), the new WorkloadEntry resource makes that easier than ever. We created this API to give non-Kubernetes workloads first-class representation in Istio. It elevates a VM or bare metal workload to the same level as a Kubernetes Pod, instead of just an endpoint with an IP address. You now even have the ability to define a Service that is backed by both Pods and VMs. Why is that useful? Well, you now have the ability to have a heterogeneous mix of deployments (VMs and Pods) for the same service, providing a great way to migrate VM workloads to a Kubernetes cluster without disrupting traffic to and from it.
VM-based workloads remain a high priority for us, and you can expect to see more in this area over the coming releases.
There are the steps you should follow to install Istio and connect a virtual machine to it.
Prerequisites
Prepare the guide environment
Install the Istio control plane
Configure the VM namespace
Create files to transfer to the virtual machine
Configure the virtual machine
Start Istio within the virtual machine
Verify Istio Works Successfully
There are videos on youtube about that.
Istio 1.7 improved VM support part 1
Istio 1.7 improved VM support part 2
There are examples on istio documentation.
https://istio.io/latest/docs/examples/virtual-machines/single-network/
https://istio.io/latest/docs/examples/virtual-machines/multi-network/
https://istio.io/latest/docs/examples/virtual-machines/bookinfo/
Related
I am trying to have 1 redis master with 2 redis replicas tied to a 3 Quorum Sentinel on Kubernetes. I am very new to Kubernetes.
My initial plan was to have the master running on a pod tied to 1 Kubernetes SVC and the 2 replicas running on their own pods tied to another Kubernetes SVC. Finally, the 3 Sentinel pods will be tied to their own SVC. The replicas will be tied to the master SVC (because without svc, ip will change). The sentinel will also be configured and tied to master and replica SVCs. But I'm not sure if this is feasible because when master pod crashes, how will one of the replica pods move to the master SVC and become the master? Is that possible?
The second approach I had was to wrap redis pods in a replication controller and the same for sentinel as well. However, I'm not sure how to make one of the pods master and the others replicas with a replication controller.
Would any of the two approaches work? If not, is there a better design that I can adopt? Any leads would be appreciated.
You can deploy Redis Sentinel using the Helm package manager and the Redis Helm Chart.
If you don't have Helm3 installed yet, you can use this documentation to install it.
I will provide a few explanations to illustrate how it works.
First we need to get the values.yaml file from the Redis Helm Chart to customize our installation:
$ wget https://raw.githubusercontent.com/bitnami/charts/master/bitnami/redis/values.yaml
We can configure a lot of parameters in the values.yaml file , but for demonstration purposes I only enabled Sentinel and set the redis password:
NOTE: For a list of parameters that can be configured during installation, see the Redis Helm Chart Parameters documentation.
# values.yaml
global:
redis:
password: redispassword
...
replica:
replicaCount: 3
...
sentinel:
enabled: true
...
Then we can deploy Redis using the configuration from the values.yaml file:
NOTE: It will deploy a three Pod cluster (one master and two slaves) managed by the StatefulSets with a sentinel container running inside each Pod.
$ helm install redis-sentinel bitnami/redis --values values.yaml
Be sure to carefully read the NOTES section of the chart installation output. It contains many useful information (e.g. how to connect to your database from outside the cluster)
After installation, check redis StatefulSet, Pods and Services (headless service can be used for internal access):
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP
redis-sentinel-node-0 2/2 Running 0 2m13s 10.4.2.21
redis-sentinel-node-1 2/2 Running 0 86s 10.4.0.10
redis-sentinel-node-2 2/2 Running 0 47s 10.4.1.10
$ kubectl get sts
NAME READY AGE
redis-sentinel-node 3/3 2m41s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-sentinel ClusterIP 10.8.15.252 <none> 6379/TCP,26379/TCP 2m
redis-sentinel-headless ClusterIP None <none> 6379/TCP,26379/TCP 2m
As you can see, each redis-sentinel-node Pod contains the redis and sentinel containers:
$ kubectl get pods redis-sentinel-node-0 -o jsonpath={.spec.containers[*].name}
redis sentinel
We can check the sentinel container logs to find out which redis-sentinel-node is the master:
$ kubectl logs -f redis-sentinel-node-0 sentinel
...
1:X 09 Jun 2021 09:52:01.017 # Configuration loaded
1:X 09 Jun 2021 09:52:01.019 * monotonic clock: POSIX clock_gettime
1:X 09 Jun 2021 09:52:01.019 * Running mode=sentinel, port=26379.
1:X 09 Jun 2021 09:52:01.026 # Sentinel ID is 1bad9439401e44e749e2bf5868ad9ec7787e914e
1:X 09 Jun 2021 09:52:01.026 # +monitor master mymaster 10.4.2.21 6379 quorum 2
...
1:X 09 Jun 2021 09:53:21.429 * +slave slave 10.4.0.10:6379 10.4.0.10 6379 # mymaster 10.4.2.21 6379
1:X 09 Jun 2021 09:53:21.435 * +slave slave 10.4.1.10:6379 10.4.1.10 6379 # mymaster 10.4.2.21 6379
...
As you can see from the logs above, the redis-sentinel-node-0 Pod is the master and the redis-sentinel-node-1 & redis-sentinel-node-2 Pods are slaves.
For testing, let's delete the master and check if sentinel will switch the master role to one of the slaves:
$ kubectl delete pod redis-sentinel-node-0
pod "redis-sentinel-node-0" deleted
$ kubectl logs -f redis-sentinel-node-1 sentinel
...
1:X 09 Jun 2021 09:55:20.902 # Executing user requested FAILOVER of 'mymaster'
...
1:X 09 Jun 2021 09:55:22.666 # +switch-master mymaster 10.4.2.21 6379 10.4.1.10 6379
...
1:X 09 Jun 2021 09:55:50.626 * +slave slave 10.4.0.10:6379 10.4.0.10 6379 # mymaster 10.4.1.10 6379
1:X 09 Jun 2021 09:55:50.632 * +slave slave 10.4.2.22:6379 10.4.2.22 6379 # mymaster 10.4.1.10 6379
A new master (redis-sentinel-node-2 10.4.1.10) has been selected, so everything works as expected.
Additionally, we can display more information by connecting to one of the Redis nodes:
$ kubectl run --namespace default redis-client --restart='Never' --env REDIS_PASSWORD=redispassword --image docker.io/bitnami/redis:6.2.1-debian-10-r47 --command -- sleep infinity
pod/redis-client created
$ kubectl exec --tty -i redis-client --namespace default -- bash
I have no name!#redis-client:/$ redis-cli -h redis-sentinel-node-1.redis-sentinel-headless -p 6379 -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-sentinel-node-1.redis-sentinel-headless:6379> info replication
# Replication
role:slave
master_host:10.4.1.10
master_port:6379
master_link_status:up
...
How can one inquire the Kubernetes pod and service subnets in use (e.g. 10.244.0.0/16 and 10.96.0.0/12 respectively) from inside a Kubernetes cluster in a portable and simple way?
For instance, kubectl get cm -n kube-system kubeadm-config -o yaml reports podSubnet and serviceSubnet. But this is not fully portable because a cluster may have been set up by another means than kubeadm.
kubectl get cm -n kube-system kube-proxy -o yaml reports clusterCIDR (i.e. pod subnet) and kubectl get pod -n kube-system kube-apiserver-master1 -o yaml reports the value
passed as command-line option --service-cluster-ip-range to kube-apiserver (i.e. service subnet). master1 stands for the name of any control plane node. But this seems a bit complex.
Is there a better way available e.g. with the Kubernetes 1.17 API?
I don't think it would be possible to obtain what you want in a portable and simple way.
If you don't specify Cidr's parameters it will assign default one.
As you have many ways to run kubernetes as unmanaged clusters like kubeadm, minikbue, k3s, micork8s or managed like Cloud providers (GKE, Azure, AWS) it's hard to find one way to list all cidrs in all environments. Another obstacle can be versions of Kubernetes or CNI.
In Kubernetes 1.17 Release notes you can find information that
Deprecate the default service IP CIDR. The previous default was 10.0.0.0/24 which will be removed in 6 months/2 releases. Cluster admins must specify their own desired value, by using --service-cluster-ip-range on kube-apiserver.
As example of Kubeadm: $ kubeadm init --pod-network-cidr 10.100.0.0/12 --service-cidr 10.99.0.0/12
There are a few ways to get this pod and service-cidr:
$ kubectl cluster-info dump | grep -E '(service-cluster-ip-range|cluster-cidr)'
"--service-cluster-ip-range=10.99.0.0/12",
"--cluster-cidr=10.100.0.0/12",
$ kubeadm config view | grep Subnet
podSubnet: 10.100.0.0/12
serviceSubnet: 10.99.0.0/12
But if you will check all pods in this cluster, some pods are starting with 192.168.190.X or 192.168.137.X
$ kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx 1/1 Running 0 62m 192.168.190.129 kubeadm-worker <none> <none>
kube-system calico-kube-controllers-77c5fc8d7f-9n6m5 1/1 Running 0 118m 192.168.137.66 kubeadm-master <none> <none>
kube-system calico-node-2kx2v 1/1 Running 0 117m 10.128.0.4 kubeadm-worker <none> <none>
kube-system calico-node-8xqd9 1/1 Running 0 118m 10.128.0.3 kubeadm-master <none> <none>
kube-system coredns-66bff467f8-sgmkw 1/1 Running 0 120m 192.168.137.65 kubeadm-master <none> <none>
kube-system coredns-66bff467f8-t84ht 1/1 Running 0 120m 192.168.137.67 kubeadm-master <none> <none>
If you will describe any CNI pods you can find another CIDRs:
CALICO_IPV4POOL_CIDR: 192.168.0.0/16
For GKE example you will have:
node CIDRs
$ kubectl describe node | grep CIDRs
PodCIDRs: 10.52.1.0/24
PodCIDRs: 10.52.0.0/24
PodCIDRs: 10.52.2.0/24
$ gcloud container clusters describe cluster-2 --zone=europe-west2-b | grep Cidr
clusterIpv4Cidr: 10.52.0.0/14
clusterIpv4Cidr: 10.52.0.0/14
clusterIpv4CidrBlock: 10.52.0.0/14
servicesIpv4Cidr: 10.116.0.0/20
servicesIpv4CidrBlock: 10.116.0.0/20
podIpv4CidrSize: 24
servicesIpv4Cidr: 10.116.0.0/20
Honestly I don't think there is an easy and portable way to list all podCidrs and serviceCidrs in one simple command.
I'm running Kubernetes in virtual machines and going through the basic tutorials, currently Add logging and metrics to the PHP / Redis Guestbook example. I'm trying to install kube-state-metrics:
git clone https://github.com/kubernetes/kube-state-metrics.git kube-state-metrics
kubectl create -f kube-state-metrics/kubernetes
but it fails.
kubectl describe pod --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7
...
Warning Unhealthy 28m (x8 over 30m) kubelet, kubernetes-node1 Readiness probe failed: Get http://192.168.129.102:8080/healthz: dial tcp 192.168.129.102:8080: connect: connection refused
kubectl logs --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7 -c kube-state-metrics
I0514 17:29:26.980707 1 main.go:85] Using default collectors
I0514 17:29:26.980774 1 main.go:93] Using all namespace
I0514 17:29:26.980780 1 main.go:129] metric white-blacklisting: blacklisting the following items:
W0514 17:29:26.980800 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0514 17:29:26.983504 1 main.go:169] Testing communication with server
F0514 17:29:56.984025 1 main.go:137] Failed to create client: ERROR communicating with apiserver: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout
I'm unsure if this 10.96.0.1 IP is correct. My virtual machines are in a bridged network 10.10.10.0/24 and a host-only network 192.168.59.0/24. When initializing Kubernetes I used the argument --pod-network-cidr=192.168.0.0/16 so that's one more IP range that I'd expect. But 10.96.0.1 looks unfamiliar.
I'm new to Kubernetes, just doing the basic tutorials, so I don't know what to do now. How to fix it or investigate further?
EDIT - additonal info:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubernetes-master Ready master 15d v1.14.1 10.10.10.11 <none> Ubuntu 18.04.2 LTS 4.15.0-48-generic docker://18.9.2
kubernetes-node1 Ready <none> 15d v1.14.1 10.10.10.5 <none> Ubuntu 18.04.2 LTS 4.15.0-48-generic docker://18.9.2
kubernetes-node2 Ready <none> 15d v1.14.1 10.10.10.98 <none> Ubuntu 18.04.2 LTS 4.15.0-48-generic docker://18.9.2
The command I used to initialize the cluster:
sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=192.168.0.0/16
The reason for this is probably overlapping of Pod network with Node network - you set Pod network CIDR to 192.168.0.0/16 which your host-only network will be included into as its address is 192.168.59.0/24.
To solve this you can either change the pod network CIDR to 192.168.0.0/24 (it is not recommended as this will give you only 255 addresses for your pod networking)
You can also use different range for your Calico. If you want to do it on a running cluster here is an instruction.
Also other way I tried:
edit Calico manifest to different range (for example 10.0.0.0/8) - sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=10.0.0.0/8) and apply it after the init.
Another way would be using different CNI like Flannel (which uses 10.244.0.0/16).
You can find more information about ranges of CNI plugins here.
I've deployed several different services and always get the same error.
The service is reachable on the node port from the machine where the pod is running. On the two other nodes I get timeouts.
The kube-proxy is running on all worker nodes and I can see in the logfiles from kube-proxy that the service port was added and the node port was opened.
In this case I've deployed the stars demo from calico
Kube-proxy log output:
Mar 11 10:25:10 kuben1 kube-proxy[659]: I0311 10:25:10.229458 659 service.go:309] Adding new service port "management-ui/management-ui:" at 10.32.0.133:9001/TCP
Mar 11 10:25:10 kuben1 kube-proxy[659]: I0311 10:25:10.257483 659 proxier.go:1427] Opened local port "nodePort for management-ui/management-ui:" (:30002/tcp)
The kube-proxy is listening on the port 30002
root#kuben1:/tmp# netstat -lanp | grep 30002
tcp6 0 0 :::30002 :::* LISTEN 659/kube-proxy
There are also some iptable rules defined:
root#kuben1:/tmp# iptables -L -t nat | grep management-ui
KUBE-MARK-MASQ tcp -- anywhere anywhere /* management-ui/management-ui: */ tcp dpt:30002
KUBE-SVC-MIYW5L3VT4JVLCIZ tcp -- anywhere anywhere /* management-ui/management-ui: */ tcp dpt:30002
KUBE-MARK-MASQ tcp -- !10.200.0.0/16 10.32.0.133 /* management-ui/management-ui: cluster IP */ tcp dpt:9001
KUBE-SVC-MIYW5L3VT4JVLCIZ tcp -- anywhere 10.32.0.133 /* management-ui/management-ui: cluster IP */ tcp dpt:9001
The interesting part is that I can reach the service IP from any worker node
root#kubem1:/tmp# kubectl get svc -n management-ui
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
management-ui NodePort 10.32.0.133 <none> 9001:30002/TCP 52m
The service IP/port can be accessed from any worker node if I do a "curl http://10.32.0.133:9001"
I don't understand why kube-proxy does not "route" this properly...
Has anyone a hint where I can find the error?
Here some cluster specs:
This is a hand build cluster inspired by Kelsey Hightower's "kubernetes the hard way" guide.
6 Nodes (3 master: 3 worker) local vms
OS: Ubuntu 18.04
K8s: v1.13.0
Docker: 18.9.3
Cni: calico
Component status on the master nodes looks okay
root#kubem1:/tmp# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
The worker nodes are looking okay if I trust kubectl
root#kubem1:/tmp# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kuben1 Ready <none> 39d v1.13.0 192.168.178.77 <none> Ubuntu 18.04.2 LTS 4.15.0-46-generic docker://18.9.3
kuben2 Ready <none> 39d v1.13.0 192.168.178.78 <none> Ubuntu 18.04.2 LTS 4.15.0-46-generic docker://18.9.3
kuben3 Ready <none> 39d v1.13.0 192.168.178.79 <none> Ubuntu 18.04.2 LTS 4.15.0-46-generic docker://18.9.3
As asked by P Ekambaram:
root#kubem1:/tmp# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-node-bgjdg 1/1 Running 5 40d
calico-node-nwkqw 1/1 Running 5 40d
calico-node-vrwn4 1/1 Running 5 40d
coredns-69cbb76ff8-fpssw 1/1 Running 5 40d
coredns-69cbb76ff8-tm6r8 1/1 Running 5 40d
kubernetes-dashboard-57df4db6b-2xrmb 1/1 Running 5 40d
I've found a solution for my "Problem".
This behavior was caused by a change in Docker v1.13.x and the issue was fixed in kubernetes with version 1.8.
The easy solution was to change the forward rules via iptables.
Run the following cmd on all worker nodes: "iptables -A FORWARD -j ACCEPT"
To fix it the right way i had to tell the kube-proxy the cidr for the pods.
Theoretical that could be solved in two ways:
Add "--cluster-cidr=10.0.0.0/16" as argument to the kube-proxy command line(in my case in the systemd service file)
Add 'clusterCIDR: "10.0.0.0/16"' to the kubeconfig file for kube-proxy
In my case the cmd line argument doesn't had any effect.
As i've added the line to my kubeconfig file and restarted the kube-proxy on all worker nodes everything works well.
Here is the github merge request for this "FORWARD" issue: link
I am running Kubernetes cluster which is configured with a master and 3 nodes.
#kubectl get nodes
NAME STATUS AGE
minion-1 Ready 46d
minion-2 Ready 46d
minion-3 Ready 46d
I have launched couple of pods in the cluster and found that the pods are in pending state.
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
httpd 0/1 Pending 0 10m <none>
nginx 0/1 Pending 0 11m <none>
One of the pod "httpd" yaml file:
# cat http.yaml
apiVersion: v1
kind: Pod
metadata:
name: httpd
labels:
env: test
spec:
containers:
- name: httpd
image: httpd
While debugging the reason for failure found that the couple of nodes configured are not ready. Only one node is reachable from master.
# ping minion-1
PING minion-1 (172.31.24.204) 56(84) bytes of data.
64 bytes from minion-1 (172.31.24.204): icmp_seq=1 ttl=64 time=0.575 ms
Whereas other nodes are not reachable:
# ping minion-2
PING minion-2 (172.31.29.95) 56(84) bytes of data.
From master (172.31.16.204) icmp_seq=1 Destination Host Unreachable
# ping minion-3
PING minion-3 (172.31.17.252) 56(84) bytes of data.
From master (172.31.16.204) icmp_seq=1 Destination Host Unreachable
The queries that I have here is
1) Why does Kubernetes showing the nodes as ready even if they are not
reachable from master?
2) Why are the pods creation failing?
Is it because of unavailability of nodes or any configuration issue in yaml file?
# kubectl describe pod httpd
Name: httpd
Namespace: default
Node: /
Labels: env=test
Status: Pending
IP:
Controllers: <none>
Containers:
httpd:
Image: httpd
Port:
Volume Mounts: <none>
Environment Variables: <none>
No volumes.
QoS Class: BestEffort
Tolerations: <none>
No events.
Following are the Kubernetes and etcd versions.
]# kubectl --version
Kubernetes v1.5.2
[root#raghavendar1 ~]# et
etcd etcdctl ether-wake ethtool
[root#raghavendar1 ~]# etcd --version
etcd Version: 3.2.5
Git SHA: d0d1a87
Go Version: go1.8.3
Go OS/Arch: linux/amd64
Kubernetes do not use ICMP protocol to check nodes master node connectivity.
Nodes become Ready when the communication node -> api-server works and this is done via https protocol.
You can read more about about node - master connectivity in kubernetes documentation https://kubernetes.io/docs/concepts/architecture/master-node-communication/
Why pod isn't scheduled?
The answer to this question is in the master logs probably, check kube-apiserver.log, kube-scheduler.log. The reason is cluster misconfiguration.
For start run it in a single network to get a grip of things and double check routing.