I am trying to install Kubernetes 1.14.3 on IPV6 environment.
I don't have any IPV4 interface on this environment , only IPV6.
I tried with p-lain kubeadm config file and it seems to work but when I try to apply the calico cni the calico-node keeps failing.
2019-07-28 07:15:26.714 [INFO][9] startup.go 331: Hit error connecting to datastore - retry error=Get https://[fd20::4001]:443/api/v1/nodes/foo: dial tcp [fd20::4001]:443: connect: network is unreachable
this is the status og the pods at the moment:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-6894d6f4f4-hwsmc 0/1 ContainerCreating 0 79s <none> master-eran <none> <none>
kube-system calico-node-fj8q7 0/1 Running 1 79s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system coredns-fb8b8dccf-8b995 0/1 ContainerCreating 0 5m53s <none> master-eran <none> <none>
kube-system coredns-fb8b8dccf-fbpwq 0/1 ContainerCreating 0 5m53s <none> master-eran <none> <none>
kube-system etcd-master-eran 1/1 Running 0 4m56s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-apiserver-master-eran 1/1 Running 0 4m53s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-controller-manager-master-eran 1/1 Running 0 5m7s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-proxy-4qzb8 1/1 Running 0 5m53s 2001:df0:8800:4::7 master-eran <none> <none>
kube-system kube-scheduler-master-eran 1/1 Running 0 4m50s 2001:df0:8800:4::7 master-eran <none> <none>
I guess that the codedns and controller will start only after the calico-node will run but it keeps failing on the error I pasted earlier.
in the kubeadm config file I chose ipvs in proxy configurations.
does any one have any idea on how to solve this?
thanks
NEW STATUS:
I was able to resOlve the calico-node issue but now I am failing on calico-controller :
7-30 07:58:22.979 [ERROR][1] client.go 255: Error getting cluster information config ClusterInformation="default" error=Get https://[fd20::4001]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp [fd20::4001]:443: connect: permission denied
2019-07-30 07:58:22.979 [FATAL][1] main.go 118: Failed to initialize Calico datastore error=Get https://[fd20::4001]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp [fd20::4001]:443: connect: permission denied
According to the Calico documentation you may need to perform a few additional steps before you can start using it with ipv6 only. About enabling IPv6 with Kubernetes you can read here.
Related
Brand new to kubernetes, but managed to install kubernetes, ubuntu 20.04 LTS, but having issues with the dashboard. followed the procedure, using flannel as CNF.
The log states issues with connection to 10.96.0.1:443, but telnet seems to work? Any suggestion how to getting further ?
bwa#prod3:~$ kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-jgmpl 0/1 Running 1 27h 10.244.0.6 prod3 <none> <none>
kube-system coredns-66bff467f8-ldr9d 0/1 Running 1 27h 10.244.0.9 prod3 <none> <none>
kube-system etcd-prod3 1/1 Running 1 27h 192.168.0.93 prod3 <none> <none>
kube-system kube-apiserver-prod3 1/1 Running 1 27h 192.168.0.93 prod3 <none> <none>
kube-system kube-controller-manager-prod3 1/1 Running 1 27h 192.168.0.93 prod3 <none> <none>
kube-system kube-flannel-ds-amd64-xm26h 1/1 Running 2 27h 192.168.0.93 prod3 <none> <none>
kube-system kube-proxy-7lk5d 1/1 Running 1 27h 192.168.0.93 prod3 <none> <none>
kube-system kube-scheduler-prod3 1/1 Running 1 27h 192.168.0.93 prod3 <none> <none>
kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-xrdbh 1/1 Running 1 27h 10.244.0.7 prod3 <none> <none>
kubernetes-dashboard kubernetes-dashboard-7f99b75bf4-lfqtf 0/1 CrashLoopBackOff 310 27h 10.244.0.8 prod3 <none> <none>
bwa#prod3:~$ kubectl logs kubernetes-dashboard-7f99b75bf4-lfqtf --namespace=kubernetes-dashboard --tail=100
2020/08/05 12:02:31 Starting overwatch
2020/08/05 12:02:31 Using namespace: kubernetes-dashboard
2020/08/05 12:02:31 Using in-cluster config to connect to apiserver
2020/08/05 12:02:31 Using secret token for csrf signing
2020/08/05 12:02:31 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: Get "https://10.96.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf": dial tcp 10.96.0.1:443: i/o timeout
goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc00000c640)
/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:41 +0x446
github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:66
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc00044f800)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:501 +0xc6
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc00044f800)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:469 +0x47
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:550
main.main()
/home/runner/work/dashboard/dashboard/src/app/backend/dashboard.go:105 +0x20d
bwa#prod3:~$ telnet 10.96.0.1 443
Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.
^CConnection closed by foreign host.
bwa#prod3:~$
By the looks of that cluster, you do not have a networking plugin (CNI) installed. I do not see any flannel pods in the kube-system namespace, and the coredns pods are not starting.
This would also explain why the dashboard panics, as it is unable to reach the K8s API server via the 10.96.0.1 service.
Can you check the flannel installation (or just reinstall flannel on the cluster)?
My Master node ip address is 192.168.56.101. there is no node connected to master yet.
master#kmaster:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kmaster Ready master 125m v1.15.1
master#kmaster:~$
When i deployed my kubernetes-dashborad using below command, why running IP Address of kubernetes-dashboard-5c8f9556c4-f2jpz is 192.168.189.6
Similarly the other pods has also different IP address.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta1/aio/deploy/recommended.yaml
master#kmaster:~$ kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-kube-controllers-7bd78b474d-r2bwg 1/1 Running 0 113m 192.168.189.2 kmaster <none> <none>
kube-system calico-node-dsgqt 1/1 Running 0 113m 192.168.56.101 kmaster <none> <none>
kube-system coredns-5c98db65d4-n2wml 1/1 Running 0 114m 192.168.189.3 kmaster <none> <none>
kube-system coredns-5c98db65d4-v5qc8 1/1 Running 0 114m 192.168.189.1 kmaster <none> <none>
kube-system etcd-kmaster 1/1 Running 0 114m 192.168.56.101 kmaster <none> <none>
kube-system kube-apiserver-kmaster 1/1 Running 0 114m 192.168.56.101 kmaster <none> <none>
kube-system kube-controller-manager-kmaster 1/1 Running 0 114m 192.168.56.101 kmaster <none> <none>
kube-system kube-proxy-bgtmr 1/1 Running 0 114m 192.168.56.101 kmaster <none> <none>
kube-system kube-scheduler-kmaster 1/1 Running 0 114m 192.168.56.101 kmaster <none> <none>
kubernetes-dashboard kubernetes-dashboard-5c8f9556c4-f2jpz 1/1 Running 0 107m 192.168.189.6 kmaster <none> <none>
kubernetes-dashboard kubernetes-metrics-scraper-86456cdd8f-w45w2 1/1 Running 0 107m 192.168.189.4 kmaster <none> <none>
master#kmaster:~$
And also not able to access the kubernetes-dashboard UI. i am using the link
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/.
and the link KubeDNS https://192.168.56.101:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy is also not working.
but when trying to access Kubernetes master at https://192.168.56.101:6443 is working.
master#kmaster:~$ kubectl cluster-info
Kubernetes master is running at https://192.168.56.101:6443
KubeDNS is running at https://192.168.56.101:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Any suggestions.
Solution (see comments): Don't mix your physical and overlay network ranges.
Accessing the KubeDNS is only possible with DNS as protocol, not HTTP. If you want to query the DNS service you need to kubectl port-forward, not the HTTP (API) proxy.
If you try to access the dashboard with localhost:8081, you have to run kubectl proxy --port 8081 from your console to setup the proxy between you localhost to the k8s apiserver.
If you want to access dashboard from apiserver directly without the local proxy, try the following url https://192.168.56.101:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy (assuming your service name is kubernetes-dashboard)
You can also run kubectl port-forward svc/kubernetes-dashboard -n kubernetes-dashboard 443, then access the dashboard with https://localhost:443
I have a K8s cluster (1 master, 2 workers) running on 3 vagrant VMs on my computer.
I've installed kubernetes dashboard, like explained here.
All my pods are running correctly:
kubectl get pods -o wide --namespace=kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-n5cpm 1/1 Running 1 61m 10.244.0.4 kmaster.example.com <none> <none>
coredns-fb8b8dccf-qwcr4 1/1 Running 1 61m 10.244.0.5 kmaster.example.com <none> <none>
etcd-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-apiserver-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-controller-manager-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-flannel-ds-amd64-hcjsm 1/1 Running 1 61m 172.42.42.100 kmaster.example.com <none> <none>
kube-flannel-ds-amd64-klv4f 1/1 Running 3 56m 172.42.42.102 kworker2.example.com <none> <none>
kube-flannel-ds-amd64-lmpnd 1/1 Running 2 59m 172.42.42.101 kworker1.example.com <none> <none>
kube-proxy-86qsw 1/1 Running 1 59m 10.0.2.15 kworker1.example.com <none> <none>
kube-proxy-dp29s 1/1 Running 1 61m 172.42.42.100 kmaster.example.com <none> <none>
kube-proxy-gqqq9 1/1 Running 1 56m 10.0.2.15 kworker2.example.com <none> <none>
kube-scheduler-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kubernetes-dashboard-5f7b999d65-zqbbz 1/1 Running 1 28m 10.244.1.3 kworker1.example.com <none> <none>
As you can see the dashboard is in "Running" status.
I also ran kubectl proxy and it's serving on 127.0.0.1:8001.
But when I try to open http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ I have the error:
This site can’t be reached
127.0.0.1 refused to connect.
ERR_CONNECTION_REFUSED
I'm trying to open the dashboard directly on my computer, not inside the vagram VM. Could that be the problem? If yes, how to solve it ? I'm able to ping my VM from my computer without any issue.
Thanks for helping me.
EDIT
Here is the ouput of kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 96m
kubernetes-dashboard NodePort 10.109.230.83 <none> 443:30089/TCP 63m
Kubernetes dashboard runs only in the cluster as default. You can control it with get svc command:
kubectl get svc -n kube-system
Default type of that service is ClusterIp, to reach from outside of the cluster yo have to change it to NodePort.
To change it follow this doc.
Created a local cluster using Vagrant + Ansible + VirtualBox. Manually deploying works fine, but when using Helm:
:~$helm install stable/nginx-ingress --name nginx-ingress-controller --set rbac.create=true
Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 10.0.52.15:10250: i/o timeout
Kubernetes cluster info:
:~$kubectl get nodes,po,deploy,svc,ingress --all-namespaces -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/ubuntu18-kube-master Ready master 32m v1.13.3 10.0.51.15 <none> Ubuntu 18.04.1 LTS 4.15.0-43-generic docker://18.6.1
node/ubuntu18-kube-node-1 Ready <none> 31m v1.13.3 10.0.52.15 <none> Ubuntu 18.04.1 LTS 4.15.0-43-generic docker://18.6.1
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/nginx-server 1/1 Running 0 40s 10.244.1.5 ubuntu18-kube-node-1 <none> <none>
default pod/nginx-server-b8d78876d-cgbjt 1/1 Running 0 4m25s 10.244.1.4 ubuntu18-kube-node-1 <none> <none>
kube-system pod/coredns-86c58d9df4-5rsw2 1/1 Running 0 31m 10.244.0.2 ubuntu18-kube-master <none> <none>
kube-system pod/coredns-86c58d9df4-lfbvd 1/1 Running 0 31m 10.244.0.3 ubuntu18-kube-master <none> <none>
kube-system pod/etcd-ubuntu18-kube-master 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-apiserver-ubuntu18-kube-master 1/1 Running 0 30m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-controller-manager-ubuntu18-kube-master 1/1 Running 0 30m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-flannel-ds-amd64-jffqn 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-flannel-ds-amd64-vc6p2 1/1 Running 0 31m 10.0.52.15 ubuntu18-kube-node-1 <none> <none>
kube-system pod/kube-proxy-fbgmf 1/1 Running 0 31m 10.0.52.15 ubuntu18-kube-node-1 <none> <none>
kube-system pod/kube-proxy-jhs6b 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-scheduler-ubuntu18-kube-master 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/tiller-deploy-69ffbf64bc-x8lkc 1/1 Running 0 24m 10.244.1.2 ubuntu18-kube-node-1 <none> <none>
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
default deployment.extensions/nginx-server 1/1 1 1 4m25s nginx-server nginx run=nginx-server
kube-system deployment.extensions/coredns 2/2 2 2 32m coredns k8s.gcr.io/coredns:1.2.6 k8s-app=kube-dns
kube-system deployment.extensions/tiller-deploy 1/1 1 1 24m tiller gcr.io/kubernetes-helm/tiller:v2.12.3 app=helm,name=tiller
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 32m <none>
default service/nginx-server NodePort 10.99.84.201 <none> 80:31811/TCP 12s run=nginx-server
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 32m k8s-app=kube-dns
kube-system service/tiller-deploy ClusterIP 10.99.4.74 <none> 44134/TCP 24m app=helm,name=tiller
Vagrantfile:
...
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
$hosts.each_with_index do |(hostname, parameters), index|
ip_address = "#{$subnet}.#{$ip_offset + index}"
config.vm.define vm_name = hostname do |vm_config|
vm_config.vm.hostname = hostname
vm_config.vm.box = box
vm_config.vm.network "private_network", ip: ip_address
vm_config.vm.provider :virtualbox do |vb|
vb.gui = false
vb.name = hostname
vb.memory = parameters[:memory]
vb.cpus = parameters[:cpus]
vb.customize ['modifyvm', :id, '--macaddress1', "08002700005#{index}"]
vb.customize ['modifyvm', :id, '--natnet1', "10.0.5#{index}.0/24"]
end
end
end
end
Workaround for VirtualBox issue: set diffenrent macaddress and internal_ip.
It is interesting to find a solution that can be placed in one of the configuration files: vagrant, ansible roles. Any ideas on the problem?
Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 10.0.52.15:10250: i/o timeout
You're getting bitten by a very common kubernetes-on-Vagrant bug: the kubelet believes its IP address is eth0, which is the NAT interface in Vagrant, versus using (what I hope you have) the :private_address network in your Vagrantfile. Thus, since all kubelet interactions happen directly to it (and not through the API server), things like kubectl exec and kubectl logs will fail in exactly the way you see.
The solution is to force kubelet to bind to the private network interface, or I guess you could switch your Vagrantfile to use the bridge network, if that's an option for you -- just so long as the interface isn't the NAT one.
The question is about how you manage TLS Certificates in the cluster, ensure that port 10250 is reachable.
Here is an example of how i fix it when i try to run exec a pod running in node (instance aws in my case),
resource "aws_security_group" "My_VPC_Security_Group" {
...
ingress {
description = "TLS from VPC"
from_port = 10250
to_port = 10250
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
For more details you can visit [1]: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250.html
I'm trying to setup istio on Google container engine, istio has been installed successfully but booking sample has been failed to load.
Is there something I have configured in wrong way?
Help me, please!
Thanks in advance!
Here's what's I have tried:
kubectl get pods
details-v1-3121678156-3h2wx 2/2 Running 0 58m
grafana-1395297218-h0tjv 1/1 Running 0 5h
istio-ca-4001657623-n00zx 1/1 Running 0 5h
istio-egress-2038322175-0jtf5 1/1 Running 0 5h
istio-ingress-2247081378-fvr33 1/1 Running 0 5h
istio-mixer-2450814972-jrrm4 1/1 Running 0 5h
istio-pilot-1836659236-kw7cr 2/2 Running 0 5h
productpage-v1-1440812148-gqrgl 0/2 Pending 0 57m
prometheus-3067433533-fqcfw 1/1 Running 0 5h
ratings-v1-3755476866-jbh80 2/2 Running 0 58m
reviews-v1-3728017321-0m7mk 0/2 Pending 0 58m
reviews-v2-196544427-6ftf5 0/2 Pending 0 58m
reviews-v3-959055789-079xz 0/2 Pending 0 57m
servicegraph-3127588006-03b93 1/1 Running 0 5h
zipkin-4057566570-0cb86 1/1 Running 0 5h
kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
details 10.11.249.214 <none> 9080/TCP
grafana 10.11.247.226 104.199.211.175 3000:31036/TCP
istio-egress 10.11.246.60 <none> 80/TCP
istio-ingress 10.11.242.178 35.189.165.119 80:31622/TCP,443:31241/TCP
istio-mixer 10.11.242.104 <none> 9091/TCP,9094/TCP,42422/TCP
istio-pilot 10.11.251.240 <none> 8080/TCP,8081/TCP
kubernetes 10.11.240.1 <none> 443/TCP
productpage 10.11.255.53 <none> 9080/TCP
prometheus 10.11.248.237 130.211.249.66 9090:32056/TCP
ratings 10.11.252.40 <none> 9080/TCP
reviews 10.11.242.168 <none> 9080/TCP
servicegraph 10.11.252.60 35.185.161.219 8088:32709/TCP
zipkin 10.11.245.4 35.185.144.62 9411:31677/TCP
get ingress IP and export env variable then curl
NAME HOSTS ADDRESS PORTS AGE
gateway * 35.189.165.119 80 1h
Abduls-MacBook-Pro:~ abdul$ export GATEWAY_URL=35.189.165.119:80
Abduls-MacBook-Pro:~ abdul$ curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage
000
I ran into a similar issue ("upstream connect error or disconnect/reset before headers") when i deployed the istio sample app on GKE. Try to delete all pods (and wait for all of to come up again). Then restart your proxy...
kubectl delete pods --all