I can't seem to get a node to join the cluster.
[discovery] Trying to connect to API Server "10.0.2.15:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.2.15:6443"
I0702 11:09:08.268102 10342 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.0 (linux/amd64) kubernetes/91e7b4f" 'https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0702 11:09:08.268676 10342 round_trippers.go:405] GET https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info in 0 milliseconds
I0702 11:09:08.268873 10342 round_trippers.go:411] Response Headers:
[discovery] Failed to request cluster info, will try again: [Get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: connect: connection refused]
The port seems closed (from the node):
telnet 10.0.2.15 6443
Trying 10.0.2.15...
telnet: Unable to connect to remote host: Connection refused
While on the master:
telnet 10.0.2.15 6443
Trying 10.0.2.15...
Connected to 10.0.2.15.
Escape character is '^]'.
^CConnection closed by foreign host.
What may be the cause of this?
Both machines are virtual machines and 10.02.15 is the NAT ip - which is the same for both machines (they are independent)...
Sigh...
In the event it is helpful to someone else:
iptables -t raw -A OUTPUT -p tcp --dport 6443 -j TRACE
iptables -t raw -A PREROUTING -p tcp --dport 6443 -j TRACE
tail -f /var/log/kern.log
If you are running on VM (like using vagrant and virtual box) run init command with the private IP which is used in the vagrant file. So if you use join command on a node it is able to reach it else no.
Syntax: kubeadm init
--apiserver-advertise-address=private-ip-address
Example: kubeadm init --apiserver-advertise-address=192.168.33.50
Related
I'm trying to connect to my google vps server, but constantly getting error :
java.net.SocketTimeoutException: timeout
java.net.SocketException: Socket closed
I've created system service on my server, which listens on the port 8080. I've forwarded the default http traffic to port 8080 and made sure ports 80 & 8080 are open:
iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
I've also saved the iptable rules :
sudo apt-get install iptables-persistent
I've checked if the service actually listens on the port by sudo netstat -tunlp:
tcp6 0 0 :::8080 :::* LISTEN 5789/java -> it does
This is my retrofitBuilder in app, which is trying to connect to server with standard http port :
return Retrofit.Builder()
.baseUrl("http://34.118.22.134/")
.addConverterFactory(MoshiConverterFactory.create())
.build()
.create()
}
When testing the service locally, it works as expected,also the service on the servers works fine.
When I do sudo ss -ltnp, I see that port 80 is not in "listening state", only port 8080 and several others are. I dont want to use uwf to enable it because that will disrupt the SSH connection.
The postman can't reach server as well and it's throwing 500-internal server error.
I do not manipulate sockets in code in any way.
I am trying to understand the istio and envoy behavior and how the proxy works!
Lets assume that I created an application which keeps on the sending the request to google search API. When I deploy that in my k8s cluster with istio and envoy as a sidecar container, It is said all the requests are routed via the proxy/sidecar container.
My question is - both the application and the proxy/sidecar are running in the same pod and sharing the same IP. In order for the app to send the request to sidecar, it should be modified to send the request to the localhost (ie to the proxy server port) so that it can forward to google. But how the outbound requests of one application is routed to another application. Where is this configuration maintained?
Can someone who understood this well please explain?
istio-init init container is used to setup the iptables rules so that inbound/outbound traffic will go through the sidecar proxy. An init container is different than an app container in following ways:
It runs before an app container is started and it always runs to completion.
If there are many init containers, each should complete with success before the next container is started.
So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules.
istio-proxy This is the actual sidecar proxy (based on Envoy).
Get into the application pod and look at the configured iptables. I am going to show an example using nsenter. Alternatively, you can enter the container in a privileged mode to see the same information. For folks without access to the nodes, using exec to get into the sidecar and running iptables is more practical.
$ docker inspect b8de099d3510 --format '{{ .State.Pid }}'
4125
$ nsenter -t 4215 -n iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISTIO_INBOUND
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-N ISTIO_REDIRECT
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 80 -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
The output above clearly shows that all the incoming traffic to port 80, which is the port the application is listening, is now REDIRECTED to port 15001, which is the port that the istio-proxy, an Envoy proxy, is listening. The same holds true for the outgoing traffic.
Update: In place of istio-init, there now seems to be an option of using the new CNI, which removes the need for the init container and associated privileges. This istio-cni plugin sets up the pods’ networking to fulfill this requirement in place of the current Istio injected pod istio-init approach.
https://istio.io/blog/2019/data-plane-setup/#traffic-flow-from-application-container-to-sidecar-proxy
OVERVIEW:: I am studying for the Kubernetes Administrator certification. To complete the training course, I created a dual node Kubernetes cluster on Google Cloud, 1 master and 1 slave. As I don't want to leave the instances alive all the time, I took snapshots of them to deploy new instances with the Kubernetes cluster already setup. I am aware that I would need to update the ens4 ip used by kubectl, as this will have changed, which I did.
ISSUE:: When I run "kubectl get pods --all-namespaces" I get the error "The connection to the server localhost:8080 was refused - did you specify the right host or port?"
QUESTION:: Would anyone have had similar issues and know if its possible to recreate a Kubernetes cluster from snapshots?
Adding -v=10 to command, the url matches info in .kube/config file
kubectl get pods --all-namespaces -v=10
I0214 17:11:35.317678 6246 loader.go:375] Config loaded from file: /home/student/.kube/config
I0214 17:11:35.321941 6246 round_trippers.go:423] curl -k -v -XGET -H "User-Agent: kubectl/v1.16.1 (linux/amd64) kubernetes/d647ddb" -H "Accept: application/json, /" 'https://k8smaster:6443/api?timeout=32s'
I0214 17:11:35.333308 6246 round_trippers.go:443] GET https://k8smaster:6443/api?timeout=32s in 11 milliseconds
I0214 17:11:35.333335 6246 round_trippers.go:449] Response Headers:
I0214 17:11:35.333422 6246 cached_discovery.go:121] skipped caching discovery info due to Get https://k8smaster:6443/api?timeout=32s: dial tcp 10.128.0.7:6443: connect: connection refused
I0214 17:11:35.333858 6246 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.16.1 (linux/amd64) kubernetes/d647ddb" 'https://k8smaster:6443/api?timeout=32s'
I0214 17:11:35.334234 6246 round_trippers.go:443] GET https://k8smaster:6443/api?timeout=32s in 0 milliseconds
I0214 17:11:35.334254 6246 round_trippers.go:449] Response Headers:
I0214 17:11:35.334281 6246 cached_discovery.go:121] skipped caching discovery info due to Get https://k8smaster:6443/api?timeout=32s: dial tcp 10.128.0.7:6443: connect: connection refused
I0214 17:11:35.334303 6246 shortcut.go:89] Error loading discovery information: Get https://k8smaster:6443/api?timeout=32s: dial tcp 10.128.0.7:6443: connect: connection refused
I replicated you issue and wrote this step by step debugging process for you so you can see what was my thinking.
I created 2 node cluster (master + worker) with kubeadm and made a snapshot.
Then I deleted all nodes and recreated them from snapshots.
After recreating master node from snapshot I started seeing the same error you are seeing:
#kmaster ~]$ kubectl get po -v=10
I0217 11:04:38.397823 3372 loader.go:375] Config loaded from file: /home/user/.kube/config
I0217 11:04:38.398909 3372 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://10.156.0.20:6443/api?timeout=32s'
^C
The connection was hanging so I interrupted it (ctrl+c).
First I noticed was that IP address of where kubectl was connecting was different than node ip, so I modified .kube/config file providing proper IP.
After doing this, here is what running kubectl showed:
$ kubectl get po -v=10
I0217 11:26:57.020744 15929 loader.go:375] Config loaded from file: /home/user/.kube/config
...
I0217 11:26:57.025155 15929 helpers.go:221] Connection error: Get https://10.156.0.23:6443/api?timeout=32s: dial tcp 10.156.0.23:6443: connect: connection refused
F0217 11:26:57.025201 15929 helpers.go:114] The connection to the server 10.156.0.23:6443 was refused - did you specify the right host or port?
As you see, connection to apiserver was beeing refused so I checked if apiserver was running:
$ sudo docker ps -a | grep apiserver
5e957ff48d11 90d27391b780 "kube-apiserver --ad…" 24 seconds ago Exited (2) 3 seconds ago k8s_kube-apiserver_kube-apiserver-kmaster_kube-system_997514ff25ec38012de6a5be7c43b0ae_14
d78e179f1565 k8s.gcr.io/pause:3.1 "/pause" 26 minutes ago Up 26 minutes k8s_POD_kube-apiserver-kmaster_kube-system_997514ff25ec38012de6a5be7c43b0ae_1
api-server was exiting for some reason.
I checked its logs (I am only including relevant logs for readability):
$ sudo docker logs 5e957ff48d11
...
W0217 11:30:46.710541 1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
panic: context deadline exceeded
Notice apiserver was trying to connect to etcd (notice port: 2379) and receiving connection refused.
My first guess was etcd wasn't running, so I checked etcd container:
$ sudo docker ps -a | grep etcd
4a249cb0743b 303ce5db0e90 "etcd --advertise-cl…" 2 minutes ago Exited (1) 2 minutes ago k8s_etcd_etcd-kmaster_kube-system_9018aafee02ebb028a7befd10063ec1e_19
b89b7e7227de k8s.gcr.io/pause:3.1 "/pause" 30 minutes ago Up 30 minutes k8s_POD_etcd-kmaster_kube-system_9018aafee02ebb028a7befd10063ec1e_1
I was right: Exited (1) 2 minutes ago. I checked its logs:
$ sudo docker logs 4a249cb0743b
...
2020-02-17 11:34:31.493215 C | etcdmain: listen tcp 10.156.0.20:2380: bind: cannot assign requested address
etcd was trying to bind with old IP address.
I modified /etc/kubernetes/manifests/etcd.yaml and changed old IP address to new IP everywhere in file.
Quick sudo docker ps | grep etcd showed its running.
After a while apierver also started running.
Then I tried running kubectl:
$ kubectl get po
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.156.0.20, not 10.156.0.23
Invalid apiserver certificate. SSL certificate was genereated for old IP so that would mean I need to generate new certificate with new IP.
$ sudo kubeadm init phase certs apiserver
...
[certs] Using existing apiserver certificate and key on disk
That's not what I expected. I wanted to generate new certificates, not use old ones.
I deleted old certificates:
$ sudo rm /etc/kubernetes/pki/apiserver.crt \
/etc/kubernetes/pki/apiserver.key
And tried to generate certificates one more time:
$ sudo kubeadm init phase certs apiserver
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kmaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.156.0.23]
Looks good. Now let's try using kubectl:
$ kubectl get no
NAME STATUS ROLES AGE VERSION
instance-21 Ready master 102m v1.17.3
instance-22 Ready <none> 95m v1.17.3
As you can see now its working.
In my Kubernetes environment I have following to pods running
NAME READY STATUS RESTARTS AGE IP NODE
httpd-6cc5cff4f6-5j2p2 1/1 Running 0 1h 172.16.44.12 node01
tomcat-68ccbb7d9d-c2n5m 1/1 Running 0 45m 172.16.44.13 node02
One is a Tomcat instance and other one is a Apache instance.
from node01 and node02 I can curl the httpd which is using port 80. But If i curl the tomcat server which is running on node2 from node1 it fails. I get below output.
[root#node1~]# curl -v 172.16.44.13:8080
* About to connect() to 172.16.44.13 port 8080 (#0)
* Trying 172.16.44.13...
* Connected to 172.16.44.13 (172.16.44.13) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.16.44.13:8080
> Accept: */*
>
^C
[root#node1~]# wget -v 172.16.44.13:8080
--2019-01-16 12:00:21-- http://172.16.44.13:8080/
Connecting to 172.16.44.13:8080... connected.
HTTP request sent, awaiting response...
But I'm able telnet to port 8080 on 172.16.44.13 from node1
[root#node1~]# telnet 172.16.44.13 8080
Trying 172.16.44.13...
Connected to 172.16.44.13.
Escape character is '^]'.
^]
telnet>
Any reason for this behavior? why am I able to telnet but unable to get the web content? I have also tried different ports but curl is working only for port 80.
I was able to get this fixed by disabling selinux on my nodes.
I have a kubernetes cluster which has 2 interfaces:
eth0: 10.10.10.100 (internal)
eth1: 20.20.20.100 (External)
There are few pods running in the cluster with flannel networking.
POD1: 172.16.54.4 (nginx service)
I want to access 20.20.20.100:80 from another host which is connected to the above k8s cluster, so that I can reach the nginx POD.
I had enabled ip forwarding and also added DNAT rules as follows:
iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.54.4:80
After this when I try to do a curl on 20.20.20.100, I get
Failed to connect to 10.10.65.161 port 80: Connection refused
How do I get this working?
You can try
iptables -t nat -A PREROUTING -p tcp -d 20.20.20.100 --dport 80 -j DNAT --to-destination 172.16.54.4:80
But I don't recommend that you manage the iptables by yourself, it's painful to maintain the rules...
You can use the hostPort in the k8s. You can use kubenet as network plugin, since cni plugin does not support hostPort.
why not use nodeport type? I think it is a better way to access service by hostIP. Please try iptables -nvL -t nat and show me the detail.