We have a rancher based kubernetes cluster with calico on openstack.
A spring config server (server pod here onwards) is deployed as a service.
Service is exposed on nodeport.
Another pod is deployed with curl (client pod here onwards)
Doing a curl from client pod on server pod nodeport with its node ip on which server pod is running gives proper result.
Doing curl from client pod on server nodeport with another node's ip where server pod is not running pod gives curl(56) connection reset by peer for bigger response
Doing curl from client pod on service and it's port gives results for small data but for bigger response again gives curl(56)
If both client and server pods are running on same node, response is fine.
My understanding is:
No issues in server pod, as able to get response on nodeport
No issues in client pod/curl as able to get response from nodeport
Service and pod linkage is fine as it works well with small response size
When I say bigger response, I mean just 1 kb+
It will give an error when there is no reply from the server.
Ex:
$ curl http://youtube.com:443
Curl: (52) Empty reply from server
Please recheck your proxy, firewall settings.
Related
I've developed a python script, using python kubernetes-client to harvest Pods' internal IPs.
But when I try to make an http request to these IPs, from another pod, I get Connection refused error.
I spin up a temporary curl container:
kubectl run curl --image=radial/busyboxplus:curl -it --rm
And having the internal IP of one of the pods, I try to make a GET request:
curl http://10.133.0.2/stats
and the response is:
curl: (7) Failed to connect to 10.133.0.2 port 80: Connection refused
Both pods are in the same default namespace and use the same default ServiceAccount.
I know that I can call the Pods thru the ClusterIP service by which they're load-balanced, but this way I will only access a single Pod at random (depending which one the service forwards the call to), when I have multiple replicas of the same Deployment.
I need to be able to call each Pod of a multi-replica Deployment separately. That's why I'm going for the internal IPs.
I guess you missed the port number here
It should be like this
curl POD_IP:PORT/stats
I have a kubernetes cluster on EKS, in which I would like to send requests to the kubelet server (not the kube-apiserver, but the kubelet, which runs on each of the worker nodes).
My understanding is that the kubelet runs an http server on port 10250, so I opened the firewall (security group) in one of the worker nodes for that port so I can reach it from my IP. Example of a request:
curl -k https://public-ip-of-worker-node:10250/metrics/probes
but I get a 401 in response. I guess this is expected, as I am not authenticating in any way.
So, how can I authenticate to the kubelet server? I can communicate without problem with the kube-apiserver using kubectl, so I do have enough permissions from the IAM side.
From the docs start the kubelet with the --authentication-token-webhook and the --kubeconfig flags.
Then you can create a service-account and define role and rolebinding on the service account and use the service accounts bearer token with the curl command to call the kubelet API.
Our on-premise Kubernetes/Kubespray cluster has suddenly stopped routing traffic between the nginx-ingress and node port services. All external requests to the ingress endpoint return a "504 - gateway timeout" error.
How do I diagnose what has broken?
I've confirmed that the containers/pods are running, the node application has started and if I exec into the pod then I can run a local curl command and get a response from the app.
I've checked the logs on the ingress pods and traffic is arriving and nginx is trying to forward the traffic on to the service endpoint/node port but it is reporting an error.
I've also tried to curl directly to the node via the node port but I get no response.
I've looked at the ipvs configuration and the settings look valid (e.g. there are rules for the node to forward traffic on the node port the service endpoint address/port)
We couldn't resolve this issue and, in the end, the only workaround was to uninstall and reinstall the cluster.
I was getting this because the nginx ingress controller pod was running out of memory, I just increased the memory for the pod and it worked.
I was facing a similar issue and the simple fix was to increase the values for the K8S_CPU_LIMIT and K8S_MEMORY_LIMIT for the application pods running on the cluster.
I've been running my kubernetes masters separate from my kubernetes nodes. So I have kube-apiserver, kube-scheduler and kube-controllermanager running on a server without kubelet, kube-proxy or flannel.
So far this has worked perfectly. However, today I attempted to set up the Web UI and access it through an API server. I got the the following error when accessing http://kube-master-0:8080/ui:
Error: 'dial tcp 172.16.72.12:9090: getsockopt: connection timed out'
Trying to reach: 'http://172.16.72.12:9090/'
This suggests to me that the API server is trying to connect to the pod IP, as we don't have flannel or kube-proxy running on this host, the 172.16.72.12 IP will not be routed.
Am I expected to run kube-proxy and flannel on my API servers? Is there another way to let the API server proxy the UI?
It's not required, but it will certainly make your life easier.
The reason this isn't working is because kube-proxy isn't directing traffic to the service. Try kube-node:8080/ui (assuming you have exposed it as with NodePort configuration
In theory, Kube apiserver does not expect the presence of kube-proxy.
This means kube apiserver will run correctly, receives requests and handles them(mostly reads from and writes to etcd).
But if you want the whole cluster working, you will need other components running, for example:
if you want pods or deployments to be scheduled, kube-scheduler should be running
if you want pods and containers be running in nodes, kubelet has to be running
if you want replications can be guarded, controller-manager should be runing
As for kube-proxy and flannel, they are critical parts to make sure networking is working. Load Balance, service, across-hosts pod communication etc all depends on them.
I've a nginx service exposed via NodePort. According to the documentation, I should now be able to hit the service on $NODE_IP:$NODE_PORT for all my K8 worker IPs. However, I'm able to access the service via curl on only the node that hosts the actual nginx pod. Any idea why?
I did verify using netstat that kube-proxy is listening on $NODE_PORT on all the hosts. Somehow, the request is not being forwarded to the actual pod by kube-proxy.
This turned out to be an issue with the security group associated with the workers. I had opened only the ports in the --service-node-port-range. This was not enough because I was deploying nginx on port 80 and kube-proxy tried to forward the request to the pod's IP on port 80 but was being blocked.