I have set up a kubernetes Cluster manually. The cluster is healthy. The nodes are up. The pods and services are also created and running.
I have a web pod which is a python flask application. A db-pod which is redis. Exposed redis as a service to be accessible from python. Exposed web pod as external service also. The external service is running in 31727 port.
When i access the web application through browser, it reports redis host is not accessible.
The application works well when deployed in a kubernetes cluster created using kubeadm/kops.
Sounds like kube-proxy or overlay networking issue at first glance. Are you sure kube-proxy is launched on nodes and you have a working overlay ? Can you ping pods directly on a pod-to-pod basis ?
Update: as your pod-to-pod connectivity is down, you need to look into your flannel configuration, and make sure it works fine, as well as make sure pods are started with flannel networking (ie. via CNI) rather than local docker0 interfaces network.
Related
I am running Cockpit on my host, its running as systemd service, at the same time I am using microk8s (single node k8s).
I dont need reverse proxy to Cockpit web ui, but I need something like external service/endpoint k8s object mapping, .. to be k8s aware of that service.
k8s should not allow to create another nodePort service with same port ( k8s layer, it should not collide on OS layer )
k8s should handle rules for that service in iptables.
Btw: Cockpit must be running as standalone application directly on host.
Any ideas?
Thanks
I have deployed two POD-s with hostnetwork set to true. When the POD-s are deployed on same OpenShfit node then everything works fine since they can discover each other using node IP.
When the POD-s are deployed on different OpenShift nodes then they cant discover each other, I get no route to host if I want to point one POD to another using node IP. How to fix this?
The uswitch/kiam (https://github.com/uswitch/kiam) service is a good example of a use case.
it has an agent process that runs on the hostnetwork of all worker nodes because it modifies a firewall rule to intercept API requests (from containers running on the host) to the AWS api.
it also has a server process that runs on the hostnetwork to access the AWS api since the AWS api is on a subnet that is only available to the host network.
finally... the agent talks to the server using GRPC which connects directly to one of the IP addresses that are returned when looking up the kiam-server.
so you have pods of the agent deployment running on the hostnetwork of node A trying to connect to kiam server running on the hostnetwork of node B.... which just does not work.
furthermore, this is a private service... it should not be available from outside the network.
If you want the two containers to be share the same physical machine and take advantage of loopback for quick communications, then you would be better off defining them together as a single Pod with two containers.
If the two containers are meant to float over a larger cluster and be more loosely coupled, then I'd recommend taking advantage of the Service construct within Kubernetes (under OpenShift) and using that for the appropriate discovery.
Services are documented at https://kubernetes.io/docs/concepts/services-networking/service/, and along with an internal DNS service (if implemented - common in Kubernetes 1.4 and later) they provide a means to let Kubernetes manage where things are, updating an internal DNS entry in the form of <servicename>.<namespace>.svc.cluster.local. So for example, if you set up a Pod with a service named "backend" in the default namespace, the other Pod could reference it as backend.default.svc.cluster.local. The Kubernetes documentation on the DNS portion of this is available at https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
This also avoids the "hostnetwork=true" complication, and lets OpenShift (or specifically Kubernetes) manage the networking.
If you have to absolutely use hostnetwork, you should be creating router and then use those routers to have the communication between pods. You can create ha proxy based router in opeshift, reference here --https://docs.openshift.com/enterprise/3.0/install_config/install/deploy_router.html
I'm running a kubernetes cluster on AWS using Weave with a private topology. I have some multi-node applications (like Spark) that have a UI web page. I can expose that via a load balancer, but all the links to the workers, etc. use the k8s local ip addresses. Is it possible (via kubectl proxy or otherwise) to temporarily "go inside" the k8s network from a browser on my laptop, so that all the k8s internal ips work as expected? I'm not looking to expose everything to the outside, but to be able to temporarily browse for things from my laptop.
You can use weave expose to expose weave Subnet.
You should be able to use kubectl port-forward my-container-name localport:serviceport on your laptop (where service port is the port exposed by your WebUI service). Then you should be able to browse to localhost:localport and everything should work as expected.
Alternatively you may need to SSH into one of the private nodes via a bastion host.
We wanted podnames to be resolved to IP's to configure the seed nodes in an akka cluster. This was happenning by using the concept of a headless service and stateful sets in Kubernetes. But, how do I expose a headless service externally to hit an endpoint from outside?
It is hard to expose a Kubernetes service to the outside, since this would require some complex TCP proxies. The reason for this is, that the headless services is only a DNS record with an IP for each pod. But these IPs are only reachable from within the cluster.
One solution is to expose this via Node ports, which means the ports are opened on the host itself. Unfortunately this makes the service discovery harder, because you don't know which host has a scheduled pod on it.
You can setup node ports via:
the services: https://kubernetes.io/docs/user-guide/services/#type-nodeport
or directly in the Pod by defining spec.containers[].ports[].hostPort
Another alternative is to use a LoadBalancer, if your cloud provider supports that. Unfortunately you cannot address each instance itself, since they share the same IP. This might not be suitable for your application.
I am working on a POC, and i find out some strange behavior after setting up my kubernetes cluster
In fact, i am working on a topology of one master and two minions.
When i tried to make up 2 pods into each minion and expose a service for them, it turned out that when i try to request the service from the master, nothing is returned (any response from 2 pods) and when i try to request the service from a minion, only the pod deployed in that minion respond but the other no.
This can heavily depend on how your cluster is provisioned.
For starters, you need to validate how networking is set up and if it works as kubernetes expects. Said short, if you launch two pods (on separate nodes), they should get IPs from their dedicated per node ranges, and be able to route that between nodes. You can use some small(ish) base image (alpine/debian/ubuntu etc.), with something like sleep 1d , exec into them interactively with bash and simply ping one from the other. If it does not work, your network setup is broken.
Make sure you test between pods, not directly from node host OS. In some configurations node is unable to access service IPs due to routing concerns, but pod-to-pod works fine (seen this in some flannel configurations)
Also, your networking is probably provided by some overlay network solution like flannel, weave, calico etc. so check their respective logs for signs of problems.