I'm attempting to transition existing applications to Kubernetes that work as follows:
An outside service calls our application through a load balancer with a new session.
Our application returns the ip of the server that processed the request.
All subsequent calls from the outside service for that session are made directly to the same server (bypassing the load balancer)
Is there any way to do this in kubernetes? I understand that pod ip's are not exposed externally, is there some way to expose them directly?
Also, I don't think I can use sessionAffinity="ClientIP" because the requests will all come in from the same place. Is there a way to write custom sessionAffinity type?
It kind of depends on how your network is set up and what you mean by an "outside service", but the answer is most likely "no".
If you're running using one of the default cluster creation scripts in a cloud environment, pod IP addresses are not routable from the Internet, so any service not in the same private network as your cluster won't be able to talk directly to pods.
However, depending on what cloud provider you're on, you'll likely get the behavior that you want anyways by just continuing to make all calls through to the external IP of a service of type LoadBalancer. For instance, on the Google Cloud Platform, the cloud load balancer that gets created for such services by default maintains connection affinity by 5-tuple (src ip and port, dst ip and port, L4 protocol), which sounds like it's what you want, since you want balancing per session rather than per IP.
As for creating a new sessionAffinity type, that's not an easy thing to extend, since it requires changing Kubernetes source code. If that's really a path you want to take, it's likely that you'd want to run your own load balancer within your cluster rather than relying on the built-in load balancing.
Related
I am learning the headless service of kubernetes.
I understand the following without question (please correct me if I am wrong):
A headless service doesn't have a cluster IP,
It is used for communicating with stateful app
When client app container/pod communicates with a database pod via headless service the pod IP address is returned instead of the service's.
What I don't quite sure:
Many articles on internet explaining headless service is vague in my opinion. Because all I found only directly state something like :
If you don't need load balancing but want to directly connect to the
pod (e.g. database) you can use headless service
But what does it mean exactly?
So, following are my thoughts of headless service in k8s & two questions with an example
Let's say I have 3 replicas of PostgreSQL database instance behind a service, if it is a regular service I know by default request to database would be routed in a round-robin fasion to one of the three database pod. That's indeed a load balancing.
Question 1:
If using headless service instead, does the above quoted statement mean the headless service will stick with one of the three database pod, never change until the pod dies? I ask this because otherwise it would still be doing load balancing if not stick with one of the three pod. Could some one please clarify it?
Question 2:
I feel no matter it is regular service or headless service, client application just need to know the DNS name of the service to communicate with database in k8s cluster. Isn't it so? I mean what's the point of using the headless service then? To me the headless service only makes sense if client application code really needs to know the IP address of the pod it connects to. So, as long as client application doesn't need to know the IP address it can always communicate with database either with regular service or with headless service via the service DNS name in cluster, Am I right here?
A normal Service comes with a load balancer (even if it's a ClusterIP-type Service). That load balancer has an IP address. The in-cluster DNS name of the Service resolves to the load balancer's IP address, which then forwards to the selected Pods.
A headless Service doesn't have a load balancer. The DNS name of the Service resolves to the IP addresses of the Pods themselves.
This means that, with a headless Service, basically everything is up to the caller. If the caller does a DNS lookup, picks the first address it's given, and uses that address for the lifetime of the process, then it won't round-robin requests between backing Pods, and it will not notice if that Pod disappears. With a normal Service, so long as the caller gets the Service's (cluster-internal load balancer's) IP address, these concerns are handled automatically.
A headless Service isn't specifically tied to stateful workloads, except that StatefulSets require a headless Service as part of their configuration. An individual StatefulSet Pod will actually be given a unique hostname connected to that headless Service. You can have both normal and headless Services pointing at the same Pods, though, and it might make sense to use a normal Service for cases where you don't care which replica is (initially) contacted.
A headless service will return all Pod IPs that are associated through the selector. The order is not stable, so if a client is making repeated DNS queries and uses only the first returned IP, this will result in some kind of load balancing as well.
Regarding your second question: That is correct. In general, if a client does not need to know all instances - and handle the unstable IPs - a regular service provides more benefits.
I have an app, which I want to run on Kubernetes (currently on AWS ECS). The app has two TCP ports, neither is http. One port, say APORT is common across all the app instances (replicas) and should be load balanced. The other, lets call it a BPORT, however, is specific to this particular instance of the app, e.g. pod/container specific.
Now here is my problem: the app register it's BPORT with an external controller and the controller should be able to reach this app via that port. I can use NodePort to expose that NodePort to the external IP. From my pod, I will obtain a value of that NodePort and register with the external controller.
However, a service only assigns single NodePort across all replicas, so if I want multiple replicas, I have to run multiple services.
Running multiple services presents a problem on the APORT side, as this port should be load balanced, ideally sitting behind IPVS, and as far as I understand IPVS does not allow to LB between multiple services.
Another wrinkle that ideally I would like to be able to add more replicas to scale this whole thing without service interruption/restarts.
Any ideas? Thanks!
Kubernetes doesn't really handle this, or at least it doesn't get involved. You can use a Service object for A port like normal, but for the B port, you wouldn't use a Service at all, things would have to directly use the pod IP instead, just like if these were servers rather than containers.
I was using NodePort to host a webapp on Google Container Engine (GKE). It allows you to directly point your domains to the node IP address, instead of an expensive Google load balancer. Unfortunately, instances are created with HTTP ports blocked by default, and an update locked down manually changing the nodes, as they are now created using and Instance Group/and an Immutable Instance Template.
I need to open port 443 on my nodes, how do I do that with Kubernetes or GCE? Preferably in an update resistant way.
Related github question: https://github.com/nginxinc/kubernetes-ingress/issues/502
Using port 443 on your Kubernetes nodes is not a standard practice. If you look at the docs you and see the kubelet option --service-node-port-range which defaults to 30000-32767. You could change it to 443-32767 or something. Note that every port under 1024 is restricted to root.
In summary, it's not a good idea/practice to run your Kubernetes services on port 443. A more typical scenario would be an external nginx/haproxy proxy that sends traffic to the NodePorts of your service. The other option you mentioned is using a cloud load balancer but you'd like to avoid that due to costs.
Update: A deamonset with a nodeport can handle the port opening for you. nginx/k8s-ingress has a nodeport on 443 which gets exposed by a custom firewall rule. the GCE UI will not show「Allow HTTPS traffic」as checked, because its not using the default rule.
You can do everything you do on the GUI Google Cloud Console using the Cloud SDK, most easily through the Google Cloud Shell. Here is the command for adding a network tag to a running instance. This works, even though the GUI disabled the ability to do so
gcloud compute instances add-tags gke-clusty-pool-0-7696af58-52nf --zone=us-central1-b --tags https-server,http-server
This also works on the beta, meaning it should continue to work for a bit.
See https://cloud.google.com/sdk/docs/scripting-gcloud for examples on how to automate this. Perhaps consider running on a webhook when downtime is detected. Obviously none of this is ideal.
Alternatively, you can change the templates themselves. With this method you can also add a startup to new nodes, which allows you do do things like fire a webhook with the new IP Address for a round robin low downtime dynamic dns.
Source (he had the opposite problem, his problem is our solution): https://stackoverflow.com/a/51866195/370238
If I understand correctly, if nodes can be destroyed and recreated themselves , how are you going to rest assured that certain service behind port reliably available on production w/o any sort of load balancer which takes care of route orchestration diverting port traffic to new node(s)
Let's say I'm using an GCE ingress to handle traffic from outside the cluster and terminate TLS (https://example.com/api/items), from here the request gets routed to one of two services that are only available inside the cluster. So far so good.
What if I have to call service B from service A, should I go all the way and use the cluster's external IP/domain and use HTTPS (https://example.com/api/user/1) to call the service or could I use the internal IP of the service and use HTTP (http://serviceb/api/user/1)? Do I have to encrypt the data or is it "safe" as long as it isn't leaving the private k8s network?
What if I want to have "internal" endpoints that should only be accessible from within the cluster - when I'm always using the external https-url those endpoints would be reachable for everyone. Calling the service directly, I could just do a http://serviceb/internal/info/abc.
What if I have to call service B from service A, should I go all the way and use the cluster's external IP/domain and use HTTPS (https://example.com/api/user/1) to call the service or could I use the internal IP of the service and use HTTP (http://serviceb/api/user/1)?
If you need to use the features that you API Gateway is offering (authentication, cache, high availability, load balancing) then YES, otherwise DON'T. The External facing API should contain only endpoints that are used by external clients (from outside the cluster).
Do I have to encrypt the data or is it "safe" as long as it isn't leaving the private k8s network?
"safe" is a very relative word and I believe that there are no 100% safe networks. You should put in the balance the probability of "somebody" or "something" sniffing data from the network and the impact that it has on your business if that happens.
If this helps you: for any project that I've worked for (or I heard from somebody I know), the private network between containers/services was more than sufficient.
What if I want to have "internal" endpoints that should only be accessible from within the cluster - when I'm always using the external https-url those endpoints would be reachable for everyone.
Exactly what I was saying on top of the answer. Keeping those endpoints inside the cluster makes them inaccessible by design from outside.
One last thing, managing a lot of SSL certificates for a lot of internal services is a pain that one should avoid if not necessary.
I am pretty new to kubernetes and I have successfully setup a cluster on google container engine .
In my cluster I have a backend api developed with dropwizard, front end developed with node js and a mysql database.
All have been deployed to the cluster and are working .However my challenge is this after setting up an external ip for my node containers and backend I can access them remotely but I can't access my backed from my front end using the service name e.g my backend is called backendapi within the cluster. I can't do this http://backendapi:8080 to call my rest services when deployed to the cluster .
The catch for me is when I deploy to the cluster I don't want my front end to hit my back end using the external ip, I want them to connect within the cluster without going via the external ip address. When I connect to a pod and ping backendapi it returns a result but when I deploy my front end and use the label name it doesn't work .What could I be doing wrong ?.
As long as kube-dns is running (which I believe is "always unless you disable it"), all Service objects have an in cluster DNS name of service_name +"."+ service_namespace + ".svc.cluster.local" so all other things would address your backendapi in the default namespace as (to use your port numbered example) http://backendapi.default.svc.cluster.local:8080. That fact is the very reason Kubernetes forces all identifiers to be a "dns compatible" name (no underscores or other goofy characters).
Even if you are not running kube-dns, all Service names and ports are also injected into the environment of Pods just like docker would do, so the environment variables ${BACKENDAPI_SERVICE_HOST}:${BACKENDAPI_SERVICE_PORT} would contain the Service's in-cluster IP (even though the env-var is named "host") and the "default" Service port (8080 in your example) if there is only one.
Whether you choose to use the DNS name or the environment-variable-ip is a matter of whether you like having the "readable" names for things in log output or error messages, versus whether you prefer to skip the DNS lookup and use the Service IP address for speed but less legibility. They behave the same.
The whole story lives in the services-networking concept documentation
But the problem
still persists when I change to this
backendapi.default.svc.cluster.local:8080. I even tried using the other
port that it is mapped to internally and my front end web page keeps
saying backendapi.default.svc.cluster.local:32208/api/v1/auth/login
net::ERR_NAME_NOT_RESOLVED. The funny thing is when I curl from my
front end pod it works . But when I'm accessing it using my web
browser it doesnt
Because it is resolvable only within the cluster. (Because only the K8s cluster with kube-dns add-on can translate the domain name backendapi.default.svc.cluster.local:8080 to it's corresponding IP address)
Could this be because i exposed an external ip for the service as well
. The external ip works though
No, It is because domain name backendapi.default.svc.cluster.local is resolvable only within the cluster, not from a random browser in a random machine.
Solution
What you did is one of the solutions, exposing an external ip for the service. If you don't want the IP to be used, you can Create an ingress (and use an ingress controller in your cluster) and expose your Microservice. Since you are on GCP, you can make use of their API gateway rather than exposing a cryptic IP address.
Note: Remember to add the authentication/Authorization to lock down your microservice as it's getting exposed to the user.
Another Solution
Proxy all the backend calls through the server which serves your web app (nginx/nodejs etc)
Advantage of this approach is, you will avoid all the Same Origin Policy/CORS headaches, your microservice (express) authentication details will be abstracted out from user's browser. (This is not necessarily an advantage).
Disadvantage of this approach is, your backend microservice will have a tight coupling with front end (or vice-versa depending on how you look at it), This will make the scaling of backend dependent on front end. Your Backend is not exposed. So, if you have another consumer (let's just say an android app) it will not be able to access your service.
Hybrid Solution
Proxy all the backend calls through the server which serves your web app (nginx/nodejs etc) so that your APIs will inherit your webapps domain. And still expose the backend service (as and when required) so that other consumers (if any, in future) can make use of it.
Kind of similar question: https://stackoverflow.com/a/47043871/6785908