Let us say I've two zones zone1 and zone2, having 2 apps deployed in each zone. Let us say App1 is a client which fetches information from App2, App1 connects to App2 using k8s service, Now how can I configure app1 of zone1 to connect to app2 of zone1(preferably, if app2 of zone1 is loaded or down connect to app2 of zon2).
Though this can be achieved by application layer using zuul and ribbon with headless service, I want to move this to infra layer. Is there any possibility to do in K8s.
I see IPVS supports Locality-Based Least Connection algorithm, but not sure k8s supports this algorithm, I see supported algos are rr, wrr, lc, sed. but no documentation regarding support for lblc. if lblc is supported is this better solution to prefer same node/pod in dc/pod in zone.
NOTE: This is solution is purely for on-prem k8s cluster.
I will answer only to part of your question, have no experience with "best practices" configuration in this area.
But what I want to share with you - is that kubernetes definitely supports Locality-Based Least Connection algorithm.
You can find this in the source code:
LocalityBasedLeastConnection IPVSSchedulerMethod = "lblc"
// LocalityBasedLeastConnectionWithReplication with Replication assigns jobs destined for the same IP address to the
// least-connection node in the server set for the IP address. If all the node in the server set are overloaded,
// it picks up a node with fewer jobs in the cluster and adds it to the sever set for the target.
// If the server set has not been modified for the specified time, the most loaded node is removed from the server set,
// in order to avoid high degree of replication.
You can find info on how to enable IPVS here: https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
P.S. Above article doesnt contains no info about lblc but as per source code - k8s supports it.
Related
My question is similar to the following question in this link: See Inbound Outbound communication of pods
I a running a Kubernetes cluster (with kubeadm and CRI-O engine) on EC2 instances (1 master, 1-2 workers, a very simple application- such as WordPress + MySQL DB).
What I'm looking for is a topology tool to map all pods' communication with other pods in the cluster, with output that I can parse later for further use.
My goal is to generate a JSON/CSV file (or any other type that I can parse later) with the following information:
<Pod i name, inbound communications list, protocol, port>, and the inbound communications list will include <Pod j name, protocol, port> for each pod in communication with the specific pod (i).
I also would like the tool to be not-Web-UI based, but an API/CLI based tool.
So far I came across WeaveScope, which can be scraped for the information I'm interested in (as mentioned in the question I referred to at the beginning). But this is a web-UI based tool and therefore does not fit for my purposes.
Thanks in advance!
We have a gRPC application deployed in a cluster (v 1.17.6) with Istio (v 1.6.2) setup. The cluster has istio-ingressgateway setup as the edge LB, with SSL termination. The istio-ingressgateway is fronted by an AWS ELB (classic LB) in passthrough mode. This setup is fully functional and the traffic flows as intended, in general. So the setup looks like:
ELB => istio-ingressgateway => virtual service => app service => [(envoy)pods]
We are running load tests on this setup using GHZ (ghz.sh), running external to the application cluster. From the tests we’ve run, we have observed that each of the app container seems to get about 300 RPS routed to it, no matter the configuration of the GHZ test. For reference, we have tried various combos of --concurrency and --connection settings for the tests. This ~300 RPS is lower than what we expect from the app and, hence, requires a lot more PODs to provide the required throughput.
We are really interested in understanding the details of the physical connection (gRPC/HTTP2) setup in this case, all the way from the ELB to the app/envoy and the details of the load balancing being done. Of particular interest is the the case when the same client, GHZ e.g., opens up multiple connections (specified via the --connection option). We have looked at Kiali and it doesn’t give us the appropriate visibility.
Questions:
How can we get visibility into the physical connections being setup from the ingress gateway to the pod/proxy?
How is the “per request gRPC” load balancing happening?
What options might exist to optimize the various components involved in this setup?
Thanks.
1.How can we get visibility into the physical connections being setup from the ingress gateway to the pod/proxy?
If Kiali doesn't show what exactly you need, maybe you could try with Jaeger?
Jaeger is an open source end to end distributed tracing system, allowing users to monitor and troubleshoot transactions in complex distributed systems.
There is istio documentation about Jaeger.
Additionally Prometheus and Grafana might be helpful here, take a look here.
2.How is the “per request gRPC” load balancing happening?
As mentioned here
By default, the Envoy proxies distribute traffic across each service’s load balancing pool using a round-robin model, where requests are sent to each pool member in turn, returning to the top of the pool once each service instance has received a request.
If you wan't to change the default round-robin model you can use Destination Rule for that. Destination rules let you customize Envoy’s traffic policies when calling the entire destination service or a particular service subset, such as your preferred load balancing model, TLS security mode, or circuit breaker settings.
There is istio documentation about that.
More about load balancing in envoy here.
3.What options might exist to optimize the various components involved in this setup?
I'm not sure if there is anything to optimize in istio components, maybe some custom configuration in Destination Rule?
Additional Resources:
itnext.io
medium.com
programmaticponderings.com
We have two IBM kubernetes cluster whenever issue happens in one cluster we need to failover to DR. Can anyone tell me how to do that automatically ? both cluster present in two different zones Montreal & torento. Also we have IBM Cloud internet service.
You could use the CIS service Global Load Balancer offering to set up a globally load-balanced and health-checked URL for your applications. You'd create a GLB for domain app.mydomain.com/app_path for example and then back it with the VIPs for your cluster ALBs in the same origin pool. Configure a health check at the GLB so traffic will be sent to the available endpoints that are healthy.
CIS GLB docs are covered at https://cloud.ibm.com/docs/cis?topic=cis-global-load-balancer-glb-concepts
I have setup an on-premises Kubernetes cluster, and I want to be ensure that my services that are not in Kubernetes, but exist on a separate class B are able to consume those services that have migrated to Kubernetes. There's a number of ways of doing this by all accounts and I'm looking for the simplest one.
Ingress + controller seems to be the one favoured - and it's interesting because of the virtual hosts and HAProxy implementation. But where I'm getting confused is how to set up the Kubernetes service:
We've not a great deal of choice - ClusterIP won't be sufficient to expose it to the outside, or NodePort. LoadBalancer seems to be a simpler, cut down way of switching between network zones - and although there are OnPrem solutions (metalLB), seems to be far geared towards cloud solutions.
But if I stick with NodePort, then my entry into the network is going to be on a non-standard port number, and I would prefer it to be over standard port; particuarly if running a percentage of traffic for that service over non-kube, and the rest over kubernetes (for testing purposes, I'd like to monitor the traffic over a period of time before I bite the bullet and move 100% of traffic for the given microservice to kubernetes). In that case it would be better those services would be available across the same port (almost always 80 because they're standard REST micro-services). More than that, if I have to re-create the service for whatever reason, I'm pretty sure the port will change, and then all traffic will not be able to enter the Kubernetes cluster and that's a frightening proposition.
What are the suggested ways of handling communication between existing on-prem and Kubernetes cluster (also on prem, different IP/subnet)?
Is there anyway to get traffic coming in without changing the network parameters (class B's the respective networks are on), and not being forced to use NodePort?
NodePort service type may be good at stage or dev environments. But i recommend you to go with LoadBalancer type service (Nginx ingress controller is one). The advantage for this over other service types are
You can use standard port (Rather random Nodeport generated by your kubernetes).
Your service is load balanced. (Load balancing will be taken care by ingress controller).
Fixed port (it will not change unless you modify something in ingress object).
Long time I did not come here and I hope you're fine :)
So for now, i have the pleasure of working with kubernetes ! So let's start ! :)
[THE EXISTING]
I have an operationnal kubernetes cluster with which I work every day.it consists of several applications, one of which is of particular interest to us, which is the web management interface.
I currently own one master and four nodes in my cluster.
For my web application, pod contain 3 containers : web / mongo /filebeat, and for technical reasons, we decided to assign 5 users max for each web pod.
[WHAT I WANT]
I want to deploy a web pod on each nodes (web0,web1,web2,web3), what I can already do, and that each session (1 session = 1 user) is distributed as follows:
For now, all HTTP requests are processed by web0.
[QUESTIONS]
Am I forced to go through an external loadbalancer (haproxy)?
Can I use an internal loadbalancer, configuring a service?
Does anyone have experience on the implementation described above?
I thank in advance those who can help me in this process :)
This generally depends how and where you've deployed your Kubernetes infrastructure, but you can do this natively with a few options.
Firstly, you'll need to scale your web deployment. This is very simple to do:
kubectl scale --current-replicas=2 --replicas=3 deployment/web
If you're deployed into a cloud provider (such as AWS using kops, or GKE) you can use a service. Just specify the type as LoadBalancer. Services will spread the sessions for your users.
Another option is to use an Ingress. In order to do this, you'll need to use an Ingress Controller, such as the nginx-ingress-controller which is the most featureful and widely deployed.
Both of these options will automatically loadbalance your incoming application sessions, but they may not necessarily do it in the order you've described in your image, it'll be random across the available web deployments