I have a typhoon kubernetes cluster on aws with an nginx ingress controller installed.
If I add an test ingress object it looks like this:
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default foo <none> * 10.0.8.180 80 143m
Question: Where does my ingress controller get that address(10.0.8.180) from?
There is no (loadbalancer) service on the system with that address.
(Because it is a private address external-dns does not work correctly.)
In order to answer your first question:
Where does a kubernetes ingress gets its IP address from?
we have to dig a bit into the code and its behavior.
It all starts here with publish-service flag:
publishSvc = flags.String("publish-service", "",
`Service fronting the Ingress controller
Takes the form "namespace/name". When used together with update-status, the
controller mirrors the address of this service's endpoints to the load-balancer
status of all Ingress objects it satisfies.`)
The flag variable (publishSvc) is later assigned to other variable (PublishService):
PublishService: *publishSvc,
Later in the code you can find that if this flag is set, this code is being run:
if s.PublishService != "" {
return statusAddressFromService(s.PublishService, s.Client)
}
statusAddressFromService function as an argument takes value of publish-service flag and queries kubernetes for service with this name, and returns IP address of related service.
If this flag is not set it will query the k8s for IP address of a node where nginx ingress pod is running. This is the bahaviour you are experiencing and it makes me think that you didn't set this flag. This also answers your second question:
Why does it take the address of the node instead of the NLB?
Meanwhile you can find all yaml for all sort of platforms in k8s nginx ingress documentation.
Lets have a look at AWS ingress yaml.
Notice here that publish-service has a value called ingress-nginx/ingress-nginx-controller (<namespace>/<service>).
and this is what you also want to do.
TLDR: All you have to do is to create a LoadBalancer Service and set the publish-service to <namespace_name>/<service_name>
Related
I am following this guide to expose a service running on my bare metal k8s cluster to the world.
The guide suggests using metallb for giving external access. The problem is, during the setup process of metallb, I am asked to give a range of available IP addresses.
The hosting provider I am using is very basic, and all I have is the IP address of the Linux instance that is running my K8s node. So my question is, how can I provision an IP address for assigning to my application? Is this possible with a single IP?
Alternatively I'd love get this done with a NodePort, but I need to support HTTPS traffic and I am not sure its possible if I go that way.
Specify a single IP using CIDR notation. your-ip/32 (192.168.10.0/32 for example)
Single IP Address Load Balancer
Using a single IP address is possible. In this case you don't need the speaker pods that announce the external IP addresses and thus no pod security labels. So, if you install using helm, prepare a metallb-values-helm.yaml file:
speaker:
enabled: false
Then install metallb:
kubectl create namespace metallb-system
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --namespace metallb-system -f metallb-values-helm.yaml
Now prepare a configuration of the public IP address in a metallb-config-ipaddress.yaml file:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: metallb-ip
namespace: metallb-system
spec:
addresses:
# The IP address of the kubernetes master node
- 192.168.178.10/32
And apply:
kubectl apply -f metallb-config-ipaddress.yaml
Multiple Services Sharing the Same IP Address
This should already work for a single service. However, if you want to apply multiple services on different ports of the same IP address, you need to provide an annotation in every service manifest, as described here. A service manifest will look like:
apiVersion: v1
kind: Service
metadata:
name: cool-service
namespace: default
annotations:
metallb.universe.tf/allow-shared-ip: "key-to-share-192.168.178.10"
...
The string "key-to-share-192.168.178.10" is arbitrary, but must be equal for all services. If there is really just a single IP address in your pool (as specified above), you don't have to specify it as loadBalancerIP: 192.168.178.10. This would be only required if you had multiple IP addresses and wanted to select one. So that's all.
What's next?
You can also use nginx-ingress as ingress controller behind your metallb load balancer, which is still required to expose the nginx-ingress service. Then you can e. g. separate your services via subdomains (pointing to the same IP address) like
service1.domain.com
service2.domain.com
I'm unable to wrap my head around the concepts of interconnectivity among DNS, Ingress controller, MetalLb and Kubeproxy.
I know what these resources/tools/services are for and get the concepts of them individually but unable to form a picture of them working in tandem.
For example, in a bare metal setup, a client accesses my site - https://mytestsite.com, still having doubts , how effectively it lands to the right pod and where the above mentioned services/resources/tools comes into picture & at what stage ?
Ex. How DNS talks to my MetalLB, if the client accesses my MetalLB hosting my application and how LB inturn speaks to IngressController and finally where does Kube-proxy comes into play here.
I went thru the K8s official documentation and few others as well but still kind of stumped here. Following article is really good but I'm unable to stitch the pieces together.
https://www.disasterproject.com/kubernetes-with-external-dns/
Kindly redirect me to the correct forum, if it is not the right place, thanks.
The ingress-controller creates a service of type LoadBalancer that serves as the entry point into the cluster. In a public cloud environment, a loadbalancer like ELB on AWS would create the counter part and set the externalIP of that service to it's ip. It is like a service of type NodePort but it also has an ExternalIP, which corresponds to the actual ip of the counterpart, a load balancer like ELB on aws.
In a bare metal environment, no external load balancer will be created, so the external ip would stay in <Pending> state forever. Here for example the service of the istio ingress controller:
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 192.12.129.119 <Pending> [...],80:32123/TCP,443:30994/TCP,[...]
In that state you would need to call http://<node-ip>:32123 to reach the http port 80 of the ingress controller service, which would be then forwarded to your pod (more on that in a bit).
When you're using metallb, it will update the service with an external ip so you can call http://<ip> instead. Metallb will also announce that ip, e.g. via BGP, so other know where to send traffic to, when someone would call the ip.
I havn't used external DNS and only scanned the article but I guess that you can use that to also have a dns record to be created so someone can call your service by it's domain, not only by it's ip. So you can call http://example.com instead.
This is basically why you run metallb and how it interacts with your ingress controller. The ingress controller creates an entry point into the cluster and metallb configures it and attracts traffic.
Until now the call to http://example.com can reach your cluster, but it needs to also reach the actual application, running in a pod inside the cluster. That's kube-proxy's job.
You read a lot about service of different types and all this kind of stuff, but in the end it all boils down to iptables rules. kube-proxy will create a bunch of those rules, that form a chain.
SSH into any kubernetes worker, run iptables-save | less command and search for the external ip configured on your ingress-controller's service by metallb. You'll find a chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.
In the end the whole chain would look something like this:
http://example.com
-> http://<some-ip> (domain translated to ip)
-> http://<node-ip>:<node-port> (ingress-controller service)
---
-> http://<cluster-internal-ip>:<some-port> (service of your application)
-> http://<other-cluster-internal-ip>:<some-port> (ip of one of n pods)
where the --- line shows the switch from cluster external to cluster internal traffic. The cluster-internal-ip will be from the configured service-cdir and the other-cluster-internal-ip will be from the configured pod-cidr.
Note that there are different ways to configure cluster internal traffic routing, how to run kube-proxy and some parts might even be a bit simplified, but this should give you a good enough understanding of the overall concept.
Also see this answer on the question 'What is a Kubernetes LoadBalancer On-Prem', that might provide additional input.
I have two-zone, each has to master node. Today I created a simple ingress-nginx controller and successfully pointed a DNS test.example.com to one of the public IP in zone-1.
But now I want to create another nginx-controller in zone-2 and point test.example.com to the public IP address of that zone with cloud DNS.
What approach should I take? Is there any best practice?
Your question is unclear and needs to be improved with any minimal reproducible example. You can find the manual here.
According to your subject, you're using Kubernetes cluster in bare-metal, however you mentioned using cloud DNS. Where did you get cloud DNS?
If you're using pure bare metal, then consider using MetalLB.
MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.
But this approach has a few other limitations one ought to be aware of, one of them is about Ingress status:
Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS
test-ingress myapp.example.com 80
Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service.
Please see the example below:
Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is )
$ kubectl get node
NAME STATUS ROLES EXTERNAL-IP
host-1 Ready master 203.0.113.1
host-2 Ready node 203.0.113.2
host-3 Ready node 203.0.113.3
one could edit the ingress-nginx Service and add the following field to the object spec
spec:
externalIPs:
- 203.0.113.1
- 203.0.113.2
- 203.0.113.3
which would in turn be reflected on Ingress objects as follows:
$ kubectl get ingress -o wide
NAME HOSTS ADDRESS PORTS
test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80
See more detailed info here
all
I knew well about k8s' nodePort and ClusterIP type in services.
But I am very confused about the Ingress way, because how will a request come into a pod in k8s by this Ingress way?
Suppose K8s master IP is 1.2.3.4, after Ingress setup, and can connect to backend service(e.g, myservice) with a port(e.g, 9000)
Now, How can I visit this myservice:9000 outside? i.e, through 1.2.3.4? As there's no entry port on the 1.2.3.4 machine.
And many docs always said visit this via 'foo.com' configed in the ingress YAML file. But that is really funny, because xxx.com definitely needs DNS, it's not a magic to let you new-invent any xxx.com you like be a real website and can map your xxx.com to your machine!
The key part of the picture is the Ingress Controller. It's an instance of a proxy (could be nginx or haproxy or another ingress type) and runs inside the cluster. It acts as an entrypoint and lets you add more sophisticated routing rules. It reads Ingress Resources that are deployed with apps and which define the routing rules. This allows each app to say what the Ingress Controller needs to do for routing to it.
Because the controller runs inside the cluster, it needs to be exposed to the outside world. You can do this by NodePort but if you're using a cloud provider then it's more common to use LoadBalancer. This gives you an external IP and port that reaches the Ingress controller and you can point DNS entries at that. If you do point DNS at it then you have the option to use routing rules base on DNS (such as using different subdomains for different apps).
The article 'Kubernetes NodePort vs LoadBalancer vs Ingress? When should I use what?' has some good explanations and diagrams - here's the diagram for Ingress:
I'm trying to use Kubernetes to make configurations and deployments explicitly defined and I also like Kubernetes' pod scheduling mechanisms. There are (for now) just 2 apps running on 2 replicas on 3 nodes. But Google's Kubernetes Engine's load balancer is extremely expensive for a small app like ours (at least for the moment) at the same time I'm not willing to change to a single instance hosting solution on a container or deploying the app on Docker swarm etc.
Using node's IP seemed like a hack and I thought that it might expose some security issues inside the cluster. Therefore I configured a Træfik ingress and an ingress controller to overcome Google's expensive flat rate for load balancing but turns out an outward facing ingress spins up a standart load balancer or I'm missing something.
I hope I'm missing something since at this rates ($16 a month) I cannot rationalize using kubernetes from start up for this app.
Is there a way to use GKE without using Google's load balancer?
An Ingress is just a set of rules that tell the cluster how to route to your services, and a Service is another set of rules to reach and load-balance across a set of pods, based on the selector. A service can use 3 different routing types:
ClusterIP - this gives the service an IP that's only available inside the cluster which routes to the pods.
NodePort - this creates a ClusterIP, and then creates an externally reachable port on every single node in the cluster. Traffic to those ports routes to the internal service IP and then to the pods.
LoadBalancer - this creates a ClusterIP, then a NodePort, and then provisions a load balancer from a provider (if available like on GKE). Traffic hits the load balancer, then a port on one of the nodes, then the internal IP, then finally a pod.
These different types of services are not mutually exclusive but actually build on each other, and it explains why anything public must be using a NodePort. Think about it - how else would traffic reach your cluster? A cloud load balancer just directs requests to your nodes and points to one of the NodePort ports. If you don't want a GKE load balancer then you can already skip it and access those ports directly.
The downside is that the ports are limited between 30000-32767. If you need standard HTTP port 80/443 then you can't accomplish this with a Service and instead must specify the port directly in your Deployment. Use the hostPort setting to bind the containers directly to port 80 on the node:
containers:
- name: yourapp
image: yourimage
ports:
- name: http
containerPort: 80
hostPort: 80 ### this will bind to port 80 on the actual node
This might work for you and routes traffic directly to the container without any load-balancing, but if a node has problems or the app stops running on a node then it will be unavailable.
If you still want load-balancing then you can run a DaemonSet (so that it's available on every node) with Nginx (or any other proxy) exposed via hostPort and then that will route to your internal services. An easy way to run this is with the standard nginx-ingress package, but skip creating the LoadBalancer service for it and use the hostPort setting. The Helm chart can be configured for this:
https://github.com/helm/charts/tree/master/stable/nginx-ingress
One option is to completely disable this feature on your GKE cluster. When creating the cluster (on console.cloud.google.com) under Add-ons disable HTTP load balancing. If you are using gcloud you can use gcloud beta container clusters create ... --disable-addons=HttpLoadBalancing.
Alternatively, you can also inhibit the GCP Load Balancer by adding an annotation to your Ingress resources, kubernetes.io/ingress.class=somerandomstring.
For newly created ingresses, you can put this in the yaml document:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: somerandomstring
...
If you want to do that for all of your Ingresses you can use this example snippet (be careful!):
kubectl get ingress --all-namespaces \
-o jsonpath='{range .items[*]}{"kubectl annotate ingress -n "}{.metadata.namespace}{" "}{.metadata.name}{" kubernetes.io/ingress.class=somerandomstring\n"}{end}' \
| sh -x
Now using Ingresses is pretty useful with Kubernetes, so I suggest you check out the nginx ingress controller and after deployment, annotate your Ingresses accordingly.
If you specify the Ingress class as an annotation on the Ingress object
kubernetes.io/ingress.class: traefik
Traefik will pick it up while the Google Load Balancer will ignore it. There is also a bit of Traefik documentation on this part.
You could deploy the nginx ingress controller using NodePort mode (e.g. if using the helm chart set controller.service.type to NodePort) and then load-balance amongst your instances using DNS. Just make sure you have static IPs for the nodes or you could even create a DaemonSet that somehow updates your DNS with each node's IP.
Traefik seems to support a similar configuration (e.g. through serviceType in its helm chart).