We have a Kubernetes cluster hosting a multitude of applications. It contains an Ingress controller and a front FortiWeb load-balancer.
Kubernetes Diagram
The Fortiweb terminated TLS connections and forwards the traffic to the Kubernetes Ingress controller.
Now, a new requirement has arrived, where we need to configure mTLS for a high-security web application. I have seen the tutorials on how to configure this on Kubernetes ingress controllers, but what I don't understand - is how does this work with the front load-balancer (which currently does TLS offloading and some inspection).
Do we need to configure both: the ingress and the load-balancer? Or only one? And which one?
Related
In GKE Ingress documentation
it states that:
When you create an Ingress object, the GKE Ingress controller creates a Google Cloud HTTP(S) Load Balancer and configures it according to the information in the Ingress and its associated Services.
To me it seems that I can not have multiple ingress resources with single GCP ingress controller. Instead, GKE creates a new ingress controller for every ingress resource.
Is this really so, or is it possible to have multiple ingress resources with a single ingress controller in GKE?
I would like to have one GCP LoadBalancer as ingress controller with static IP and DNS configured, and then have multiple applications running in cluster, each application registering its own ingress resource with application specific host and/or path specifications.
Please note that I'm very new to GKE, GCP and Kubernetes in general, so it might be that I have misunderstood something.
I think the question you're actually asking is slightly different than what you have written. You want to know if multiple Ingress resources can be linked to a single GCP Load Balancer, not GKE Ingress controller. Based on the concept of a controller, there is only one GKE Ingress controller in a cluster, which is responsible for fulfilling multiple resources and provisioning multiple load balancers.
So, to answer the question directly (because I've been searching for a straight answer for a long time!):
Combining multiple Ingress resources into a single Google Cloud load
balancer is not supported.
Source: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Sad.
However, using the nginx-ingress controller is one way to at least minimize the number of external (GCP) load balancers provisioned (it only provisions a single TCP load balancer), but since the load balancer is for TCP traffic, it cannot terminate SSL, or apply Firewall rules for you (Cloud Armor cannot be used, for instance).
The only way I know of to have a single HTTPS load-balancer in GCP terminate SSL and route traffic to multiple services in GKE is to combine the ingresses into a single resource with all paths and certificates defined in one place.
(If anybody figures out a way to do it with multiple separate ingress resources, I'd love to hear it!)
Yes it is possible to have the single ingress controller for multiple ingress resources.
You can create multiple ingress resources as per path requirement and all will be managed by single ingress controller.
There are multiple ingress controller options also available you can use Nginx also that will create one LB and manage the paths.
Inside Kubernetes if you are creating a service with type LoadBalancer it will create the new LB resource in GCP so make sure your microservice type is ClusterIP and your all traffic goes inside K8s cluster via ingress path.
When you setup the ingress controller it will create one service with type LoadBalancer you can can use that IP in DNS servers to forward the subdomain and path to K8s cluster.
I'm trying to get mTLS between two applications in two kubernetes clusters without the way Istio does it (with its ingress gateway), and I was wondering if the following would work (for Istio, for Likerd, for Consul...).
Let's say we have a k8s cluster A with an app A.A. and a cluster B with an app B.B. and I want them to communicate with mTLS.
Cluster A has letsEncrypt cert for nginx ingress controller, and a mesh (whatever) for its application.
Cluster B has self signed cert from our root CA.
Cluster A and B service meshes have different certificates signed by our root CA.
Traffic goes from the internet to Cluster A ingress controller (HTTPS), from there to app A.A.
After traffic gets to app A.A., this app wants to talk to app B.B.
Apps A.A. and B.B. have endpoints exposed via ingress (using their ingress controllers).
The TLS certificates are ended in the endpoints and are wildcards.
Do you think the mTLS will work in this situation?
Basically this blog from portshift answer your question.
The answer is depends on how your clusters are built, because
Istio offers few options to deploy service mesh in multiple kubernetes clusters, more about it here.
So, if you have Single Mesh deployment
You can deploy a single service mesh (control-plane) over a fully connected multi-cluster network, and all workloads can reach each other directly without an Istio gateway, regardless of the cluster on which they are running.
BUT
If you have Multi Mesh Deployment
With a multi-mesh deployment you have a greater degree of isolation and availability, but it increases the set-up complexity. Meshes that are otherwise independent are loosely coupled together using ServiceEntries, Ingress Gateway and use a common root CA as a base for secure communication. From a networking standpoint, the only requirement is that the ingress gateways be reachable from one another. Every service in a given mesh that needs to be accessed a service in a different mesh requires a ServiceEntry configuration in the remote mesh.
In multi-mesh deployments security can become complicated as the environment grows and diversifies. There are security challenges in authenticating and authorizing services between the clusters. The local Mixer (services policies and telemetries) needs to be updated with the attributes of the services in the neighbouring clusters. Otherwise, it will not be able to authorize these services when they reaching its cluster. To achieve this, each Mixer needs to be aware of the workload identities, and their attributes, in neighbouring clusters Each Citadel needs to be updated with the certificates of neighbouring clusters, to allow mTLS connections between clusters.
Federation of granular workloads identities (mTLS certificates) and service attributes across multi-mesh control-planes can be done in the following ways:
Kubernetes Ingress: exposing HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can terminate SSL / TLS, and offer name based virtual hosting. Yet, it requires an Ingress controller for fulfilling the Ingress rules
Service-mesh gateway: The Istio service mesh offers a different configuration model, Istio Gateway. A gateway allows Istio features such as monitoring and route rules to be applied to traffic entering the cluster. An ingress gateway describes a load balancer operating at the edge of the mesh that receives incoming HTTP/TCP connections. It configures exposed ports, protocols, etc. Traffic routing for ingress traffic is configured instead using Istio routing rules, exactly the same way as for internal service requests.
Do you think the mTLS will work in this situation?
Based on above informations
If you have Single Mesh Deployment
It should be possible without any problems.
If you have Multi Mesh Deployment
It should work, but since you don't want to use istio gateway then the only option is kubernetes ingress.
I hope it answer your question. Let me know if you have any more questions.
You want Consul's Mesh Gateways. They mTLS service-to-service connectivity between federated Consul clusters deployed in different data centers, clusters, or runtime environments.
I have a kubernetes cluster created. According to security policies in the company, I need to have first an application gateway WAF in front that hits the cluster (which has a public IP). And as an ingress controller for this cluster I need to configure a Nginx ingress controller (also has a public IP). How can I connect or point the waf to the ingress controller? Is this possible to be done?
Thanks!
Native support for Nginx ingress controller is with a load balancer and not with app gateway. One possible approach is to create a nginx ingress controller loadbalancer as private using this link docs.
Now add this private Ip of load balancer as the backend pool of app gateway and now your app gateway should start serving the traffic from aks cluster.
The App gateway ingress controller as suggested by another comment is GA now but still is buggy. It takes time to update the backend pools on deploying new pods.
GKE Ingress: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Nginx Ingress: https://kubernetes.github.io/ingress-nginx/
Why GKE Ingress
GKE Ingress can be used along with Google's managed SSL certificates. These certificates are deployed in edge servers of load balancer which results in very low TTFB (time to first byte)
What's wrong about GKE Ingress
The HTTP/domain routing is done in the load balancer using 'forward rules' which is very pricy. Costs around $7.2 per rule. Each domain requires one rule.
Why Nginx Ingress
Nginx Ingress also creates (TCP/UP) load balancer where we can specify routing of HTTP/domain using ingress controller. Since the routing is done inside the cluster there are no additional costs on adding domains into the rules
What's wrong about Nginx Ingress
To enable SSL, we can use cert-manager. But as I mentioned above, Google's managed certificate deploy certificates in edge servers which results in very low latency
My Question
Is it possible to use both of them together? So that HTTPS requests first hit GKE ingress which will terminate SSL and route the traffic to Nginx ingress which will route it to corresponding pods
Is not possible to point an Ingress to another Ingress. Furthermore and in your particular case, is also not possible to point a GCE ingress class to Nginx since it relies in an HTTP(S) Load Balancer, which can only have GCE instances/instances groups (basically the node pools in GKE), or GCS buckets as backends.
If you were to deploy an Nginx ingress using GKE, it will spin up a Network Load Balancer which is not a valid backend for the HTTP(S) Load Balancer.
So is neither possible via Ingress nor GCP infrastructure features. However, if you need the GCE ingress class to be hit first, and then, manage further routing with Nginx, you might want to consider having Nginx as a Kubernetes Service/Deployment to manage the incoming traffic once is within the cluster network.
You can create a ClusterIP service for internally accessing your Nginx deployment and from there, using cluster-local hostnames to redirect to other services/applications within the cluster.
I just started using Istio and securing service to service communication and have two questions:
When using nginx ingress, will Istio secure the data from the ingress controller to the service with TLS?
Is it possible to secure with TLS all the way to the pod?
With "Envoy" deployed as sidecar container to both i.e. (a) NGINX POD and (b) Application POD, istio will ensure that both the services communicate to each-other over TLS.
Infact that's the whole idea behind using Istio i.e. to secure all the communication way till the POD using ENVOY side-car. Envoy will intercept all the traffic going in/out of the POD and perform TLS communication with the peer Envoy counterpart.
All this is done in a transparent manner i.e transparent to the application container. The responsibility to perform TLS layer jobs ex. handshake, encryption/decryption, peer discovery etc. are all offloaded to the envoy sidecar.