Moving from VMs to Kubernetes.
We are running our services on multiple VMs. Services are running on multiple VMs and have VIP in front of them. Clients will be accessing VIP and VIP will be routing traffic to services. Here, we use SSL cert for VIP and VIP to VM also using HTTPS.
Here the service will be deployed into VM with a JKS file. This JKS file will have a cert for exposing HTTPS and also to communicate with SSL enabled database.
How to achieve the same thing in Kubernetes cluster? Need HTTPS for VIP and services and also for communication to SSL enabled database from service.
Depends on the platform where you running Kubernetes (on-premises, AWS, GKE, GCE etc.) you have several ways to do it, but I will describe a solution which will work on all platforms - Ingress with HTTPS termination on it.
So, in Kubernetes you can provide access to your application inside a cluster using Ingress object. It can provide load balancing, HTTPS termination, routing by path etc. In most of the cases, you can use Ingress controller based on Nginx. Also, it providing TCP load balancing and SSL Passthrough if you need it.
For providing routing from users to your services, you need:
Deploy your application as a combination of Pods and Service for them.
Deploy Ingress controller, which will manage your Ingress objects.
Create a secret for your certificate.
Create an Ingress object with will point to your service with TLS settings for ask Ingress to use your secret with your certificate, like that:
spec:
tls:
hosts:
- foo.bar.com
secretName: foo-secret
Now, when you call the foo.bar.com address, Ingress with using FQDN-based routing and provide HTTPS connection between your client and pods in a cluster using a service object, which knows where exactly your pod is. You can read how it works here and here.
What about encrypted communication between your services inside a cluster - you can use the same scheme with secrets for providing SSL keys to all your services and setup Service to use HTTPS endpoint of an application instead of HTTP. Technically it is same as using https upstream in installations without Kubernetes, but all configuration for Nginx will be provided automatically based on your Service and Ingress objects configuration.
Related
I'm trying to setup a reverse proxy to work with Kubernetes. I currently have an ingress load-balancer using Metallb and Contour with Envoy.
I also have a working certificate issuer with Let's Encrypt and cert-manager allowing services and deployments to get certificates for HTTPS.
My problem is trying to get other websites and servers not run in Kubernetes but are in our DNS range to have HTTPS certificates and I feel like I am missing something.
My IP for my load-balancer is 10.64.1.35 while the website I am trying to get a certificate for is 10.64.0.145.
Thank you if you could offer any help!
I think that will never work. Something needs to request a certificate, in kubernetes this usually is the presence of a Resource. The cert-manager listens to the creation of that resource, and requests a certificate from let's encrypt.
Then that certificate must be configured in some loadbalancer and the loadbalancer must reload its configuration (That's what Metallb does).
When you have applications running elsewhere outside of this setup, those applications will never have certificates.
If you really want to have that Metallb loadbalancer request and attach the certificates, you'll need to create a resource in kubernetes and proxy all the traffic for that application through kubernetes.
myapp.com -> metallb -> kubernetes -> VPS
However, I think the better way for you is to setup let's encrypt on the server where you need it. That way you prevent 2 additional network hops, and resources on the metallb and kubernetes server(s).
I want to achieve TLS mutual auth between my different services running in a kubernetes cluster and I have found that Istio is a good solution to achieve this without making any changes in code.
I am trying to use Istio sidecar injection to do TLS mutual auth between services running inside the cluster.
Outside traffic enters the mesh through nginx ingress controller. We want to keep using it instead of the Istio ingress controller(we want to make as little changes as possible).
The services are able to communicate with each other properly when the Istio Sidecar injection is disabled. But as soon as I enable the sidecar in the application's namespace, the app is not longer able to serve requests(I am guessing the incoming requests are dropped by the envoy sidecar proxy).
What I want to do:
Enable istio sidecar proxy injection on namespace-2(nginx ingress controller, service 1 and service 2) so that all services communicate with each other through TLS mutual auth.
What I don't want to do:
Enable istio sidecar proxy injection on the nginx ingress controller(I don't want to make any changes in it as it is serving as frontend for multiple other workloads).
I have been trying to make it work since a couple of weeks with no luck. Any help from the community will be greatly appreciated.
my goal is to atleast enable TLS mutual auth between service-1 and service-2
AFAIK if you have enabled injection in namespace-2 then services here already have mTLS enabled. It's enabled by default since istio 1.5 version. There are related docs about this.
Automatic mutual TLS is now enabled by default. Traffic between sidecars is automatically configured as mutual TLS. You can disable this explicitly if you worry about the encryption overhead by adding the option -- set values.global.mtls.auto=false during install. For more details, refer to automatic mutual TLS.
Take a look here for more information about how mtls between services works.
Mutual TLS in Istio
Istio offers mutual TLS as a solution for service-to-service authentication.
Istio uses the sidecar pattern, meaning that each application container has a sidecar Envoy proxy container running beside it in the same pod.
When a service receives or sends network traffic, the traffic always
goes through the Envoy proxies first.
When mTLS is enabled between two services, the client side and server side Envoy proxies verify each other’s identities before sending requests.
If the verification is successful, then the client-side proxy encrypts the traffic, and sends it to the server-side proxy.
The server-side proxy decrypts the traffic and forwards it locally to the actual destination service.
NGINX
But the problem is, the traffic from outside the mesh is getting terminated at the ingress resource. The nginx reverse proxy in namespace-2 does not see the incoming calls.
I see there is similar issue on github about that, worth to try with this.
Answer provided by #stono.
Hey,
This is not an istio issue, getting nginx to work with istio is a little bit difficult. The issue is because fundamentally nginx is making an outbound request to an ip that is has resolved from your hostname foo-bar. This won't work as envoy doesn't know what cluster ip belongs to, so it fails.
I'd suggest using the ingress-nginx kubernetes project and in turn using the following value in your Ingress configuration:
annotations:
nginx.ingress.kubernetes.io/service-upstream: "true"
What this does is ensure that nginx doesn't resolve the upstream address to an ip, and maintains the correct Host header which the sidecar uses in order to route to your destination.
I recommend using this project because I use it, with Istio, with a 240 odd service deployment.
If you're not using ingress-nginx, I think you can set proxy_ssl_server_name on; or another thing you could try is forcefully setting the Host header on the outbound request to the internal fqdn of the service so:
proxy_set_header Host foo-bar;
Hope this helps but as I say, it's an nginx configuration rather than an istio problem.
GKE Ingress: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Nginx Ingress: https://kubernetes.github.io/ingress-nginx/
Why GKE Ingress
GKE Ingress can be used along with Google's managed SSL certificates. These certificates are deployed in edge servers of load balancer which results in very low TTFB (time to first byte)
What's wrong about GKE Ingress
The HTTP/domain routing is done in the load balancer using 'forward rules' which is very pricy. Costs around $7.2 per rule. Each domain requires one rule.
Why Nginx Ingress
Nginx Ingress also creates (TCP/UP) load balancer where we can specify routing of HTTP/domain using ingress controller. Since the routing is done inside the cluster there are no additional costs on adding domains into the rules
What's wrong about Nginx Ingress
To enable SSL, we can use cert-manager. But as I mentioned above, Google's managed certificate deploy certificates in edge servers which results in very low latency
My Question
Is it possible to use both of them together? So that HTTPS requests first hit GKE ingress which will terminate SSL and route the traffic to Nginx ingress which will route it to corresponding pods
Is not possible to point an Ingress to another Ingress. Furthermore and in your particular case, is also not possible to point a GCE ingress class to Nginx since it relies in an HTTP(S) Load Balancer, which can only have GCE instances/instances groups (basically the node pools in GKE), or GCS buckets as backends.
If you were to deploy an Nginx ingress using GKE, it will spin up a Network Load Balancer which is not a valid backend for the HTTP(S) Load Balancer.
So is neither possible via Ingress nor GCP infrastructure features. However, if you need the GCE ingress class to be hit first, and then, manage further routing with Nginx, you might want to consider having Nginx as a Kubernetes Service/Deployment to manage the incoming traffic once is within the cluster network.
You can create a ClusterIP service for internally accessing your Nginx deployment and from there, using cluster-local hostnames to redirect to other services/applications within the cluster.
I would like to add multiple certificates for the same IP with different sub domain. I've created cluster and added ingress. i have HTTPS applications installed as well. I succeeded to add certificated to two different hosts (using the same ingress but 2 public IPs).
How can i write in ingress to use cert X for aks-hello-world and in cert Y for ingress-demo? (see attachment)
Is there something i should do in azure as well?
Ingress example
You can deploy the Kubernetes nginx ingress controller and forward the traffic from one (public accessible) load balancer to the exposed node port service of the ingress service (type: LoadBalancer). This forwarding is done via TCP (no TLS happening here). You need to point the multiple DNS entries to the IP of the load balancer.
The ingress controller is capable of SNI and handles the forwarding from there on.
This approach however requires that your TLS certificates are added to Kubernetes as secrets (for example via Cert Manager).
For configuration see the docs.
I have Istio Ingress which is working with traffic going in to microservices and inbetween microservices is being encrypted within ISTIO domain. But i dont want to expose ISTIO ingress to public.
So tried deploying NGINX or HAPROXY ingress (with https certs) and point them to ISTIO ingress and everything is working great.
My only worry now is that traffic between NGINX INGRESS (https term) > ISTIO INGRESS is not encrypted.
What is the usual way on Istio to get full encryption of traffic but with NGINX/HAPROXY ingress.
I guess one way is to HAPROXY tcp mode to ISTIO ingress with certs on Istio ingress. Haven't tried it but it should work. Wild idea is running NGINX ingress within ISTIO mash but then i would loose some Istio Ingress capabilities.
What is the recommended way or any suggestion. How is usualy Istio being exposed on some real Prod env example.
Since i dont use cloud loadbalancer on voyager instances but expose Voyager/Haproxy on Host-Port
I collocated Voyager(HostPort) and Istio(HostPort) via DeamonSet/node-selector(and taints) on same machines called frontend. Then just pointed Voyager to loadbalance the loopback/localhost with port of Istio HostPort I specified.
backendRule:
- 'server local-istio localhost:30280'
This way no unenctypted traffic is traversing the network between Voyager/Haproxy and Istio Ingress since they communicate now on same Host. I have 2 frontend nodes witch are beeing loadbalanced so i have redundancy. But its kind of improvisation and breaking kubernetes logic. On the other hand it works great.
Other solution was to use selfsigned certificates on Istio, than just point Voyager/Haproxy to Istio instances. But this requires multiple terminations since Voyager is also terminating Https. Advanteg of this is that you can leave Voyager and Istio instances to Kubernetes to distribute. No need to bind them to specific machines.