My application is running a SSL NodeJS server with mutual authentication.
How do I tell k8s to access the container thought HTTPS?
How do I forward the client SSL certificates to the container?
I tried to setup a Gateway & a Virtual host without success. In every configuration I tried I hit a 503 error.
The Istio sidecar proxy container (when injected) in the pod will automatically handle communicating over HTTPS. The application code can continue to use HTTP, and the Istio sidecar will intercept the request, "upgrading" it to HTTPS. The sidecar proxy in the receiving pod will then handle "downgrading" the request to HTTP for the application container.
Simply put, there is no need to modify any application code. The Istio sidecar proxies requests and responses between Kubernetes pods with TLS/HTTPS.
UPDATE:
If you wish to use HTTPS at the application level, you can tell Istio to exclude certain inbound and outbound ports. To do so, you can add the traffic.sidecar.istio.io/excludeInboundPorts and traffic.sidecar.istio.io/excludeOutboundPorts annotations, respectively, to the Kubernetes deployment YAML.
Example:
...
spec:
selector:
matchLabels:
app: podinfo
template:
metadata:
annotations:
traffic.sidecar.istio.io/includeInboundPorts: "443"
traffic.sidecar.istio.io/excludeInboundPorts: "443"
labels:
...
We are using Kubernetes v1.19.13 hosted on Google Kubernetes Engine. We want to configure an Ingress controller so that the Google HTTP(S) LoadBalancer is configured to allow only TLS 1.2 and 1.3 and these features/ciphers:
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
We would prefer to do this using annotations but most examples we have found uses a ConfigMap or FrontendConfig.
Is this possible to configure this using annotations? If not, what is the recommended way of achieving this?
Note that we want to configure this using Kubernetes and not using the Google Cloud Console.
You won't be able to do this using annotations. You cannot currently create an SSL Policy via annotations. SSL Policies need to be created via gcloud CLI or via the GCP API.
You'll then need to create a FrontendConfig resource which references the policy and then attach it to your ingress resource:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: FRONTENDCONFIG_NAME
spec:
sslPolicy: allowed-ciphers
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
networking.gke.io/v1beta1.FrontendConfig: "FRONTENDCONFIG_NAME"
...
The good news is that you can (re)use the SSL Policy and/or FrontendConfig.
If you want to do everything via the k8s API, you can try using Config Connector and create ComputeSSLPolicy resource.
To configure an Ingress controller to allow only TLS 1.2 and 1.3, you can use ngnix.ingress.kubernetes.io/proxy-ssl-protocols annotation.
nginx.ingress.kubernetes.io/proxy-ssl-protocols: Enables the specified protocols for requests to a proxied HTTPS server.
For Example:
annotations["nginx.ingress.kubernetes.io/proxy-ssl-protocols"] = "TLSv1.2 TLSv1.3"
To configure an Ingress controller to ciphers, you can use ngnix.ingress.kubernetes.io/proxy-ssl-ciphers annotation. nginx.ingress.kubernetes.io/proxy-ssl-ciphers: Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library.
Using ssl_ciphers annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host.
For Example Cipher :
nginx.ingress.kubernetes.io/ssl-ciphers: "ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
Refer SSL-ciphers for more information.
Is there a way to force an SSL upgrade for incoming connections on the ingress load-balancer? Or if that is not possible with, can I disable port :80? I haven't found a good documentation pages that outlines such an option in the YAML file. Thanks a lot in advance!
https://github.com/kubernetes/ingress-gce#frontend-https
You can block HTTP through the annotation kubernetes.io/ingress.allow-http: "false" or redirect HTTP to HTTPS by specifying a custom backend. Unfortunately GCE doesn't handle redirection or rewriting at the L7 layer directly for you, yet. (see https://github.com/kubernetes/ingress-gce#ingress-cannot-redirect-http-to-https)
Update: GCP now handles redirection rules for load balancers, including HTTP to HTTPS. There doesn't appear to be a method to create these through Kubernetes YAML yet.
This was already correctly answered by a comment on the accepted answer. But since the comment is buried I missed it several times.
As of GKE version 1.18.10-gke.600 you can add a k8s frontend config to redirect from http to https.
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: ssl-redirect
spec:
redirectToHttps:
enabled: true
# add below to ingress
# metadata:
# annotations:
# networking.gke.io/v1beta1.FrontendConfig: ssl-redirect
The annotation has changed:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
annotations:
kubernetes.io/ingress.allow-http: "false"
spec:
...
Here is the annotation change PR:
https://github.com/kubernetes/contrib/pull/1462/files
If you are not bound to the GCLB Ingress Controller you could have a look at the Nginx Ingress Controller. This controller is different to the builtin one in multiple ways. First and foremost you need to deploy and manage one by yourself. But if you are willing to do so, you get the benefit of not depending on the GCE LB (20$/month) and getting support for IPv6/websockets.
The documentation states:
By default the controller redirects (301) to HTTPS if TLS is enabled for that ingress . If you want to disable that behaviour globally, you
can use ssl-redirect: "false" in the NGINX config map.
The recently released 0.9.0-beta.3 comes with an additional annotation for explicitly enforcing this redirect:
Force redirect to SSL using the annotation ingress.kubernetes.io/force-ssl-redirect
Google has responded to our requests and is testing HTTP->HTTPS SSL redirection on their load balancers. Their latest answer said it should be in Alpha sometime before the end of January 2020.
Their comment:
Thank you for your patience on this issue. The feature is currently in testing and we expect to enter Alpha phase before the end of January. Our PM team will have an announcement with more details as we get closer to the Alpha launch.
My fingers are crossed that we'll have a straightforward solution to this very common feature in the near future.
UPDATE (April 2020):
HTTP(S) rewrites is now a Generally Available feature. It's still a bit rough around the edges and does not work out-of-the-box with the GCE Ingress Controller unfortunately. But time will tell and hopefully a native solution will appear.
A quick update. Here
Now a FrontEndConfig can be make to configure the ingress. Hopes it helps.
Example:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: 301
You'll need to make sure that your load balancer supports HTTP and HTTPS
Worked on this for a long time. In case anyone isn't clear on the post above. You would rebuild your ingress with annotation -- kubernetes.io/ingress.allow-http: "false” --
Then delete your ingress and redeploy. The annotation will have the ingress only create a LB for 443, instead of both 443 and 80.
Then you do a compute HTTP LB, not one for GKE.
Gui directions:
Create a load balancer and choose HTTP(S) Load Balancing -- Start configuration.
choose - From Internet to my VMs and continue
Choose a name for the LB
leave the backend configuration blank.
Under Host and path rules, select Advanced host and path rules with the action set to
Redirect the client to different host/path.
Leave the Host redirect field blank.
Select Prefix Redirect and leave the Path value blank.
Chose the redirect response code as 308.
Tick the Enable box for HTTPS redirect.
For the Frontend configuration, leave http and port 80, for ip address select the static
IP address being used for your GKE ingress.
Create this LB.
You will now have all http traffic go to this and 308 redirect to your https ingress for GKE. Super simple config setup and works well.
Note: If you just try to delete the port 80 LB that GKE makes (not doing the annotation change and rebuilding the ingress) and then adding the new redirect compute LB it does work, but you will start to see error messages on your Ingress saying error 400 invalid value for field 'resource.ipAddress " " is in use and would result in a conflict, invalid. It is trying to spin up the port 80 LB and can't because you already have an LB on port 80 using the same IP. It does work but the error is annoying and GKE keeps trying to build it (I think).
Thanks to the comment of #Andrej Palicka and according to the page he provided: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect now I have an updated and working solution.
First we need to define a FrontendConfig resource and then we need to tell the Ingress resource to use this FrontendConfig.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-app-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: myapp-prd
networking.gke.io/managed-certificates: managed-cert
kubernetes.io/ingress.class: "gce"
networking.gke.io/v1beta1.FrontendConfig: myapp-frontend-config
spec:
defaultBackend:
service:
name: myapp-app-service
port:
number: 80
---
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: myapp-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: MOVED_PERMANENTLY_DEFAULT
You can disable HTTP on your cluster (note that you'll need to recreate your cluster for this change to be applied on the load balancer) and then set HTTP-to-HTTPS redirect by creating an additional load balancer on the same IP address.
I spend couple of hours on the same question, and ended up doing what I've just described. It works perfectly.
Redirecting to HTTPS in Kubernetes is somewhat complicated. In my experience, you'll probably want to use an ingress controller such as Ambassador or ingress-nginx to control routing to your services, as opposed to having your load balancer route directly to your services.
Assuming you're using an ingress controller, then:
If you're terminating TLS at the external load balancer and the LB is running in L7 mode (i.e., HTTP/HTTPS), then your ingress controller needs to use X-Forwarded-Proto, and issue a redirect accordingly.
If you're terminating TLS at the external load balancer and the LB is running in TCP/L4 mode, then your ingress controller needs to use the PROXY protocol to do the redirect.
You can also terminate TLS directly in your ingress controller, in which case it has all the necessary information to do the redirect.
Here's a tutorial on how to do this in Ambassador.
This is my first time trying to deploy a microservices architecture into Kubernetes. At the beginning, I was considering to use Ambassador as my API Gateway. I also have an authentication service which validates users and generates a JWT token, however, I need to validate this token
every time a service is called. This represents an overload problem (since every time the API Gateway receives traffic it will go to this external authentication service to validate the JWT token) and Ambassador does not have an option to do this filtering without the use of the external service.
Using the Zuul Gateway seems like the best option in this case, since it allows me to validate the JWT token inside the gateway (not through an external service like Ambassador). However, I'm not sure how Zuul is going to work if I deploy it in Kubernetes since, as I understand, Zuul requires to have the address of the service discovery (like Eureka).
if I deploy Zuul in my Kubernetes cluster, then how it will be able to locate my services?
Locally, for example, there is no problem since I was using Eureka before, and I knew its address. Also, I don't think having Eureka deployed in Kubernetes will be a good idea, since it will be redundant.
If it is not possible to do it with Zuul, is there another API Gateway or approach where I can validate tokens using the Gateway instead of relying on an external authentication service like Ambassador does?
Thank you.
In kubernetes you already have "discovery" service which is kubernetes-service. It locates pods and serves as load balancer for them.
Lets say you have Zuul configuration like this:
zuul:
routes:
books-service:
path: /books/**
serviceId: books-service
which routes requests matching /books/** to the service books-service. Usually you have an Eureka which gives you real address of books-service, but not now.
And this is where Ribbon can help you - it allows you to manually tune routing after Zuul has matched it's request. So you need to add this to configuration:
books-service.ribbon.listOfServers: "http://books:8080"
and after Zuul had found serviceId (books-service) it will route the request to books:8080
And books:8080 is just a kubernetes-service:
kind: Service
apiVersion: v1
metadata:
name: books
spec:
selector:
app: spring-books-service
ports:
- protocol: TCP
port: 8080
targetPort: 9376
You can say its a load balancer that takes traffic from :8080 and redirects it to pods with label app: spring-books-service.
All you have to do next is to assign labels to pods (via deployments for example)
Btw, you can configure Ribbon like this in any app and kubernetes will locate all your apps (pods) with its services so you dont need any discovery service at all! And since k8s-services are much more reliable than Eureka, you can simply remove it.
Hello I tried looking at the auth options in the annotations for kubernetes traefik ingress. I couldn't find anything where I could configure Forward Authentication as documented here: https://docs.traefik.io/configuration/entrypoints/#forward-authentication
I would like to be able to configure forward authentication per ingress resource. This is possible in the nginx ingress controller.
Is that supported currently?
According to the Traefik documentation that feature will be available in version 1.7 of Traefik (currently a release candidate).
Here is a link to the authentication documentation
My guess is that you will need to add the following 2 annotations:
ingress.kubernetes.io/auth-type: forward
ingress.kubernetes.io/auth-url: https://example.com
and probably also the following annotation with the corresponding header fields your auth service returns as value:
ingress.kubernetes.io/auth-response-headers: X-Auth-User, X-Secret