We are using Kubernetes v1.19.13 hosted on Google Kubernetes Engine. We want to configure an Ingress controller so that the Google HTTP(S) LoadBalancer is configured to allow only TLS 1.2 and 1.3 and these features/ciphers:
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
We would prefer to do this using annotations but most examples we have found uses a ConfigMap or FrontendConfig.
Is this possible to configure this using annotations? If not, what is the recommended way of achieving this?
Note that we want to configure this using Kubernetes and not using the Google Cloud Console.
You won't be able to do this using annotations. You cannot currently create an SSL Policy via annotations. SSL Policies need to be created via gcloud CLI or via the GCP API.
You'll then need to create a FrontendConfig resource which references the policy and then attach it to your ingress resource:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: FRONTENDCONFIG_NAME
spec:
sslPolicy: allowed-ciphers
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
networking.gke.io/v1beta1.FrontendConfig: "FRONTENDCONFIG_NAME"
...
The good news is that you can (re)use the SSL Policy and/or FrontendConfig.
If you want to do everything via the k8s API, you can try using Config Connector and create ComputeSSLPolicy resource.
To configure an Ingress controller to allow only TLS 1.2 and 1.3, you can use ngnix.ingress.kubernetes.io/proxy-ssl-protocols annotation.
nginx.ingress.kubernetes.io/proxy-ssl-protocols: Enables the specified protocols for requests to a proxied HTTPS server.
For Example:
annotations["nginx.ingress.kubernetes.io/proxy-ssl-protocols"] = "TLSv1.2 TLSv1.3"
To configure an Ingress controller to ciphers, you can use ngnix.ingress.kubernetes.io/proxy-ssl-ciphers annotation. nginx.ingress.kubernetes.io/proxy-ssl-ciphers: Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library.
Using ssl_ciphers annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host.
For Example Cipher :
nginx.ingress.kubernetes.io/ssl-ciphers: "ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
Refer SSL-ciphers for more information.
Related
We've an application and API, running on kubernetes on Azure, using an nginx-ingress and cert-manager which automatically creates letsencrypt certificates. The connection to the application/API is encrypted with TLS1.3.
From an older application, running on a Win 2012 server, we want to retrieve data from the API (on k8s). This isn't successful, since TLS1.3 isn't supported on that server.
I'd like to set the minimum version of TLS to 1.2 on kubernetes.
How can I achieve that?
I've read, that with kubelet, the tls-min-version can be configured, but I don't know how to apply this.
Note: we use az aks create to create the k8s clusters.
As your win server connects to the application on K8s you have to set the version of TLS on the Nginx ingress level.
Nginx ingress & cert-manager is point where you server connects and access API so you just have to update the TLS version of Nginx.
You can do it by changing the config map for Nginx ingress controller. Also, you might need to update the certificate also, there could be a chance by default Let's encrypt(CA) providing the default TLS 1.3.
So after enabling TLS 1.2 for Nginx you might need to re-generate the cert-manager secret(certificate).
Configmap Nginx ingress controller
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: ingress-nginx
data:
ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
ssl-protocols: "TLSv1.2 TLSv1.3"
above configmap will start both TLS versions for Nginx ingress controller.
Is there a way to force an SSL upgrade for incoming connections on the ingress load-balancer? Or if that is not possible with, can I disable port :80? I haven't found a good documentation pages that outlines such an option in the YAML file. Thanks a lot in advance!
https://github.com/kubernetes/ingress-gce#frontend-https
You can block HTTP through the annotation kubernetes.io/ingress.allow-http: "false" or redirect HTTP to HTTPS by specifying a custom backend. Unfortunately GCE doesn't handle redirection or rewriting at the L7 layer directly for you, yet. (see https://github.com/kubernetes/ingress-gce#ingress-cannot-redirect-http-to-https)
Update: GCP now handles redirection rules for load balancers, including HTTP to HTTPS. There doesn't appear to be a method to create these through Kubernetes YAML yet.
This was already correctly answered by a comment on the accepted answer. But since the comment is buried I missed it several times.
As of GKE version 1.18.10-gke.600 you can add a k8s frontend config to redirect from http to https.
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: ssl-redirect
spec:
redirectToHttps:
enabled: true
# add below to ingress
# metadata:
# annotations:
# networking.gke.io/v1beta1.FrontendConfig: ssl-redirect
The annotation has changed:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
annotations:
kubernetes.io/ingress.allow-http: "false"
spec:
...
Here is the annotation change PR:
https://github.com/kubernetes/contrib/pull/1462/files
If you are not bound to the GCLB Ingress Controller you could have a look at the Nginx Ingress Controller. This controller is different to the builtin one in multiple ways. First and foremost you need to deploy and manage one by yourself. But if you are willing to do so, you get the benefit of not depending on the GCE LB (20$/month) and getting support for IPv6/websockets.
The documentation states:
By default the controller redirects (301) to HTTPS if TLS is enabled for that ingress . If you want to disable that behaviour globally, you
can use ssl-redirect: "false" in the NGINX config map.
The recently released 0.9.0-beta.3 comes with an additional annotation for explicitly enforcing this redirect:
Force redirect to SSL using the annotation ingress.kubernetes.io/force-ssl-redirect
Google has responded to our requests and is testing HTTP->HTTPS SSL redirection on their load balancers. Their latest answer said it should be in Alpha sometime before the end of January 2020.
Their comment:
Thank you for your patience on this issue. The feature is currently in testing and we expect to enter Alpha phase before the end of January. Our PM team will have an announcement with more details as we get closer to the Alpha launch.
My fingers are crossed that we'll have a straightforward solution to this very common feature in the near future.
UPDATE (April 2020):
HTTP(S) rewrites is now a Generally Available feature. It's still a bit rough around the edges and does not work out-of-the-box with the GCE Ingress Controller unfortunately. But time will tell and hopefully a native solution will appear.
A quick update. Here
Now a FrontEndConfig can be make to configure the ingress. Hopes it helps.
Example:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: 301
You'll need to make sure that your load balancer supports HTTP and HTTPS
Worked on this for a long time. In case anyone isn't clear on the post above. You would rebuild your ingress with annotation -- kubernetes.io/ingress.allow-http: "false” --
Then delete your ingress and redeploy. The annotation will have the ingress only create a LB for 443, instead of both 443 and 80.
Then you do a compute HTTP LB, not one for GKE.
Gui directions:
Create a load balancer and choose HTTP(S) Load Balancing -- Start configuration.
choose - From Internet to my VMs and continue
Choose a name for the LB
leave the backend configuration blank.
Under Host and path rules, select Advanced host and path rules with the action set to
Redirect the client to different host/path.
Leave the Host redirect field blank.
Select Prefix Redirect and leave the Path value blank.
Chose the redirect response code as 308.
Tick the Enable box for HTTPS redirect.
For the Frontend configuration, leave http and port 80, for ip address select the static
IP address being used for your GKE ingress.
Create this LB.
You will now have all http traffic go to this and 308 redirect to your https ingress for GKE. Super simple config setup and works well.
Note: If you just try to delete the port 80 LB that GKE makes (not doing the annotation change and rebuilding the ingress) and then adding the new redirect compute LB it does work, but you will start to see error messages on your Ingress saying error 400 invalid value for field 'resource.ipAddress " " is in use and would result in a conflict, invalid. It is trying to spin up the port 80 LB and can't because you already have an LB on port 80 using the same IP. It does work but the error is annoying and GKE keeps trying to build it (I think).
Thanks to the comment of #Andrej Palicka and according to the page he provided: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect now I have an updated and working solution.
First we need to define a FrontendConfig resource and then we need to tell the Ingress resource to use this FrontendConfig.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-app-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: myapp-prd
networking.gke.io/managed-certificates: managed-cert
kubernetes.io/ingress.class: "gce"
networking.gke.io/v1beta1.FrontendConfig: myapp-frontend-config
spec:
defaultBackend:
service:
name: myapp-app-service
port:
number: 80
---
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: myapp-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: MOVED_PERMANENTLY_DEFAULT
You can disable HTTP on your cluster (note that you'll need to recreate your cluster for this change to be applied on the load balancer) and then set HTTP-to-HTTPS redirect by creating an additional load balancer on the same IP address.
I spend couple of hours on the same question, and ended up doing what I've just described. It works perfectly.
Redirecting to HTTPS in Kubernetes is somewhat complicated. In my experience, you'll probably want to use an ingress controller such as Ambassador or ingress-nginx to control routing to your services, as opposed to having your load balancer route directly to your services.
Assuming you're using an ingress controller, then:
If you're terminating TLS at the external load balancer and the LB is running in L7 mode (i.e., HTTP/HTTPS), then your ingress controller needs to use X-Forwarded-Proto, and issue a redirect accordingly.
If you're terminating TLS at the external load balancer and the LB is running in TCP/L4 mode, then your ingress controller needs to use the PROXY protocol to do the redirect.
You can also terminate TLS directly in your ingress controller, in which case it has all the necessary information to do the redirect.
Here's a tutorial on how to do this in Ambassador.
Hello I tried looking at the auth options in the annotations for kubernetes traefik ingress. I couldn't find anything where I could configure Forward Authentication as documented here: https://docs.traefik.io/configuration/entrypoints/#forward-authentication
I would like to be able to configure forward authentication per ingress resource. This is possible in the nginx ingress controller.
Is that supported currently?
According to the Traefik documentation that feature will be available in version 1.7 of Traefik (currently a release candidate).
Here is a link to the authentication documentation
My guess is that you will need to add the following 2 annotations:
ingress.kubernetes.io/auth-type: forward
ingress.kubernetes.io/auth-url: https://example.com
and probably also the following annotation with the corresponding header fields your auth service returns as value:
ingress.kubernetes.io/auth-response-headers: X-Auth-User, X-Secret
A typical ingress with TLS configuration is like below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: no-rules-map
spec:
tls:
- secretName: testsecret
backend:
serviceName: s1
servicePort: 80
By default, the load balancer will talk to backend service in HTTP. Can I configure Ingress so that the communication between load balancer and the backend service is also HTTPS?
Update:
Found GLBC talking about enabling HTTPS backend for GCE Ingress. Excerpt from the document:
"Backend HTTPS
For encrypted communication between the load balancer and your Kubernetes service, you need to decorate the service's port as expecting HTTPS. There's an alpha Service annotation for specifying the expected protocol per service port. Upon seeing the protocol as HTTPS, the ingress controller will assemble a GCP L7 load balancer with an HTTPS backend-service with a HTTPS health check."
It is not clear if the load balancer accepts 3rd party signed server certificate, self-signed, or both. How CA cert should be configured on load balancer to do backend server authentication. Or it will bypass authentication check.
On ingress-nginx you can use the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation and point it to the HTTPS port on the service.
This is ingress controller specific and other ingress controllers might not provide the option (or provide an option that works differently)
With this setup, the ingress controller decrypts the traffic. (This allows the ingress controller to control things like ciphers and the certificate presented to the user and do path-based routing, which SSL passthrough does not allow)
It is possible to configure certificate validation with serveral other (ingress-nginx-specific) annotations:
Docs
nginx.ingress.kubernetes.io/proxy-ssl-verify (it defaults to "off")
nginx.ingress.kubernetes.io/proxy-ssl-verify-depth
nginx.ingress.kubernetes.io/proxy-ssl-ciphers (ciphers, not validation related)
nginx.ingress.kubernetes.io/proxy-ssl-name (Override the name that the cert is checked against)
nginx.ingress.kubernetes.io/proxy-ssl-protocols (SSL / TLS versions)
nginx.ingress.kubernetes.io/proxy-ssl-server-name (SNI passthrough)
Follow the instructions in "Backend HTTPS" section of GLBC,
GCP HTTP(S) load balancer will build a HTTPS connection with the backend, traffic will be encrypted. There is no need to configure CA certificate on LB side (Actually you can't). This implies the load balancer will skip server certificate authentication.
You should enable SSL-Passthrough config for the ingress or load balancer.
I suggest you using nginx ingress and kube-lego for SSL.
with this combination, you can use the ssl-passthrough config.
Nginx ingress for k8s
kube-lego for generating SSL certificate on the fly
Guidance for enable ssl-passthrough config
My goal is to make my web application (deployed on Kubernetes 1.4 cluster) see the IP of the client that originally made the HTTP request. As I'm planning to run the application on a bare-metal cluster, GCE and the service.alpha.kubernetes.io/external-traffic: OnlyLocal service annotation introduced in 1.4 is not applicable for me.
Looking for alternatives, I've found this question which is proposing to set up an Ingress to achieve my goal. So, I've set up the Ingress and the NginX Ingress Controller. The deployment went smoothly and I was able to connect to my web app via the Ingress Address and port 80. However in the logs I still see cluster-internal IP (from 172.16.0.0/16) range - and that means that the external client IPs are not being properly passed via the Ingress. Could you please tell me what do I need to configure in addition to the above to make it work?
My Ingress' config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myWebApp
spec:
backend:
serviceName: myWebApp
servicePort: 8080
As a layer 4 proxy, Nginx cannot retain the original source IP address in the actual IP packets. You can work around this using the Proxy protocol (the link points to the HAProxy documentation, but Nginx also supports it).
For this to work however, the upstream server (meaning the myWebApp service in your case) also needs to support this protocol. In case your upstream application also uses Nginx, you can enable proxy protocol support in your server configuration as documented in the official documentation.
According to the Nginx Ingress Controller's documentation, this feature can be enabled in the Ingress Controller using a Kubernetes ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-ingress-controller
data:
use-proxy-protocol: "true"
Specify the name of the ConfigMap in your Ingress controller manifest, by adding the --nginx-configmap=<insert-configmap-name> flag to the command-line arguments.