Kubernetes Ingress on GKE - kubernetes

I deployed Kubernetes on AWS with KOPS and the nginx-ingress.
To evaluate multiple clouds (and cut costs), I want to deploy on GKE. Everything worked, except the darn Ingress's. (That was the hardest part on AWS).
Below is the Ingress I'm using on GKE. It makes two Ingresses in the dashboard, each with an IP address.
If I point my DNS at those addresses, the connection is refused. I'm checking the DNS resultion with ping.
All HTTPS fail to connect with "Unable to establish SSL connection.", except button which is "502 Bad Gateway"
HTTP fails to connect with 502 except admin which is 503.
In the Google Cloud Platform dashboard, I see two load balancers. "all" points to my SSL cert. "button" isn't doing HTTPS, but that's another problem.
Clearly I'm missing something. What did I miss?
I'm using kubectl v1.4.6 and whatever version on GKE would have installed yesterday.
```
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# this is for nginx ingress controler on AWS
# kubernetes.io/ingress.class: "nginx"
name: all-ingress
spec:
tls:
- hosts:
- admin-stage.example.com
- dashboard-stage.example.com
- expert-stage.example.com
- signal-stage.example.com
- stage.example.com
secretName: tls-secret
rules:
- host: admin-stage.example.com
http:
paths:
- backend:
serviceName: admin-service
servicePort: http-port
path: /
- host: dashboard-stage.example.com
http:
paths:
- backend:
serviceName: dashboard-service
servicePort: http-port
path: /
- host: expert-stage.example.com
http:
paths:
- backend:
serviceName: expert-service
servicePort: http-port
path: /
- host: signal-stage.example.com
http:
paths:
- backend:
serviceName: signal-service
servicePort: http-port
path: /
- host: stage.example.com
http:
paths:
- backend:
serviceName: www-service
servicePort: http-port
path: /
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# this is for nginx ingress controler on AWS
# kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/ssl-redirect: "false"
name: button-ingress
spec:
tls:
- hosts:
- button-stage.example.com
secretName: tls-secret
rules:
- host: button-stage.example.com
http:
paths:
- backend:
serviceName: button-service
servicePort: http-port
path: /
```

Prashanth's comments were helpful, in the end, native cloud Ingress (AWS/GCE) isn't finished in Kubernetes enough to be useful for my purposes. There's no point learning an abstraction that is more complicated and less functional than the thing underneath.
I ended up using the nginx-ingress from this answer: Kubernetes 1.4 SSL Termination on AWS
On the resulting Ingress is an IP you can point DNS (not the "External Endpoints" on the service). Good luck!

Related

In Kubernetes, how to setup multiple hosts in one ingress with let's encrypt certificates

I have setup a backend and frontend service running on Kubernetes. Frontend would be www.<myDomain>.com and backend would be api.<myDomain>.com
I need to expose and secure both services. I wish to use one ingress. I want to use free certificates from let's encrypt + cert manager. I guess a certificate for <myDomain>.com should cover both www. and api..
Pretty normal use case, right? But when these normal stuff comes together, I couldn't figure out the combined yaml. I was able to get single service, the www.<myDomain>.com working with https. Things doesn't work when I tried to add the api.<myDomain>.com
I'm using GKE, but this doesn't seem to be a platform related question. Now creating ingress takes forever. This following events has been tried again and again
Error syncing to GCP: error running load balancer syncing routine: loadbalancer <some id here> does not exist: googleapi: Error 404: The resource 'projects/<project>/global/sslCertificates/<some id here>' was not found, notFound
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-ingress
annotations:
kubernetes.io/ingress.class: gce
kubernetes.io/ingress.allow-http: "true"
cert-manager.io/issuer: letsencrypt-staging
spec:
tls:
- secretName: web-ssl
hosts:
- <myDomain>.com
rules:
- host: "www.<myDomain>.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: angular-service
port:
number: 80
- host: "api.<myDomain>.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: spring-boot-service
port:
number: 8080
I figured it out. I used different path for same domain to solve this problem. I'm not saying 2 domains won't work. There are some other configs like CORS related. But single domain different path is a happy path for me. Here is the rules part I used in my ingress.
- host: "www.<myDomain>.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: angular-service
port:
number: 80
- pathType: Prefix
path: "/api"
backend:
service:
name: spring-boot-service
port:
number: 8080
I faced the same requirement as you.
from
tls:
- secretName: web-ssl
hosts:
- <myDomain>.com
change to
tls:
- hosts:
- www.<myDomain>.com
secretName: web-ssl
- hosts:
- api.<myDomain>.com
secretName: web-ssl
Help me to solve the issue!

Any solution for multi-tenant setup with different DNS?

I have set up my frontend cluster in my Kubernetes and exposed as frontend.loaner.com and I want to point the DNS record of these both johndoe.loaner.com, janedoe.loaner.com to see the frontend.loaner.com.
Is that possible to just point two DNS to a 1 running server and works fine still having the hostname?
I read about the CNAME but it will redirect me to frontend.loaner.com
You can do it with a Kubernetes Ingress. Basically, something like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: frontend.loaner.com
http:
paths:
- path: /
backend:
serviceName: backend1
servicePort: 80
- host: johndoe.loaner.com
http:
paths:
- path: /
backend:
serviceName: backend2
servicePort: 80
- host: janedoe.loaner.com
http:
paths:
- path: /
backend:
serviceName: backend3
servicePort: 80
The above Ingress resource assumes you are using an Nginx Ingress Controller in your cluster.

how to configure ingress to direct traffic to an https backend using https

I have a backend using https.
I want to separate load on that back-end based on URL/path.
I decided to use ingress to do this url/path based logic in order to move traffic to different back-ends ( same back-ends , just duplicated to different NodePorts )
my question is how I can configure the ingress to receive https requests and to forward those https requests to the https back-end?
thanks
edit:
I added the yaml file:
spec:
rules:
- http:
paths:
- backend:
serviceName: service
servicePort: 9443
path: /carbon
- backend:
serviceName: service2
servicePort: 9443
path: /oauth
for some reason I can;t change the rule form http to https
Attention: This answer applies to the ingress-nginx solution provided by the kubernetes organisation on github (https://github.com/kubernetes/ingress-nginx)
If you want to use load balancing mechanisms in k8s you should use services instead and start multiple instances behind that service that way k8s will do the load balancing. If you want to use different versions of your backend (e.g. prod and test) your way of separating them is fine
if your service is only reachable via https you need to add the following annotation to your ingress yaml: (documentation)
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
To secure ingress itself take a look at this: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
But if you want that the backend services decrypt the TLS communication use the following annotation instead: (documentation)
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
Edit:
The Ingress YAML should look like this if you want to reach the backend via TLS:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-name
namespace: namespace-name
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- http:
paths:
- backend:
serviceName: service
servicePort: 9443
path: /carbon
- backend:
serviceName: service2
servicePort: 9443
path: /oauth
The Ingress YAML should look like this if you want to reach the backend via TLS with TLS decryption in the ingress controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-name
namespace: namespace-name
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- hosts:
- app.myorg.com
secretName: tls-secret
rules:
- http:
paths:
- backend:
serviceName: service
servicePort: 9443
path: /carbon
- backend:
serviceName: service2
servicePort: 9443
path: /oauth
It's important to note that tls-secret is the name of a SecretConfig with a valid Certificate issued for the host (app.myorg.com)
The Ingress YAML should look like this if you want to reach the backend via TLS with TLS decryption in the backend:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-name
namespace: namespace-name
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
rules:
- http:
paths:
- backend:
serviceName: service
servicePort: 9443
path: /carbon
- backend:
serviceName: service2
servicePort: 9443
path: /oauth
I never tested the last version myself so i don't know if that actually works but I'd strongly advise reading this passage for that variant.
If you are using the NGINX Ingress controller (https://docs.nginx.com/nginx-ingress-controller/), the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" does not work. However, the nginx.org/ssl-services will let you pick the services that require TLS on the backend. The name is confusing, so it took me a while to realize the real purpose of it.
This does not work with the standard Kubernetes Ingress controller that uses NGINX under the hood; it only works with the NGINX-sourced controller.
Advanced annotation docs: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
In this example, NGINX will connect to the ssl-svc using TLS; it ignores any self-signed certificates. Example (https://github.com/nginxinc/kubernetes-ingress/tree/v1.12.0/examples/ssl-services):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
nginx.org/ssl-services: "ssl-svc"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
- path: /ssl
backend:
serviceName: ssl-svc
servicePort: 443

k8s, Ingress, Minio, and a Static Site

We have a k8s cluster with an nginx Ingress and Minio installed. In Minio I have a bucket called tester with a hello world index.html file. I used the Minio MC client to set the tester bucket to public. Now I am able to see the hello world file when I visit my (altered) minio url like so: https://minio.example.com/tester/index.html.
My goal is to set up an Ingress resource to access the public bucket. Here is my manifest to try and do so, however I only ever get a 404 error . . .
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: frontend-site
namespace: "default"
labels:
type: "frontend"
awesomeness: "super-mega"
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: app.example.com
http:
paths:
- path: /tester/
backend:
serviceName: minio-svc
servicePort: 9000
- path: /tester/*
backend:
serviceName: minio-svc
servicePort: 9000
tls:
- hosts:
- app.example.com
secretName: ssl-certs
I have also tried to set the paths with the index fileto no avail like so:
path: /tester/index.html
path: /tester/index.html/*
I do have another Ingress which points to Minio in general and it works perfect at the url like minio.example.com. The minio has a service called minio-svc on port 9000.
Unfortunately I have only ever received a 404 from my Ingress thus far. Anyone else deploying static sites with Ingress to public Minio bucket? What am I doing wrong???
Updates
So I kind of got somewhere. I added an annotation and set the paths to simply / and /*.
Here is my new config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: frontend-site
namespace: "default"
labels:
type: "frontend"
awesomeness: "super-mega"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /tester/index.html
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
backend:
serviceName: minio-svc
servicePort: 9000
- path: /*
backend:
serviceName: minio-svc
servicePort: 9000
tls:
- hosts:
- app.example.com
secretName: ssl-certs
Now I just get access denied from Minio even though the bucket is public and I can still access from https://minio.example.com/tester/index.html!?
Found out you can't do what I'm asking very easily. I got around it all by simply mounting the directory from the bucket directly to Nginx. Voila!

Redirection from http to https not working for custom backend service in Kubernetes Nginx Ingress Controller

I have setup Custom Nginx ingress controller with Ingress resource in Kubernetes and instead of "default-http-backend service", I used custom application as the default backend service to be served for default requests. I have also used custom SSL which is set as kubernetes secret, for my service. The issue is that when I request the hostnames which are mentioned in the rules, the https redirection works. But when the requests other than the hosts mentioned in the rules are made, it serves the default app, but the https redirection does not work.
How can I redirect requests from http to https for all the requests including default requests. In other words, how to setup https redirection for wildcard domains in ingress resource.
Please find my yaml files for ingress resource.
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-resource
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/ingress.allow-http: "false"
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/proxy-connect-timeout: "14400"
ingress.kubernetes.io/proxy-send-timeout: "14400"
ingress.kubernetes.io/proxy-read-timeout: "14400"
spec:
tls:
- secretName: tls-secret
rules:
- host: service1.example.com
http:
paths:
- path: /
backend:
serviceName: service1
servicePort: 80
- host: service2.example.com
http:
paths:
- path: /
backend:
serviceName: service2
servicePort: 80
---
I needed to configure custom service (not default-http-backend service) for default requests which does not have rules set and this custom service should use custom SSL. At present nginx-ingress-controller doesn't do anything if the domain names are omitted from the Ingress rules (with the intention of the "wildcard" TLS cert being used).
Therefore I have added the following code in the ingress yaml I used and this works perfectly. I have added the wildcard tls name in ingress rules at the bottom for the custom default service. Please find the code below:
rules:
- host: service1.example.com
http:
paths:
- path: /
backend:
serviceName: service1
servicePort: 80
- host: service2.example.com
http:
paths:
- path: /
backend:
serviceName: service2
servicePort: 80
- host: '*.example.com'
http:
paths:
- path: /
backend:
serviceName: custom-backend-service
servicePort: 80