Ingress on self-hosted Kubernetes on custom interface - kubernetes

Question

I would like to deploy an ngingx-ingress controller on my self-hosted Kubernetes (microk8s) that is configurable to listen on one or more interfaces (external IPs).
Not even sure if that is easily possible and I should just switch to an external solution, such as HAProxy or an nginx.
Required behavior:
192.168.0.1-H"domain.com":443/frontend -> 192.168.0.1 (eth0) -> ingress -> service-frontend
192.168.0.1-H"domain.com":443/backend -> 192.168.0.1 (eth0) -> ingress -> service-backend
88.88.88.88-H"domain.com":443/frontend -> 88.88.88.88 (eth1) -> ? -> [403 or timeout]
88.88.88.88-H"domain.com":443/backend -> 88.88.88.88 (eth1) -> ? -> [403 or timeout]
And then later the eth1 interface should be able to be switched on, so that requests on that interface behave the same as on eth0.
I would like to be able to deploy multiple instances of services for load-balancing. I would like to keep the configuration in my namespace (if possible) so I can always delete and apply everything at once.
I'm using this guide as a reference: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
I was able to get something working with minikube, but obviously could not expose any external IPs and performance was quite bad. For that, I just configured a "kind: Ingress" and that was it.
So far, the ingress controller that's default on microk8s seems to listen on all interfaces and I can only configure it in its own namespace. Defining my own ingress seems to not have any effect.

Answer 1

I would like to deploy an ngingx-ingress controller on my self-hosted
Kubernetes (microk8s) that is configurable to listen on one or more
interfaces (external IPs).
For above scenario, you have to deploy the multiple ingress controller of Nginx ingress and keep the different class name in it.
Official document : https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/
So in this scenario, you have to create the Kubernetes service with Loadbalancer IP and each will point to the respective deployment and class will be used in the ingress object.
If you looking forward to use the multiple domains with a single ingress controller you can easily do it by mentioning the host into ingress.
Example for two domain :
bar.foo.dev
foo.bar.dev
YAML example
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: frontdoor-bar
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
tls:
- hosts:
- bar.foo.dev
secretName: tls-secret-bar
rules:
- host: bar.foo.dev
http:
paths:
- backend:
serviceName: barfoo
servicePort: 80
path: /(.*)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: frontdoor-foo
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- foo.bar.dev
secretName: tls-secret-foo
rules:
- host: foo.bar.dev
http:
paths:
- backend:
serviceName: foobar
servicePort: 9000
path: /(.*)

Answer 2

One potential fix was much simpler than anticipated, no messing with MetalLB needed or anything else.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: "public"
nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.0.0/24
...
This does not answer the question of splitting an Ingress across multiple interfaces, but it does solve the problem of restricting public access.
By default, bare-metal ingress will listen on all interfaces, which might be a security issue.

Answer 3

This solution works without enabling ingress on Microk8s:
install ingress controller : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/baremetal/deploy.yaml
create your deployment and service and add this Ingress resource (all in the one namespace):
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/service-upstream: 'true'
nginx.ingress.kubernetes.io/rewrite-target: "/$2"
name: ingress-resource
namespace: namespace-name
spec:
rules:
- http:
paths:
- backend:
service:
name: service-name
port:
number: service-port
path: /namespace-name/service-name(/|$)(.*)
pathType: Prefix
kubectl get svc -n ingress-nginx
now get either CLUSTER-IP or EXTERNAL-IP and :
curl ip/namespace-here/service-here