Microk8s + metallb + ingress - kubernetes

Question

Im quite new to kubernetes and Im trying to set up a microk8s test environment on a VPS with CentOS.
What I did:
I set up the cluster, enabled the ingress and metallb
microk8s enable ingress
microk8s enable metallb
Exposed the ingress-controller service:
apiVersion: v1
kind: Service
metadata:
name: ingress
namespace: ingress
spec:
type: LoadBalancer
selector:
name: nginx-ingress-microk8s
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
- name: https
protocol: TCP
port: 443
targetPort: 443
Exposed an nginx deployment to test the ingress
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: nginx
name: nginx-deploy
spec:
replicas: 1
selector:
matchLabels:
run: nginx-deploy
template:
metadata:
labels:
run: nginx-deploy
spec:
containers:
- image: nginx
name: nginx
This is the status of my cluster:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/hostpath-provisioner-5c65fbdb4f-m2xq6 1/1 Running 3 41h
kube-system pod/coredns-86f78bb79c-7p8bs 1/1 Running 3 41h
kube-system pod/calico-node-g4ws4 1/1 Running 6 42h
kube-system pod/calico-kube-controllers-847c8c99d-xhmd7 1/1 Running 4 42h
kube-system pod/metrics-server-8bbfb4bdb-ggvk7 1/1 Running 0 41h
kube-system pod/kubernetes-dashboard-7ffd448895-ktv8j 1/1 Running 0 41h
kube-system pod/dashboard-metrics-scraper-6c4568dc68-l4xmg 1/1 Running 0 41h
container-registry pod/registry-9b57d9df8-xjh8d 1/1 Running 0 38h
cert-manager pod/cert-manager-cainjector-5c6cb79446-vv5j2 1/1 Running 0 12h
cert-manager pod/cert-manager-794657589-srrmr 1/1 Running 0 12h
cert-manager pod/cert-manager-webhook-574c9758c9-9dwr6 1/1 Running 0 12h
metallb-system pod/speaker-9gjng 1/1 Running 0 97m
metallb-system pod/controller-559b68bfd8-trk5z 1/1 Running 0 97m
ingress pod/nginx-ingress-microk8s-controller-f6cdb 1/1 Running 0 65m
default pod/nginx-deploy-5797b88878-vgp7x 1/1 Running 0 20m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 42h
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 41h
kube-system service/metrics-server ClusterIP 10.152.183.243 <none> 443/TCP 41h
kube-system service/kubernetes-dashboard ClusterIP 10.152.183.225 <none> 443/TCP 41h
kube-system service/dashboard-metrics-scraper ClusterIP 10.152.183.109 <none> 8000/TCP 41h
container-registry service/registry NodePort 10.152.183.44 <none> 5000:32000/TCP 38h
cert-manager service/cert-manager ClusterIP 10.152.183.183 <none> 9402/TCP 12h
cert-manager service/cert-manager-webhook ClusterIP 10.152.183.99 <none> 443/TCP 12h
echoserver service/echoserver ClusterIP 10.152.183.110 <none> 80/TCP 72m
ingress service/ingress LoadBalancer 10.152.183.4 192.168.0.11 80:32617/TCP,443:31867/TCP 64m
default service/nginx-deploy ClusterIP 10.152.183.149 <none> 80/TCP 19m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 42h
metallb-system daemonset.apps/speaker 1 1 1 1 1 beta.kubernetes.io/os=linux 97m
ingress daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 65m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 41h
kube-system deployment.apps/coredns 1/1 1 1 41h
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 42h
kube-system deployment.apps/metrics-server 1/1 1 1 41h
kube-system deployment.apps/dashboard-metrics-scraper 1/1 1 1 41h
kube-system deployment.apps/kubernetes-dashboard 1/1 1 1 41h
container-registry deployment.apps/registry 1/1 1 1 38h
cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 12h
cert-manager deployment.apps/cert-manager 1/1 1 1 12h
cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 12h
metallb-system deployment.apps/controller 1/1 1 1 97m
default deployment.apps/nginx-deploy 1/1 1 1 20m
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/hostpath-provisioner-5c65fbdb4f 1 1 1 41h
kube-system replicaset.apps/coredns-86f78bb79c 1 1 1 41h
kube-system replicaset.apps/calico-kube-controllers-847c8c99d 1 1 1 42h
kube-system replicaset.apps/metrics-server-8bbfb4bdb 1 1 1 41h
kube-system replicaset.apps/kubernetes-dashboard-7ffd448895 1 1 1 41h
kube-system replicaset.apps/dashboard-metrics-scraper-6c4568dc68 1 1 1 41h
container-registry replicaset.apps/registry-9b57d9df8 1 1 1 38h
cert-manager replicaset.apps/cert-manager-cainjector-5c6cb79446 1 1 1 12h
cert-manager replicaset.apps/cert-manager-794657589 1 1 1 12h
cert-manager replicaset.apps/cert-manager-webhook-574c9758c9 1 1 1 12h
metallb-system replicaset.apps/controller-559b68bfd8 1 1 1 97m
default replicaset.apps/nginx-deploy-5797b88878 1 1 1 20m
It looks like Metallb works, as the ingress services received an ip from the pool I specified.
Now, when I try to deploy an ingress to reach the nginx deployment, I dont get the ADDRESS:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
name: ingress-nginx-deploy
spec:
rules:
- host: test.com
http:
paths:
- backend:
serviceName: nginx-deploy
servicePort: 80
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default ingress-nginx-deploy <none> test.com 80 13m
An help would be really appreciated. Thank you!

Answer 1

TL;DR
There are some ways to fix your Ingress so that it would get the IP address.
You can either:
Delete the kubernetes.io/ingress.class: nginx and add ingressClassName: public under spec section.
Use the newer example (apiVersion) from official documentation that by default will have assigned an IngressClass:
Kubernetes.io: Docs: Concepts: Services networking: Ingress
Example of Ingress resource that will fix your issue:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-nginx-deploy
spec:
ingressClassName: public
# above field is optional as microk8s default ingressclass will be assigned
rules:
- host: test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-deploy
port:
number: 80
You can read more about IngressClass by following official documentation:
Kubernetes.io: Blog: Improvements to the Ingress API in Kubernetes 1.18
I've included more explanation that should shed some additional light on this particular setup.
After you apply above Ingress resource the output of:
$ kubectl get ingress
Will be following:
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-nginx-deploy public test.com 127.0.0.1 80 43s
As you can see the ADDRESS contains 127.0.0.1. It's because this particular Ingress controller enabled by an addon, binds to your host (MicroK8S node) to ports 80,443.
You can see it by running:
$ sudo microk8s kubectl get daemonset -n ingress nginx-ingress-microk8s-controller -o yaml
A side note!
Look for hostPort and securityContext.capabilities.
The Service of type LoadBalancer created by you will work with your Ingress controller but it will not be displayed under ADDRESS in $ kubectl get ingress.
A side note!
Please remember that in this particular setup you will need to connect to your Ingress controller with a Header Host: test.com unless you have DNS resolution configured to support your setup. Otherwise you will get a 404.
Additional resource:
Github.com: Ubuntu: Microk8s: Ingress with MetalLb issues
Kubernetes.io: Docs: Concepts: Configuration: Overview