Kubernetes Ingress Controller: Failed calling webhook, dial tcp connect: connection refused - kubernetes

Question

I have set up a Kubernetes cluster (a master and a worker) on two Centos 7 machines. They have the following IPs:
Master: 192.168.1.40
Worker: 192.168.1.41
They are accessible by SSH and I am not using a VPN. For both boxes, I have sudo access.
For the work I am doing, I had to add an Nginx Ingress Controller, which I did by doing:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.43.0/deploy/static/provider/baremetal/deploy.yaml
This yaml file seems fine to me and is a common one that occurs when trying to add an nginx ingress controller to a kubernetes cluster.
I don't see any errors when I do the above command.
However, when I try to install a helm configuration, such as:
helm install dai eggplant/dai --version 0.6.5 -f dai.yaml --namespace dai
I am getting an error with my Nginx Ingress Controller:
W0119 11:58:00.550727 60628 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": dial tcp 10.108.86.48:443: connect: connection refused
I think this is because of some kind of DNS error. I don't know where the IP 10.108.86.48:443 is coming from or how to find out.
I have also enabled a bunch of ports with firewall-cmd.
[root#manager-node ~]# sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens33
sources:
services: dhcpv6-client ssh
ports: 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 10255/tcp 443/tcp 30154/tcp 31165/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
However, my nginx ingress pod doesn't seem to start either:
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-7bc44b4bb-rwmh2 0/1 ContainerCreating 0 19h
It remains as ContainerCreating for hours.

Answer 1

The issue is that as part of that kubectl apply -f you are also applying a ValidatingWebhookConfiguration (check the applied manifest file).
See Using Admission Controllers | Kubernetes
Using Admission Controllers | Kubernetes for more info.
The error you are seeing is because your Deployment is not starting up, and thus the ValidatingWebhook service configured as part of it isn't starting up either, so the Validating Controller in Kubernetes is failing every request.
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
Your pod is most likely not starting for another reason. More information is required to further debug.
I would recommend removing the ValidatingWebhookConfiguration from the applied manfiest.
You can also remove it manually with
kubectl delete ValidatingWebhookCOnfiguration ingress-nginx-admission
(Validating Controllers aren't namespaced)