Should Kubernetes host nodes be able to access services running in the cluster? - kubernetes

Question

I am running:
Kubernetes v1.19.7 (On-premise, VMs. Provisioned via Kubespray)
MetalLB
Calico
nginx-ingress
Summary: Services are refusing to respond when queried from the host nodes. Is this even supposed to work? If not I can stop banging my head against this particular wall...
I am able to access service.foo.com from anywhere on my local network, however if I try to use something like cURL to make a request to service.foo.com from any of the host nodes I get "Connection refused" errors (but I can ping the service with no issue). I get the same behavior from within any pod running on the k8s cluster.
This is making things particularly difficult since I'm trying to set up and OIDC provider to use for gating access to the k8s dashboard, and host node needs to be able to query the provider.
Network Setup:
kube service addresses: 10.233.0.0/18
pods cidr: 10.233.64.0/18
MetalLB config:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.16.31.75-172.16.31.79
Ingress Controller Service described
Name: foo-com-ic-nginx-ingress
Namespace: default
Labels: app.kubernetes.io/instance=foo-com-ic
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=foo-com-ic-nginx-ingress
helm.sh/chart=nginx-ingress-0.8.0
Annotations: <none>
Selector: app=foo-com-ic-nginx-ingress
Type: LoadBalancer
IP Families: <none>
IP: 10.233.48.18
IPs: <none>
IP: 172.16.31.76
LoadBalancer Ingress: 172.16.31.76
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 31445/TCP
Endpoints: 10.233.105.18:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31173/TCP
Endpoints: 10.233.105.18:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30406
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal nodeAssigned 9m4s (x4 over 43m) metallb-speaker announcing from node "node4"
Service Ingress described
Name: my-service
Namespace: default
Address: 172.16.31.76
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
SNI routes service.foo.com
Rules:
Host Path Backends
---- ---- --------
service.foo.com / my-service:80 (10.233.96.27:80)
Annotations: kubernetes.io/ingress.class: service.com
meta.helm.sh/release-name: my-service
meta.helm.sh/release-namespace: default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 46m (x2 over 46m) nginx-ingress-controller Configuration for default/my-service was added or updated

Answer 1

Just in case someone comes across this issue while researching their own, I was eventually able to work around this. By chance I noticed that I could cURL the service from the node that the ingress controller pod was running on.
My work-around then was to change my ingress controller's installation kind from "deployment" to "daemonset". Now that the ingress controller pod runs on every node I am able to access the service from every node in the cluster.