externalTrafficPolicy: local with ingress - kubernetes

Question

We want to access only local services via Ingress using K3S (1.23) and Traefik.
We have an NGINX gateway running as a DaemonSet on all nodes, exposed as a NodePort 30123 called gateway with externalTrafficPolicy: Local. When we ping https://node1:30123 we consistently get only a local pod from the nginx instance on node1. ✔
We also have our Ingress Controller Traefik running as a DaemonSet exposed as a NodePort 30999 with externalTrafficPolicy: Local. When we hit https://node1:30999 we get a load-balanced answer ❌🤷‍♂️:
answer from node1
answer from node2
answer from node1
answer from node3
etc
How can we ensure that https://node1:30999 only gets routed to local pods?
Ingress Resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik
name: my-ingress
spec:
rules:
- host: node1
http:
paths:
- backend:
service:
name: gateway
port:
number: 8443
path: /
pathType: Prefix
- host: node2
http:
paths:
- backend:
service:
name: gateway
port:
number: 8443
path: /
pathType: Prefix
tls:
- hosts:
- node1
- node2
secretName: tls-secret
Gateway Service
apiVersion: v1
kind: Service
metadata:
annotations:
name: gateway
spec:
clusterIP: ***
clusterIPs:
- ***
externalTrafficPolicy: Local
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: https
nodePort: 30123
port: 8443
protocol: TCP
targetPort: 8443
selector:
app: gateway
sessionAffinity: None
type: NodePort
The reason we want only local traffic is because Kubernetes is too slow at evicting pods/endpoints from a service when a node goes down. It continues to send traffic to dead nodes/pods for minutes after a node disappears. We use an external load balancer with active health checks every 2s to avoid this problem. However, evern if our LB targets only healthy nodes, Kubernetes "Services" still have invalid endpoints and round-robin traffic into nowhere.

Answer 1

I guess this might be due to the fact that you are running traefic as a daemonset which behaves like a shared daemon running on all nodes and every node will then apply the local traffic policy to the request.
So basically running the Ingress Controller as a daemonset and setting the controllers service traffic Policy to Local will result in some behavior that equals the Cluster Policy.
Further the idea of the Ingress Controller is to route the traffic to a specific service in the cluster. Accessing a node directly on the Ingress-Controller Service Port will not trigger any forwarding rules. You should instead create a Ingress Resource that maps some existing service (Nginx-test-pod). This services traffic policy must then be set to "Local" too to see the real SourceIP in the services log.