Kubernetes ingress to pod running on same host? - kubernetes

Question

We are just getting started with k8s (bare metal on Ubuntu 20.04). Is it possible for ingress traffic arriving at a host for a load balanced service to go to a pod running on that host (if one is available)?
We have some apps that use client side consistent hashing (using customer ID) to select a service instance to call. The service instances are stateless but maintain in memory ML models for each customer. So it is useful (but not essential) to have repeated requests for a given customer go to the same service. Then we can just use antiAffinity to have one pod per host.
Our existing service discovery mechanism lets the clients find all the instances of the service and the nodes they are running on. All our k8s nodes are running the Nginx ingress controller.

Answer 1

I finally got this figured out. This was way harder than it should be IMO! Update: It's not working. Traffic frequently goes to the wrong pod.
The service needs externalTrafficPolicy: Local (see docs).
apiVersion: v1
kind: Service
metadata:
name: starterservice
spec:
type: LoadBalancer
selector:
app: starterservice
ports:
- port: 8168
externalTrafficPolicy: Local
The Ingress needs nginx.ingress.kubernetes.io/service-upstream: "true" (service-upstream docs).
The nginx.ingress.kubernetes.io/server-alias: "~^starterservice-[a-z0-9]+\\.example\\.com" bit is because our service discovery updates DNS so each instance of the service includes the name of the host it is running on in its DNS name.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: starterservice
namespace: default
annotations:
nginx.ingress.kubernetes.io/server-alias: "~^starterservice-[a-z0-9]+\\.example\\.com"
nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
rules:
- host: starterservice.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: starterservice
port:
number: 8168
So now a call https://starterservice-foo.example.com will go to the instance running on k8s host foo.

Answer 2

I believe Sticky Sessions is what you are looking for. Ingress does not communicate directly with pods, but with services. Sticky sessions try to bind requests from the same client to the same pod by setting an affinity cookie.
This is used for example with SignalR sessions, where the negotiation request has to be on the same host as the following websocket connection.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
rules:
- http:
paths:
- path: /testpath
pathType: Prefix
backend:
service:
name: test
port:
number: 80
Affinity mode "balanced" is the default. If your pod count does not change, part of your clients will lose the session. Use "persistent" to have users connect to the same pod always (unless it dies of course). Further reading: https://github.com/kubernetes/ingress-nginx/issues/5944