Load is not balanced with Kubernetes Services - kubernetes

Question

I created two replicas of nginx with following yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.20-alpine
ports:
- containerPort: 80
And I created service with:
apiVersion: v1
kind: Service
metadata:
name: nginx-test-service
spec:
selector:
app: nginx
ports:
- port: 8082
targetPort: 80
Everything looks good. But when I do
minikube service nginx-test-service
I am able to access the nginx. But when I see the two pods logs, the request is always going to single pod. The other pod is not getting any request.
But, kubernetes service should do the load balancing right?
Am I missing anything?

Answer 1

One way to get load balancing on-premise running is with ip virtual services. (ipvs). It;s a service which hands out ip's of the next pod to schedule/call
it's likely installed already.
lsmod | grep ip_vs
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 19
Have your cni properly setup and run
kubectl edit cm -n kube-system kube-proxy
edit the ipvs section
set mode to ipvs
mode: "ipvs"
and the ipvs section
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: "rr"
As always there are lots of variables biting each other with k8s, but it is possible with ipvs.
https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/