Kubernetes way to Load balance/Failover external MariaDB PXC cluster nodes - kubernetes

Question

I am trying to setup access to an external MySQL database that is setup using MariaDB PXC three node cluster.
Let say my external database nodes has these IP addresses
172.16.10.100
172.16.10.101
172.16.10.102
In case one of those nodes goes down, I want Kubernetes to automatically route traffic to only available two nodes.
If I create a simple Service and Endpoints in kubernetes (showing below), does it automatically do a failover?
#
# Service
#
kind: Service
apiVersion: v1
metadata:
name: mariadb-service
spec:
clusterIP: None
sessionAffinity: None
ports:
- port: 3306
protocol: TCP
targetPort: 3306
---
#
# Endpoints
#
kind: Endpoints
apiVersion: v1
metadata:
name: mariadb-service
subsets:
- addresses:
- ip: 172.16.10.100
- ip: 172.16.10.101
- ip: 172.16.10.102
ports:
- port: 3306
protocol: TCP

Answer 1

Please note this:
Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. The implementations of network load balancers that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.
I found the option to have LoadBalancer with on premises Kubernetes cluster - that is MetalLB
MetalLB aims to redress this imbalance by offering a network load balancer implementation that integrates with standard network equipment, so that external services on bare-metal clusters also “just work” as much as possible.
See the requirements and the installation options you prefer.
MetalLB respects the spec.loadBalancerIP parameter, so if you want your service to be set up with a specific address, you can request it by setting that parameter. MetalLB also supports requesting a specific address pool, if you want a certain kind of address but don’t care which one exactly. To request assignment from a specific pool, add the metallb.universe.tf/address-pool annotation to your service, with the name of the address pool as the annotation value. For example:
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
metallb.universe.tf/address-pool: production-public-ips
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
type: LoadBalancer
And for address-pool annotation you can define it something like:
address-pools:
name: production-public-ips
protocol: TCP
addresses:
- ip: 172.16.10.100
- ip: 172.16.10.101
- ip: 172.16.10.102
ports:
- port: 3306
Find full example usage here.
However, there is no health check implemented, that is main option for you.
There is an ideal issue example on GitHub and this thread explaining on why health check is not implemented for Kubernetes CRDs.
If we jump into Kubernetes concepts, this use case is not doable and you can seek for some custom endpoint controller:
readinessProbe: Indicates whether the container is ready to respond to requests. If the readiness probe fails, the endpoints controller removes the Pod's IP address from the endpoints of all Services that match the Pod. The default state of readiness before the initial delay is Failure. If a Container does not provide a readiness probe, the default state is Success.
Please find more information in official documentation