Docker swarm network latency with mesh and DNSRR - docker-compose

Question

I have a 3 node docker swarm.
One stack deployed is a database cluster with 3 replicas. (MariaDB Galera)
Another stack deployed is a web application with 2 replicas.
The web application looks like this:
version: '3'
networks:
web:
external: true
galera_network:
external: true
services:
application:
image: webapp:latest
networks:
- galera_network
- web
environment:
DB_HOST: galera_node
deploy:
replicas: 2
FWIW, the web network is what traefik is hooked up to.
The issue here is galera_node (used for the webapp's database host) resolves to a VIP that ends up leveraging swarm's mesh routing (as far as I can tell) and that adds extra latency when the mesh routing ends up going over the WAN instead of resolving to the galera_node container that is deployed on the same physical host.
Another option I've found is to use tasks.galera_node, but that seems to use DNSRR for the 3 galera cluster containers. So 33% of the time, things are good and fast... but the rest of the time, I have unnecessary latency added to the mix.
These two behaviors look to be documented as what we'd expect from the different endpoint_mode options. Reference: Docker endpoint_mode
To illustrate the latency, notice when pinging from within the webapp container:
Notice the IP addresses that are resolving for each ping along with the response time.
### hitting VIP that "masks" the fact that there is extra latency
### behind it depending on where the mesh routing sends the traffic.
root#294114cb14e6:/var/www/html# ping galera_node
PING galera_node (10.0.4.16): 56 data bytes
64 bytes from 10.0.4.16: icmp_seq=0 ttl=64 time=0.520 ms
64 bytes from 10.0.4.16: icmp_seq=1 ttl=64 time=0.201 ms
64 bytes from 10.0.4.16: icmp_seq=2 ttl=64 time=0.153 ms
--- galera_node ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.153/0.291/0.520/0.163 ms
### hitting DNSRR that resolves to worst latency server
root#294114cb14e6:/var/www/html# ping tasks.galera_node
PING tasks.galera_node (10.0.4.241): 56 data bytes
64 bytes from 10.0.4.241: icmp_seq=0 ttl=64 time=60.736 ms
64 bytes from 10.0.4.241: icmp_seq=1 ttl=64 time=60.573 ms
--- tasks.galera_node ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 60.573/60.654/60.736/0.082 ms
### hitting DNSRR that resolves to local galera_node container
root#294114cb14e6:/var/www/html# ping tasks.galera_node
PING tasks.galera_node (10.0.4.242): 56 data bytes
64 bytes from 10.0.4.242: icmp_seq=0 ttl=64 time=0.133 ms
64 bytes from 10.0.4.242: icmp_seq=1 ttl=64 time=0.117 ms
--- tasks.galera_node ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.117/0.125/0.133/0.000 ms
### hitting DNSRR that resolves to other "still too much" latency server
root#294114cb14e6:/var/www/html# ping tasks.galera_node
PING tasks.galera_node (10.0.4.152): 56 data bytes
64 bytes from 10.0.4.152: icmp_seq=0 ttl=64 time=28.218 ms
64 bytes from 10.0.4.152: icmp_seq=1 ttl=64 time=40.912 ms
64 bytes from 10.0.4.152: icmp_seq=2 ttl=64 time=26.293 ms
--- tasks.galera_node ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 26.293/31.808/40.912/6.486 ms
The only way I've been able to get decent performance that bypasses the latency is to hard code the IP address of the local container, but that is obviously not a long-term solution as containers should be treated as ephemeral things.
I totally get that I might need to rethink my geographic node locations due to this latency, and there might be some other performance tuning things I can do. It seems like there should be a way to enforce my desired behavior, though.
I essentially want to bypass DNSRR and the VIP/mesh routing behavior when a local container is available to service the given request.
So the question is:
How can I have each replica of my webapp only hit the local swarm host's galera container without hard coding that container's IP address?

Answer 1

If anyone else is fighting with this sort of issue, I wanted to post a solution (though I wouldn't necessarily call it an "answer" to the actual question) that is more of a workaround than something I'm actually happy with.
Inside of my webapp, I can use galera_node as my database host and it resolves to the mesh routing VIP that I mentioned above. This gives me functionality no matter what, so if my workaround gets tripped up I know that my connectivity is still in tact.
I whipped up a little bash script that I could call as a cron job and give me the results that I want. It could be used for other use cases that stem from this same issue.
It takes in three parameters:
$1 = database container name
$2 = database network name
$3 = webapp container name
The script looks for the container name, finds its IP address for the specified network, and then adds that container name and IP address to the webapp container's /etc/hosts file.
This works because the container name is also galera_node (in my case) so adding it to the hosts file just overrides the hostname that docker resolves to the VIP.
As mentioned, I don't love this, but it does seem to work for my purposes and it avoids me having to hardcode IP addresses and manually maintain them. I'm sure there are some scenarios that will require tweaks to the script, but it's a functional starting point.
My script: update_container_hosts.sh
#!/bin/bash
HOST_NAME=$1
HOST_NETWORK=$2
CONTAINER_NAME=$3
FMT="{{(index (index .NetworkSettings.Networks \"$HOST_NETWORK\") ).IPAddress}}"
CONTAINERS=`docker ps | grep $CONTAINER_NAME | cut -d" " -f1`
HOST_ID=`docker ps | grep $HOST_NAME | cut -d" " -f1`
HOST_IP=$(docker inspect $HOST_ID --format="$FMT")
echo --- containers ---
echo $CONTAINERS
echo ------------------
echo host: $HOST_NAME
echo network: $HOST_NETWORK
echo ip: $HOST_IP
echo ------------------
for c in $CONTAINERS;
do
if [ "$HOST_IP" != "" ]
then
docker cp $c:/etc/hosts /tmp/hosts.tmp
IP_COUNT=`cat /tmp/hosts.tmp | grep $HOST_IP | wc -l`
rm /tmp/hosts.tmp
if [ "$IP_COUNT" = "0" ]
then
docker exec $c /bin/sh -c "echo $HOST_IP $HOST_NAME >> /etc/hosts"
echo "$c: Added entry to container hosts file."
else
echo "$c: Entry already exists in container hosts file. Skipping."
fi
fi
done

Answer 2

I wrote a PoC for adjusting the loadbalancer to exlude containers on other hosts. It adjusts the config of the virtual IP itself, so there is no need to change anything in the container filesystem. It needs to be rerun, on every node in the cluster, whenever a container is stopped or started. It takes one argument, that is the exposed port, it will then figure out the virtual IP and the IPs of the containers. It needs nsenter and ipvsadm. I thought someone may find it useful.
#!/bin/bash
port="$1"
if [ -z "$port" ]; then
echo "Please specify port"
exit 1
fi
echo "Collecting data"
INGRESS_IP=$(iptables -t nat -S DOCKER-INGRESS |grep -- "--dport $port "|cut -d\ -f 12|cut -d: -f1)
if [ -z "$INGRESS_IP" ]; then
echo "Can't find ingress IP"
exit 1
fi
echo "INRESS_IP = $INGRESS_IP"
FWM_HEX=$( nsenter --net=/var/run/docker/netns/ingress_sbox iptables -t mangle -S PREROUTING|grep -- "--dport $port "|cut -d\ -f12|cut -d/ -f 1|cut -dx -f2)
FWM=$((16#$FWM_HEX))
echo "Firewall mark = $FWM"
declare -A LOCAL_CONTAINER_IPS
LOCAL_CONTAINERS=$(docker ps -q)
for c in $LOCAL_CONTAINERS; do
i=$(docker inspect $c|jq '.[0]["NetworkSettings"]["Networks"]["ingress"]["IPAMConfig"]["IPv4Address"]'|cut -d\" -f 2)
LOCAL_CONTAINER_IPS[$i]=1
done
LB_IPS=$(nsenter --net=/var/run/docker/netns/ingress_sbox ipvsadm -S|grep -- "-a -f $FWM -r"|cut -d\ -f5|cut -d: -f1)
declare -A EXISTING_CONTAINER_IPS
echo "Checking for IPs to remove"
for i in $LB_IPS; do
EXISTING_CONTAINER_IPS[$i]=1
if [ ! ${LOCAL_CONTAINER_IPS[$i]+_} ]; then
echo "$i is not a local container IP, deleting"
nsenter --net=/var/run/docker/netns/ingress_sbox ipvsadm -d -f $FWM -r $i:0
fi
done
echo "Checking for IPs to add"
for i in "${!LOCAL_CONTAINER_IPS[#]}"; do
if [ ! ${EXISTING_CONTAINER_IPS[$i]+_} ]; then
echo "$i is a local container IP but not in the load balancer, adding"
nsenter --net=/var/run/docker/netns/ingress_sbox ipvsadm -a -f $FWM -r $i:0 -m -w 1
fi
done
echo "done"