"notify" is not getting called in Keepalived - haproxy

Question

I have these setting in the keepalived.conf file but when I stop the HAProxy service it's not executing the notify script but when I restart the keepalived service it's getting executed every time. Here are the details,
HAProxy: 1.8.8
Keepalived: 2.0.18
OS: Ubuntu 18.04
Python: 2.7
Cloud Service Provider: Hetzner
/etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
# Requires keepalived-1.1.13
script "/usr/bin/pkill -0 haproxy" # cheaper than pidof
interval 2 # check every 2 seconds
weight 2 # add 2 points of priority if OK
}
vrrp_instance real {
interface eth0
state MASTER
virtual_router_id 51
priority 101 # 101 on primary, 100 on secondary
virtual_ipaddress {
11.23.10.19/32 dev eth0 label eth0:1
}
track_script {
chk_haproxy
}
notify "/etc/keepalived/master.sh"
#notify_backup "/etc/keepalived/master.sh"
#notify_fault "/etc/keepalived/master.sh"
}
/etc/keepalived/master.sh
#!/bin/bash
export API_TOKEN='<api_token>'
export MASTER_SERVER_ID='<master_server_id>'
export BACKUP_SERVER_ID='<backup_server_id>'
BASE_API='https://api.hetzner.cloud/v1'
FLOATING_IP_ID='<floating_ip_id>'
INSTANCE="Load-Balancer-Master"
if [ "$HOSTNAME" = "$INSTANCE" ]; then
SERVER_ID=$BACKUP_SERVER_ID # switch to the backup server if
# master gets down
else
SERVER_ID=$MASTER_SERVER_ID # vice-versa
fi
echo "Server ID: " $SERVER_ID
HAS_FLOATING_IP=$(curl -H "Authorization: Bearer $API_TOKEN" -s 'https://api.hetzner.cloud/v1/servers/'$SERVER_ID|python -c "import sys,json; print( True if json.load(sys.stdin)['server']['public_net']['floating_ips'] else False)")
echo "Has Floating Ip: " $HAS_FLOATING_IP
if [ $HAS_FLOATING_IP = "False" ]; then
n=0
while [ $n -lt 10 ]
do
python /usr/local/bin/assign-ip $FLOATING_IP_ID $SERVER_ID && break
n=$((n+1))
sleep 3
done
fi
/usr/local/bin/assign-ip
#!/usr/bin/python
import os
import sys
import requests
import json
api_base = 'https://api.hetzner.cloud/v1'
def usage():
print('{0} [Floating IP] [Server ID]'.format(sys.argv[0]))
print('\nYour Hetzner API token must be in the "API_TOKEN"'
' environmental variable.')
def main(floating_ip_id, server_id):
payload = {'server': server_id}
headers = {'Authorization': 'Bearer {0}'.format(os.environ['API_TOKEN']),
'Content-type': 'application/json'}
url = api_base + "/floating_ips/{0}/actions/assign".format(floating_ip_id)
r = requests.post(url, headers=headers, data=json.dumps(payload))
resp = r.json()
if resp['action']['error']:
print('{0}: {1}'.format(resp['action']['command'], resp['error']['message']))
sys.exit(1)
else:
print('Moving IP address to server: {0} with status:{1}'.format(server_id, resp['action']['status']))
if __name__ == "__main__":
if 'API_TOKEN' not in os.environ or not len(sys.argv) > 2:
usage()
sys.exit()
main(sys.argv[1], sys.argv[2])
When I stop the HAProxy server using sudo service haproxy stop and check the status I get this response,
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-09-28 22:12:57 IST; 1s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 26434 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, stat
Process: 26423 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SU
Main PID: 26434 (code=exited, status=143)
Sep 28 00:44:18 Load-Balancer-Master haproxy[26434]: Proxy nginx_pool started.
Sep 28 00:44:18 Load-Balancer-Master haproxy[26434]: Proxy nginx_pool started.
Sep 28 00:44:18 Load-Balancer-Master systemd[1]: Started HAProxy Load Balancer.
Sep 28 22:12:57 Load-Balancer-Master haproxy[26434]: [WARNING] 270/004418 (26434) : Exiting Master pr
Sep 28 22:12:57 Load-Balancer-Master haproxy[26434]: [ALERT] 270/004418 (26434) : Current worker 2643
Sep 28 22:12:57 Load-Balancer-Master haproxy[26434]: [WARNING] 270/004418 (26434) : All workers exite
Sep 28 22:12:57 Load-Balancer-Master systemd[1]: Stopping HAProxy Load Balancer...
Sep 28 22:12:57 Load-Balancer-Master systemd[1]: haproxy.service: Main process exited, code=exited, s
Sep 28 22:12:57 Load-Balancer-Master systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 28 22:12:57 Load-Balancer-Master systemd[1]: Stopped HAProxy Load Balancer.
and in the /var/log/syslog I get this,
Sep 28 18:35:41 Load-Balancer-Master systemd[1]: Started Session 114 of user driveu.
Sep 28 18:42:57 Load-Balancer-Master systemd[1]: Stopping HAProxy Load Balancer...
Sep 28 18:42:57 Load-Balancer-Master systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Sep 28 18:42:57 Load-Balancer-Master systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 28 18:42:57 Load-Balancer-Master systemd[1]: Stopped HAProxy Load Balancer.
Sep 28 18:42:57 Load-Balancer-Master Keepalived_vrrp[26884]: Script `chk_haproxy` now returning 1
Sep 28 18:42:57 Load-Balancer-Master Keepalived_vrrp[26884]: VRRP_Script(chk_haproxy) failed (exited with status 1)
Sep 28 18:42:57 Load-Balancer-Master Keepalived_vrrp[26884]: (real) Changing effective priority from 103 to 101
But the notify script does not get called and the floating ip does not get assigned to the BACKUP instance. As I am really new to Keepalived could anyone please help me to fix this issue?
Update: I have solved this problem
The interface should be the private network and have to specify the private ips of the MASTER and the BACKUP using unicast_src_ip and unicast_peer. The Modified setting is here,
vrrp_script chk_haproxy {
# Requires keepalived-1.1.13
script "/usr/bin/pkill -0 haproxy" # cheaper than pidof
interval 2 # check every 2 seconds
weight 2 # add 2 points of priority if OK
}
vrrp_instance real {
interface ens10 # changed it from eth0
state MASTER
virtual_router_id 51
priority 101 # 101 on primary, 100 on secondary
unicast_src_ip 192.168.0.3
unicast_peer {
192.168.0.2
}
authentication {
auth_type PASS
auth_pass password
}
virtual_ipaddress {
11.23.10.19/32 dev eth0 label eth0:1
}
track_script {
chk_haproxy
}
notify "/etc/keepalived/master.sh"
#notify_backup "/etc/keepalived/master.sh"
#notify_fault "/etc/keepalived/master.sh"
}