How to make keepalived respond on failover on HAProxy? - haproxy

I am using HAProxy as load balancer for my application and to make it highly available I am using keepalive service and floating ip address concept. But whenever my primary load balancer server gets down, by removing it from network or turning it off, my all services go down instead of making secondary load balancer server available.
My keepalived.conf for master server is,
global_defs
{
# Keepalived process identifier
lvs_id haproxy_DH
}
# Script used to check if HAProxy is running
vrrp_script check_haproxy
{
script "pidof haproxy"
interval 2
weight 2
}
# Virtual interface
vrrp_instance VI_01
{
state MASTER
interface eno16777984 #here eth0 is the name of network interface
virtual_router_id 51
priority 101
# The virtual ip address shared between the two loadbalancers
virtual_ipaddress {
172.16.231.162
}
track_script {
check_haproxy
}
}
For backup server it is like,
global_defs
{
# Keepalived process identifier
lvs_id haproxy_DH_passive
}
# Script used to check if HAProxy is running
vrrp_script check_haproxy
{
script "pidof haproxy"
interval 2
weight 2
}
# Virtual interface
vrrp_instance VI_01
{
state BACKUP
interface eno16777984 #here eth0 is the name of network interface
virtual_router_id 51
priority 100
# The virtual ip address shared between the two loadbalancers
virtual_ipaddress {
172.16.231.162
}
track_script {
check_haproxy
}
}
The virtual IP address is assigned and working when both load balancers are up. But whenever machine goes down, my service also goes down. I am using CentOS7, Please help.

Use this,
global_defs {
router_id ovp_vrrp
}
vrrp_script haproxy_check {
script "killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance OCP_EXT {
interface ens192
virtual_router_id 51
priority 100
state MASTER
virtual_ipaddress {
10.19.114.231 dev ens192
}
track_script {
haproxy_check
}
authentication {
auth_type PASS
auth_pass 1cee4b6e-2cdc-48bf-83b2-01a96d1593e4
}
}
more info: read here, https://www.openshift.com/blog/haproxy-highly-available-keepalived

Related

Connect two machines in AKKA remotely ,connection refused

I'm new to akka and wanted to connect two PC using akka remotely just to run some code in both as (2 actors). I had tried the example in akka doc. But what I really do is to add the 2 IP addresses into config file I always get this error?
First machine give me this error:
[info] [ERROR] [11/20/2018 13:58:48.833]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.remote.artery.Association(akka://ClusterSystem)] Outbound
control stream to [akka://ClusterSystem#192.168.1.2:2552] failed.
Restarting it. Handshake with [akka://ClusterSystem#192.168.1.2:2552]
did not complete within 20000 ms
(akka.remote.artery.OutboundHandshake$HandshakeTimeoutException:
Handshake with [akka://ClusterSystem#192.168.1.2:2552] did not
complete within 20000 ms)
And second machine:
Exception in thread "main"
akka.remote.RemoteTransportException: Failed to bind TCP to
[192.168.1.3:2552] due to: Bind failed because of
java.net.BindException: Cannot assign requested address: bind
Config file content :
akka {
actor {
provider = cluster
}
remote {
artery {
enabled = on
transport = tcp
canonical.hostname = "192.168.1.3"
canonical.port = 0
}
}
cluster {
seed-nodes = [
"akka://ClusterSystem#192.168.1.3:2552",
"akka://ClusterSystem#192.168.1.2:2552"]
# auto downing is NOT safe for production deployments.
# you may want to use it during development, read more about it in the docs.
auto-down-unreachable-after = 120s
}
}
# Enable metrics extension in akka-cluster-metrics.
akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"]
# Sigar native library extract location during tests.
# Note: use per-jvm-instance folder when running multiple jvm on one host.
akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native
First of all, you don't need to add cluster configuration for AKKA remoting. Both the PCs or nodes should be enabled remoting with a concrete port instead of "0" that way you know which port to connect.
Have below configurations
PC1
akka {
actor {
provider = remote
}
remote {
artery {
enabled = on
transport = tcp
canonical.hostname = "192.168.1.3"
canonical.port = 19000
}
}
}
PC2
akka {
actor {
provider = remote
}
remote {
artery {
enabled = on
transport = tcp
canonical.hostname = "192.168.1.4"
canonical.port = 18000
}
}
}
Use below actor path to connect any actor in remote from PC1 to PC2
akka://<PC2-ActorSystem>#192.168.1.4:18000/user/<actor deployed in PC2>
Use below actor path to connect from PC2 to PC1
akka://<PC2-ActorSystem>#192.168.1.3:19000/user/<actor deployed in PC1>
Port numbers and IP address are samples.

Obtaining Azure VM scale set private IP addresses through Azure REST API

Can you get a list of Azure VM scale set instance private IP addresses through the Azure REST API?
It seems that Microsoft does not publish the VMSS IP configuration objects under the normal methods for retrieving a list of "ipConfigurations".
Here are some relevant API doc pages:
https://learn.microsoft.com/en-us/rest/api/compute/virtualmachinescalesets/listall
https://learn.microsoft.com/en-us/rest/api/compute/virtualmachinescalesets/get
https://learn.microsoft.com/en-us/rest/api/compute/virtualmachines/listall
In particular, this one only gives you the IP configuration of VMs, not VMSSes:
https://learn.microsoft.com/en-us/rest/api/virtualnetwork/networkinterfaces/listall
Here's how to get a list of private IP addresses for VMs and VMSS instances through Ruby:
require 'openssl'
require 'azure_mgmt_network'
require 'azure_mgmt_compute'
require 'awesome_print'
options = {
tenant_id: '<tenant_id>',
client_id: '<client_id>',
client_secret: '<client_secret>',
subscription_id: '<subscription_id>'
}
def net_interface_to_ip_mapping(client)
network_interfaces = client.network_interfaces.list_all
pairs = network_interfaces.collect { |ni| [ni.id.split('/').last, ni.ip_configurations.collect { |ip| ip.private_ipaddress }.flatten.compact[0] ] }
[network_interfaces, pairs]
end
def net_interface_to_vm(ni)
interface_vm_set = ni.collect { |prof| [prof.id, prof.virtual_machine, prof.ip_configurations.collect(&:id)] }
ipconf_to_host = interface_vm_set.collect { |x| [x[2][0], x[1]&.id&.split('/')&.last] }.to_h
conf_ip_map = ni.collect(&:ip_configurations).flatten.compact.collect { |ipconf| [ipconf&.id, ipconf&.private_ipaddress] }.to_h
[ipconf_to_host, conf_ip_map]
end
puts "*** Network Interfaces"
puts
client = Azure::Network::Profiles::Latest::Mgmt::Client.new(options)
ni, pairs = net_interface_to_ip_mapping(client)
pairs.to_h.each do |ni, ip|
puts " #{ni}: #{ip}"
end
puts
puts "*** Virtual Machines"
puts
ipconf_to_host, conf_ip_map = net_interface_to_vm(ni)
ipconf_to_host.each do |ipconf, host|
ni_name = ipconf.split('/')[-3]
puts " #{host || '# ' + ni_name} - #{conf_ip_map[ipconf]}"
end
puts
puts "*** Virtual Machine Scale Sets"
puts
vns = client.virtual_networks.list_all
vns.each do |vn|
resource_group = vn.id.split('/')[4]
puts
vn_details = client.virtual_networks.get(resource_group, vn.name, expand: 'subnets/ipConfigurations')
ip_configs = vn_details&.subnets&.collect { |subnet| subnet&.ip_configurations&.collect { |ip| [ip&.id, ip&.name, ip&.private_ipaddress] } }.compact
vmss_ipconf = ip_configs.collect { |subnet| subnet.select { |ipconf| ipconf[0].include?('/virtualMachineScaleSets/') } }
vmss_ipconf.each do |subnet|
subnet.each do |ipconf|
vmss_name = ipconf[0].split('/')[8]
vmss_instance = ipconf[0].split('/')[10]
puts "#{vmss_name} ##{vmss_instance} - #{ipconf[2]}"
end
end
end
Looking at the Azure CLI, there is az vmss nic list which returns all network interfaces in a virtual machine scale set. Looking at the results, there is
{
"dnsSettings": {
...
},
"ipConfigurations": [
{
privateIpAddress: "..."
}
]
}
You can use the --query syntax to get all private IPs.
az vmss nic list -g <resource_group> --vmss-name <vmss_name> --query [].{ip:ipConfigurations[0].privateIpAddress} -o tsv
you can get VM hostnames that will resolve to IPs thanks to Azure DNS
$ curl -H "Authorization: Bearer $JWT_TOCKEN" -sf https://management.azure.com/subscriptions/${subscription_id}/resourceGroups/${resourc_group}/providers/Microsoft.Compute/virtualMachineScaleSets/${scale_set}/virtualMachines?api-version=2018-10-01 | jq '.value[].properties.osProfile.computerName'
"influx-meta000000"
"influx-meta000001"
$ getent hosts influx-meta000001
10.120.10.7 influx-meta000001.l55qt5nuiezudgvyxzyvtbihmf.gx.internal.cloudapp.net

K8s pods affinity & anti-affinity, soft (preferredDuringScheduling) not respected in 1.4?

I am experimenting with K8s 1.4 pod affinity-antiaffinity. I am trying to get K8s to cluster together pods of the same service on the same node as much as possible (i.e. only go to next node if not possible to put more on the node where the service is already in). To do so, I setup:
A hard (requiredDuringScheduling) anti-affinity to exclude running where a diffrent service is already running (pod_label_xyz not in [value-a])
A soft (preferredDuringScheduling) affinity to try to run where the same service (pod_label_xyz in [value-a]) - weight 100
A soft (preferredDuringScheduling) anti-affinity to try not to run where the same service is not already running (pod_label_xyz not present) - weight 100
When there is 5 nodes and 3 services pod_label_xyz & (value-a, value-b, value-c) with 1 pod each created with a replication controller, the first pods get scheduled properly and when scaling up any of them, the 1st hard rule is respected by K8s. However, the 2nd and 3rd (which is actually redundant of the 2nd) is not respected. I see that when I scale up K8s tries to push pods to empty node (not used by any other service) even though there is capacity to schedule more where the service is already running. In fact if I scale up even more, new pods get created on the original node as well as the new (previously unused nodes).
Please advise if I am missing something
Thank you
Here is the annotation i used
scheduler.alpha.kubernetes.io/affinity: >
{
"podAffinity":{
"preferredDuringSchedulingIgnoredDuringExecution":[
{
"weight":100,
"podAffinityTerm":{
"labelSelector":{
"matchExpressions":[
{
"key":"pod_label_xyz",
"operator":"Exists"
},
{
"key":"pod_label_xyz",
"operator":"In",
"values":[
"value-a"
]
}
]
},
"namespaces":[
"sspni-882-frj"
],
"topologyKey":"kubernetes.io/hostname"
}
}
]
},
"podAntiAffinity":{
"requiredDuringSchedulingIgnoredDuringExecution":[
{
"labelSelector":{
"matchExpressions":[
{
"key":"pod_label_xyz",
"operator":"Exists"
},
{
"key":"pod_label_xyz",
"operator":"NotIn",
"values":[
"value-a"
]
}
]
},
"namespaces":[
"sspni-882-frj"
],
"topologyKey":"kubernetes.io/hostname"
}
],
"preferredDuringSchedulingIgnoredDuringExecution":[
{
"weight":100,
"podAffinityTerm":{
"labelSelector":{
"matchExpressions":[
{
"key":"pod_label_xyz",
"operator":"DoesNotExist"
}
]
},
"namespaces":[
"sspni-882-frj"
],
"topologyKey":"kubernetes.io/hostname"
}
}
]
}
}

Multiple control conditions validation in puppet

I need to validate agent like memory, processor, and port connectivity before running my other manifest files. So I've created a manifest like following by keeping global facts with and statement and exec resource.
class vc {
#Validateing Infra Before applying chnages
if $::facts['memorysize'] >= '4.00 GiB'and $::facts['processorcount'] >= 2 and Exec['port_connectivity'] {
notify { "Infra validated" : }
include vc::configs
}
else {
notify { "Infra not meeting requirements" : }
}
# Checking port connecitivity to puppet master
exec { 'port_connectivity':
command => 'New-Item c:\debug.txt -type file -Force',
unless => 'if((New-Object System.Net.Sockets.TcpClient ("linux-NAS-storage.com",6163)).connected -eq $true) { exit 1 }',
provider => powershell,
}
}
my theme is puppet should only execute if this if $::facts['memorysize'] >= '4.00 GiB'and $::facts['processorcount'] >= 2 and Exec['port_connectivity'] condition was successful. If exec command was successful and facter returns true then only it should execute, but following manifest executing individually without checking whether that exec statement true or not. My main purpose I need to validate ports before running puppet manifest. Can any please help

Keepalived vrrp_script does not failover

I have 2 nodes with keepalived and haproxy services (CentOS7).
If I'm shutdown one node all working fine. But I want to failover the VIPS if haproxy is down.
This is 1st node config:
vrrp_script ha_check {
script "/etc/keepalived/haproxy_check"
interval 2
weight 21
}
vrrp_instance VI_1 {
state MASTER
interface eno16777984
virtual_router_id 151
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass 11111
}
virtual_ipaddress {
10.0.100.233
}
smtp_alert
track_script {
ha_check
}
}
2nd node:
vrrp_script ha_check {
script "/etc/keepalived/haproxy_check"
interval 2
fall 2
rise 2
timeout 1
weight 2
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777984
virtual_router_id 151
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 11111
}
virtual_ipaddress {
10.0.100.233
}
smtp_alert
track_script {
ha_check
}
}
cat /etc/keepalived/haproxy_check
systemctl status haproxy | grep "inactive"
When I stop haproxy it still does not failover the VIPs to the next
host.
[root#cks-hatest1 keepalived]# tail /var/log/messages
Nov 30 10:35:24 cks-hatest1 Keepalived_vrrp[5891]: VRRP_Script(ha_check) failed
Nov 30 10:35:33 cks-hatest1 systemd: Started HAProxy Load Balancer.
Nov 30 10:35:45 cks-hatest1 systemd: Stopping HAProxy Load Balancer...
Nov 30 10:35:45 cks-hatest1 systemd: Stopped HAProxy Load Balancer.
Nov 30 10:35:46 cks-hatest1 Keepalived_vrrp[5891]: VRRP_Script(ha_check) succeeded
What I am doing wrong? Thank you in advance!
In your script you are checking if
systemctl status haproxy
contains keyword "inactive". Is that the value you get when you stop haproxy service manually?
As soon as haproxy service is stopped your logs contains it is started again. Can you verify that?
Also, try with replacing the script as
script "killall -0 haproxy"
It's easy. Try this for example:
vrrp_script check_haproxy {
script "pidof haproxy"
interval 2
weight 2
}
In the end of config you should add following part too:
track_script {
check_haproxy
}