Artemis HA configuration for core bridges - activemq-artemis

Question

We have 4 servers in two shared disk HA pairs, with core bridges between them. The core bridge configuration and the connectors they use (sms and sms1b) are identical on all 4 servers. The only differences being master vs slave ha, and the host names in the other fields (acceptor, artemis and node0 connector, name)
In testing we found the bridge works perfectly when the two lives are up, but sometimes when shutting down a live server, the backup never opens a consumer for the bridge.
Is this the intended way to configure a pair of HA servers with a core bridge, or is the backup server configured wrong?
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>ba-artms3.example.com</name>
<security-enabled>false</security-enabled>
<persistence-enabled>true</persistence-enabled>
<paging-directory>/data/ba_artemis/msg-sms1/paging</paging-directory>
<bindings-directory>/data/ba_artemis/msg-sms1/bindings</bindings-directory>
<journal-directory>/data/ba_artemis/msg-sms1/journal</journal-directory>
<large-messages-directory>/data/ba_artemis/msg-sms1/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>132000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<connectors>
<connector name="artemis">tcp://ba-artms3.example.com:2539</connector>
<connector name = "node0">tcp://ba-artms4.example.com:2539</connector>
<connector name="sms1">(tcp://ba-artms3.example.com:61616,tcp://ba-artms4.example.com:61616)</connector>
<connector name="sms1b">(tcp://ba-artms9.example.com:61616,tcp://ba-artms10.example.com:61616)</connector>
</connectors>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>620000</page-sync-timeout>
<acceptors>
<acceptor name="artemis">tcp://ba-artms3.example.com:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;</acceptor>
<acceptor name="cluster">tcp://ba-artms3.example.com:2539?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE;useEpoll=true</acceptor>
<acceptor name="amqp">tcp://ba-artms3.example.com:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>
</acceptors>
<cluster-user>msg-sms1-cluster</cluster-user>
<cluster-password>redacted</cluster-password>
<cluster-connections>
<cluster-connection name="msg-sms1">
<connector-ref>artemis</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>0</max-hops>
<static-connectors>
<connector-ref>node0</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<shared-store>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
</shared-store>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<xi:include href="${configDir}/addresses.xml"/>
<bridges>
<bridge name="sms1_forwarder">
<queue-name>UpdateOutboundForward_0</queue-name>
<forwarding-address>UpdateOutbound</forwarding-address>
<ha>true</ha>
<failover-on-server-shutdown>true</failover-on-server-shutdown>
<user>rave</user>
<password>redacted</password>
<static-connectors>
<connector-ref>sms1</connector-ref>
</static-connectors>
</bridge>
<bridge name="sms1b_forwarder">
<queue-name>UpdateOutboundForward_1</queue-name>
<forwarding-address>UpdateOutbound</forwarding-address>
<ha>true</ha>
<failover-on-server-shutdown>true</failover-on-server-shutdown>
<user>rave</user>
<password>redacted</password>
<static-connectors>
<connector-ref>sms1b</connector-ref>
</static-connectors>
</bridge>
</bridges>
</core>
</configuration>
Keep in mind that the acceptor on port 2539 is specifically used for clustering. There are 4 servers total: ba-artms3 (live), ba-artms4 (slave) & ba-artms9 (live), ba-artms10 (slave).

Answer 1

Your configuration looks essentially correct, but it's hard to tell with so many moving pieces - especially the extra acceptor for clustering. I've seen folks do that before, but it's not something I've ever tested or recommended so I'm not sure how it would function in practice. Theoretically it would be fine, but there are always complicating factors many of which are subtle.
It would be worth simplifying your configuration to only what's absolutely necessary to reproduce the problem. For example, configure just 1 bridge on 1 server connecting to a live/backup pair with all the brokers on your local machine on unique ports (i.e. no docker). Once you have that working you can keeping adding complexity and testing as you go to see where things break down (assuming they do).

Answer 2

The solution that worked for me was using one connector per server instead of the bracket and comma syntax:
<connectors>
<connector name="artemis">tcp://ba-artms3.example.com:2539</connector>
<connector name = "node0">tcp://ba-artms4.example.com:2539</connector>
<connector name="sms1_1">tcp://ba-artms3.example.com:2539</connector>
<connector name="sms1_2">tcp://ba-artms4.example.com:2539</connector>
<connector name="sms1b_1">tcp://ba-artms9.example.com:2539</connector>
<connector name="sms1b_2">tcp://ba-artms10.example.com:2539</connector>
</connectors>
And then listing both of them in the list:
<bridge name="sms1b_forwarder">
<queue-name>UpdateOutboundForward_1</queue-name>
<forwarding-address>UpdateOutbound</forwarding-address>
<ha>true</ha>
<failover-on-server-shutdown>true</failover-on-server-shutdown>
<user>rave</user>
<password>redacted</password>
<static-connectors>
<connector-ref>sms1b_1</connector-ref>
<connector-ref>sms1b_2</connector-ref>
</static-connectors>
</bridge>
After that in the log instead of seeing:
2021-07-20 09:13:51,809 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge BridgeImpl#5886a172 [name=sms1b_forwarder, queue=QueueImpl[name=UpdateOutboundForward_1, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=13be52e3-cb0f-11eb-a851-000c29d5fa03], temp=false]#56414412 targetConnector=ServerLocatorImpl (identity=Bridge sms1b_forwarder) [initialConnectors=[TransportConfiguration(name=sms1b, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ba-artms9-example-com], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
2021-07-20 09:13:51,880 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge BridgeImpl#983e222 [name=sms1_forwarder, queue=QueueImpl[name=UpdateOutboundForward_0, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=13be52e3-cb0f-11eb-a851-000c29d5fa03], temp=false]#7440d62 targetConnector=ServerLocatorImpl (identity=Bridge sms1_forwarder) [initialConnectors=[TransportConfiguration(name=sms1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ba-artms3-example-com], discoveryGroupConfiguration=null]] is connected
I was seeing:
2021-07-20 09:30:40,333 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge BridgeImpl#339a572d [name=sms1b_forwarder, queue=QueueImpl[name=UpdateOutboundForward_1, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=13be52e3-cb0f-11eb-a851-000c29d5fa03], temp=false]#3d5daf2e targetConnector=ServerLocatorImpl (identity=Bridge sms1b_forwarder) [initialConnectors=[TransportConfiguration(name=sms1b_1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ba-artms9-example-com, TransportConfiguration(name=sms1b_2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ba-artms10-example-com], discoveryGroupConfiguration=null]] is connected
2021-07-20 09:30:42,206 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge BridgeImpl#47c22520 [name=sms1_forwarder, queue=QueueImpl[name=UpdateOutboundForward_0, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=13be52e3-cb0f-11eb-a851-000c29d5fa03], temp=false]#4905c3a8 targetConnector=ServerLocatorImpl (identity=Bridge sms1_forwarder) [initialConnectors=[TransportConfiguration(name=sms1_1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=2539&host=ba-artms3-example-com, TransportConfiguration(name=sms1_2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=2539&host=ba-artms4-example-com], discoveryGroupConfiguration=null]] is connected
Note both servers now showing up in the initialConnectors.