Artemis HA and cluster not working - activemq-artemis

Question

Below are setting of artemis cluster (3 servers) in broker.xml
<!-- Clustering configuration -->
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<broadcast-period>5000</broadcast-period>
<jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
<jgroups-channel>active_broadcast_channel</jgroups-channel>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
<jgroups-channel>active_broadcast_channel</jgroups-channel>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<ha-policy>
<shared-store>
<colocated>
<backup-port-offset>100</backup-port-offset>
<backup-request-retries>-1</backup-request-retries>
<backup-request-retry-interval>2000</backup-request-retry-interval>
<max-backups>2</max-backups>
<request-backup>true</request-backup>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
<slave>
<scale-down/>
</slave>
</colocated>
</shared-store>
</ha-policy>
Cluster and ha configuration are same in all servers. The failover scenario which i am trying to understand and execute is as below.
Start broker1,broker2,broker3 in sequence mentioned. Here I can
see from admin UI that broker1 have backing up broker2 and broker3.
broker2 have backing of broker1. broker3 does not have any backup.
I wrote below program to connect to server
public static void main(final String[] args) throws Exception {
Connection connection = null;
InitialContext initialContext = null;
try {
Properties properties = new Properties();
properties.put(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory");
properties.put("connectionFactory.ConnectionFactory",
"(tcp://localhost:61616,tcp://localhost:61617,tcp://localhost:61618)?ha=true&retryInterval=1000&retryIntervalMultiplier=1.0&reconnectAttempts=-1");
properties.put("queue.queue/exampleQueue", "exampleQueue");
// Step 1. Create an initial context to perform the JNDI lookup.
initialContext = new InitialContext(properties);
ConnectionFactory cf = (ConnectionFactory) initialContext.lookup("ConnectionFactory");
// Step 2. Look-up the JMS Queue object from JNDI
Queue queue = (Queue) initialContext.lookup("queue/exampleQueue");
// Step 3. Create a JMS Connection
connection = cf.createConnection("admin", "admin");
// Step 4. Start the connection
connection.start();
// Step 5. Create a JMS session with AUTO_ACKNOWLEDGE mode
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Step 8. Create a text message
BytesMessage message = session.createBytesMessage();
message.setStringProperty(InfoSearchEngine.QUERY_ID_HEADER_PARAM, "123");
MessageConsumer consumer0 = session.createConsumer(queue);
// Step 9. Send the text message to the queue
while (true) {
try {
Thread.sleep(500);
// Step 7. Create a JMS message producer
MessageProducer messageProducer = session.createProducer(queue);
messageProducer.send(message);
System.out.println("Sent message: " + message.getBodyLength());
} catch (Exception e) {
System.out.println("Exception - " + e.getLocalizedMessage());
}
}
} finally {
if (connection != null) {
// Step 20. Be sure to close our JMS resources!
connection.close();
}
if (initialContext != null) {
// Step 21. Also close the initialContext!
initialContext.close();
}
}
}
if I shutdown broker1, program diverts to broker2 and runs fine. If
i shutdown broker2 then the program doest not connect to broker3.
I expected that broker3 should have started taking up the request since it was in cluster.

Answer 1

I can see from admin UI that broker1 have backing up broker2 and broker3. broker2 have backing of broker1. broker3 does not have any backup.
Failover in Artemis only works between a live and a backup. In your scenario broker1 is backing up broker2 so when you shutdown broker1 that means broker2 no longer has a backup so that when you shutdown broker2 no failover happens. You should specify <group-name> in your master and slave configurations so that your backups form in a more organized way so that this kind of situation doesn't happen.