ActiveMQ Artemis topology mistakes - activemq-artemis

I have a topologic issue working with ActiveMQ Artemis (version 2.27.1).
I work with 3 servers, which on every one there are 2 nodes – master and slave (lets call them A, B, C, A*, B*, C*).
I configured for each master node his replication as the slave node seats next to him (A -> B*, B -> C*, C -> A*).
Here is an example for A & B* broker.xml configuration:
Master broker.xml:
<connectors>
<connector name="Master_A">tcp://[HOSTNAME]:[A_MASTER_PORT]</connector>
<connector name="Master_B">tcp://[HOSTNAME]:[B_MASTER_PORT]</connector>
<connector name="Master_C">tcp://[HOSTNAME]:[C_MASTER_PORT]</connector>
<connector name="Slave_B">tcp://[HOSTNAME]:[B_SLAVE_PORT]</connector>
</connectors>
<cluster-connections>
<cluster-connection name="artemis-cluster">
<address></address>
<connector-ref>A_master_connector</connector-ref>
<check-period>30000</check-period>
<connection-ttl>60000</connection-ttl>
<min-large-message-size>102400</min-large-message-size>
<call-timeout>30000</call-timeout>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<initial-connect-attempts>-1</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<confirmation-window-size>1048576</confirmation-window-size>
<call-failover-timeout>-1</call-failover-timeout>
<notification-interval>1000</notification-interval>
<notification-attempts>2</notification-attempts>
<static-connectors allow-direct-connections-only="true">
<connector-ref>Master_B</connector-ref>
<connector-ref>Master_C</connector-ref>
<connector-ref>Slave_B</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
<vote-on-replication-failure>false</vote-on-replication-failure>
</master>
</replication>
</ha-policy>
and Slave broker.xml:
<connectors>
<connector name="Slave_B">tcp://[HOSTNAME]:[B_SLAVE_PORT]</connector>
<connector name="Master_A">tcp://[HOSTNAME]:[A_MASTER_PORT]</connector>
</connectors>
<cluster-connections>
<cluster-connection name="artemis-cluster">
<address></address>
<connector-ref>B_slave_connector</connector-ref>
<check-period>30000</check-period>
<connection-ttl>60000</connection-ttl>
<min-large-message-size>102400</min-large-message-size>
<call-timeout>30000</call-timeout>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<initial-connect-attempts>-1</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<confirmation-window-size>1048576</confirmation-window-size>
<call-failover-timeout>-1</call-failover-timeout>
<notification-interval>1000</notification-interval>
<notification-attempts>2</notification-attempts>
<static-connectors allow-direct-connections-only="true">
<connector-ref> Master_A</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<max-saved-replicated-journals-size>-1</max-saved-replicated-journals-size>
<restart-backup>false</restart-backup>
<allow-failback>true</allow-failback>
<vote-on-replication-failure>false</vote-on-replication-failure>
</slave>
</replication>
</ha-policy>
The problem is that I am starting those nodes by the recommended order from the [Artemis documentation][1] – all the masters together, and the all the slaves together. Even tough, the topology is not how I declared, master nodes are connecting to the wrong slave nodes...(that they actually not supposed to have connection to those nodes, because I configured the connections as only-direct-connections).

Your configuration isn't appropriate if you want to pair specific primary and backup together. The documentation discusses the necessary configuration:
Within a cluster, there are two ways that a backup server will locate a live server to replicate from, these are:
specifying a node group. You can specify a group of live servers that a backup server can connect to. This is done by configuring group-name in either the master or the slave element of the broker.xml. A Backup server will only connect to a live server that shares the same node group name
connecting to any live. This will be the behaviour if group-name is not configured allowing a backup server to connect to any live server
Currently you're aren't specifying group-name which means the backup will connect to any live broker.

Related

Make Artemis Slave Replication Use SSL

In Artemis when using replication to keep master/slave pairs synchronized the data will be replicated to the slave using a 'connection'.
I want to ensure this replication connection is encrypted. I suspect that this is done by using SSL on the connectors section of the broker.xml. However digging through the guides/official docs does not explicitly state how this is done. Yeah I can go waddling through source code and play with settings and try and sniff the traffic just thought asking here might be a bit easier.
Lets assume I have just a master/slave pair for now(I know not good for split brain but lets keep it simple for now) and will be using static connection lists as UDP is not allowed in my data center I have the following setup.
<connectors xmlns="urn:activemq:core">
<connector name="master">
tcp://master:61616?sslEnabled=true;keyStorePath=/d1/usr/dltuser/keystore/qcsp6ab2001.jks;keyStorePassword=1q2w3e4r;needClientAuth=true;trustStorePath=/d1/usr/dltuser/keystore/qcsp6ab2001_trust.jks;truststorepassword=1q2w3e4r
</connector>
<connector name="slave">
tcp://slave:61616?sslEnabled=true;keyStorePath=/d1/usr/dltuser/keystore/qcsp6ab2001.jks;keyStorePassword=1q2w3e4r;needClientAuth=true;trustStorePath=/d1/usr/dltuser/keystore/qcsp6ab2001_trust.jks;truststorepassword=1q2w3e4r
</connector>
</connectors>
<cluster-connections>
<cluster-connection name="amq-cluster">
<connector-ref>master</connector-ref>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.1</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<initial-connect-attempts>-1</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<forward-when-no-consumers>false</forward-when-no-consumers>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>master</connector-ref>
<connector-ref>slave</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
<!-- what master/slave group is this broker part of, master and slave must match -->
<group-name>nft-group-1</group-name>
<!-- does the broker initiate a quorum vote if connection to slave fails -->
<vote-on-replication-failure>true</vote-on-replication-failure>
<!-- how many votes should backup intiate when requesting a quorum?-->
<vote-retries>5</vote-retries>
<!-- how long should the broker wait between vote retries -->
<vote-retry-wait>5000</vote-retry-wait>
<vote-on-replication-failure>true</vote-on-replication-failure>
<cluster-name>amq-cluster</cluster-name>
</master>
</replication>
</ha-policy>
From my understanding the connectors will be used when forming the master slave pairs and then the replication will be done via SSL using the configuration from connectors section is this the case?
From my understanding the connectors will be used when forming the master slave pairs and then the replication will be done via SSL using the configuration from connectors section is this the case?
Yes, that is the case.

active-mq artemis springboot clustered topic load balancing (round robin) issue

After spending a lot of time in configuring and trying a lot of solutions to make Artemis work in a cluster mode like the local mode in a publish-subscribe (topic).
So, I 've prepared 3 consumers on different nodes and a producer that publish messages on only one node.
I expect that the 3 consumers receives their own copy of messages like described in here!
The problem is the cluster (Core Bridge) still round robin messages between the 3 nodes.
My project Github Repo
spring-boot-artemis-clustered-topic
Broker Cluster Config
<!-- Using STRICT is like setting the legacy forward-when-no-consumers
parameter to true-->
<!-- Using ON_DEMAND is like setting the legacy forward-when-no-consumers
parameter to false.-->
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
Consumers behavior
artemis-b1-b2-b3
In your ConnectionFactoryClusteredConfig.pubSubFactory() method, try moving factory.setPubSubDomain(true) after configurer.configure(factory, connectionFactory) as explained here: https://stackoverflow.com/a/44416121/832268.

Facing cluster testing issue with ActiveMQ Artemis

I have 2 instances of ActiveMQ Artemis , simply created with command
/.artemis create artemis/server1 and
/.artemis create artemis/server2
I am using linux ubantu.
here is broker.xml for server1:
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://localhost:61616</connector>
<!-- connector to the server1 -->
<connector name="server1-connector">tcp://localhost:61617</connector>
</connectors>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>server1-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
and here is broker.xml for server2:
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61617?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://localhost:61617</connector>
<!-- connector to the server0 -->
<connector name="server0-connector">tcp://localhost:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>server0-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
Also in server2, change in bootstrap.xml, changes web bind port
<web bind="http://localhost:8163" path="web">
I am testing it with StaticClusteredQueueExample and this example working file.
Now I am running the ActiveMQ Artemis JMeter Performance against my cluster, I am using JMeter Testing Examples which is here
Now when i am running point to point test with Jmeter is giving me near 50% errors rate (Aggregate Report in Jmeter) in consumer,
But where i am running only one node(any of server1 or server2) in ubantu system, it's working fine, 0% error rate(Aggregate Report in Jmeter).
Can you please help why i am getting 50% errors rate(Aggregate Report in Jmeter) when running multiple instances(nodes) with docker
The problem is that you're mixing one example (i.e. the JMeter example) with a cluster configuration (i.e. from the clustered-static-discovery example) that really isn't compatible.
The <message-load-balancing> of the cluster is STRICT which means messages will be load-balanced across the cluster regardless of the presence of consumers. Furthermore, the default <redistribution-delay> is -1 meaning the messages sent to the other nodes in the cluster due to the STRICT message-load-balancing type will stay on those nodes and will not be redistributed based on consumer demand.
The JMeter example was written with a single node in mind so it only send messages to and consumes messages from 1 node which means it will only receive back half of the messages that it sends as the other half will have been forwarded to the other node in the cluster due to the configuration.
If you change the <message-load-balancing> to ON_DEMAND you won't see any errors as all the messages will stay on the node where they were specifically sent which is also where the consumer will be connected.

Message redistribution on ArtemisMQ 2.x does not work

I would like to enable message redistribution on my 2-nodes cluster with static hosts. But it does not seem to work.
1) I have 10 producers that write to the queue "MyTestQueue" on node 1 (but no consumers).
2) I have 1 consumer on node 2 (but no producers) that consumes messages from node 2.
I expect that node 1 will redistribute the messages to node 2 where the consumer exists, but it does not. The message count on node 1 is still equal the amount of messages that was sent to node 1.
I have the following configuration in my broker.xml that sets forward-when-no-consumers to false.
I also have set redistribution-delay to a value of zero.
<jms xmlns="urn:activemq:jms">
<queue name="MyTestQueue"/>
</jms>
...
<cluster-connections>
<cluster-connection name="my-test-cluster">
<address>jms</address>
<connector-ref>server0-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<confirmation-window-size>1024</confirmation-window-size>
<static-connectors>
<connector-ref>server1-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
...
<address-settings>
<address-setting match="#">
<redelivery-delay>5000</redelivery-delay>
<redelivery-delay-multiplier>3</redelivery-delay-multiplier>
<max-redelivery-delay>10000</max-redelivery-delay>
<max-delivery-attempts>10</max-delivery-attempts>
<max-size-bytes>104857600</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<address-full-policy>PAGE</address-full-policy>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>
How can I get the message redistribution to work?
This might be related to a known issue. There is a situation on which the broker fails to load balance the messages if they don't contain the application properties field.
Could you, please, try with that?
Couple of things...
What version of Artemis are you using? Have you tried reproducing this with version 2.2.0?
What kind of client are you using (e.g. JMS, AMQP, STOMP, etc.)?
Do you have a reproducible test-case (e.g. a modified version of one of the examples shipped with Artemis)?
The configuration element <forward-when-no-consumers> is not valid in Artemis (although it was in older versions of HornetQ).
Remove the <address>jms</address> from cluster connection configuration - each cluster connection only applies to addresses that match the specified address. And make sure that you're using compatible client because messages from 1.x clients to 2.x cluster are lost when the are load balanced to nodes with matching consumers.
Here's official, working example of ActiveMQ Artemis configuration with symmetric cluster, on demand load balancing and message redistribution

How to configure standalone HornetQ along with EAP 6.3 or Jboss 7 for durable JMS subscription?

I would like to send JMS messages from one Jboss server to another but through a standalone HornetQ server.
This way I can have messages delivered later in case of the destination server crash (provided durable subscriptions).
However I have already messages routed internally at each Jboss. I would like a configuration that will not conflict with those.
The topology of the desired solution is visualised on the diagram.
How to achieve this configuration?
Lets start from configuring the standalone HornetQ. You download the standalone server from their download page.
Next you have to configure the topic. Please add your topic in %HORNETQ-HOME%\config\stand-alone\non-clustered\hornetq-jms.xml file:
<topic name="Topic1">
<entry name="java:/topic/Topic1"/>
</topic>
You probably want to test the configuration on one machine first, so I recommend changing the port at which HornetQ will be listening for messages from 5455 to 5456.
Please edit %HORNETQ-HOME%\config\stand-alone\non-clustered\hornetq-configuration.xml file to change these ports. You also want to be able to register durable subscribers, so add these two lines to the <security-setting match="#"> element in the same file:
<permission type="createDurableQueue" roles="guest"/>
<permission type="deleteDurableQueue" roles="guest"/>
Then start the standalone HornetQ by running %HORNETQ-HOME%\run.bat.
First we gonna see how to send a message to this newly created topic. For this we need a designated connection factory on the Jboss Server 1. In the jboss:domain:messaging subsystem of %JBOSS-HOME1%\standalone\configuration\standalone-full.xml, please add a new pooled connection factory:
<pooled-connection-factory name="StandaloneHornetQConnectionFactory">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="standalone-hornetq-connector"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/StandaloneHornetQConnectionFactory"/>
</entries>
</pooled-connection-factory>
From now on you have to use this connection factory when you want to send a message to Topic 1. This is usually done by dependency injection:
#Resource(lookup = "java:jboss/exported/jms/StandaloneHornetQConnectionFactory")
private ConnectionFactory connectionFactory;
As you can see above we referenced standalone-hornetq-connector but don't have one yet. Lets create it by adding another netty connector into <connectors>:
<connectors>
<netty-connector name="standalone-hornetq-connector" socket-binding="standalone-hornetq-socket"/>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
As you can see we need standalone-hornetq-socket socket binding. Lets create it in <socket-binding-group> subelement:
<outbound-socket-binding name="standalone-hornetq-socket">
<remote-destination host="localhost" port="5446"/>
</outbound-socket-binding>
As you can see, this is an outbound socket binding that will be used to send messages to our HornetQ Standalone server that is listening on the 5446 port. This configuration is enough on Jboss Server 1 to send messages to Jboss Server 2 via the Standalone HornetQ server.
To be able to receive the messages on Jboss Server 2, we have to once again repeat above configuration in the %JBOSS-HOME1%\standalone\configuration\standalone-full.xml. However this time we offset the ports of the Jboss Server 2 by port-offset:3, to be able to work on the same machine:
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:3}">
This step is not nesessary if you will be putting the servers in the separate machines (If you do, please change localhost accordingly ;) ).
Now we can create MDB that will be a durable subscriber of the Topic 1.
#MessageDriven(name = "MyDurableSubscriber", activationConfig = {
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
#ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
#ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/topic/Topic1"),
#ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"),
#ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "Topic1Subscription"),
#ActivationConfigProperty(propertyName = "clientId", propertyValue = "MySubscriber"),
})
#ResourceAdapter("StandaloneHornetQConnectionFactory")
public class MyDurableSubscriber implements MessageListener {
#Override
public void onMessage(Message message) {
// ...
}
}
The #ResourceAdapter("StandaloneHornetQConnectionFactory") line is the most important, because by default all MDBs are using hornetq-ra resource adapter to subscribe (local subscription). The ResourceAdapter annotation is from org.jboss.ejb3.annotation package
and you can make this class available to you via maven dependency:
<dependency>
<groupId>org.jboss.ejb3</groupId>
<artifactId>jboss-ejb3-ext-api</artifactId>
<version>2.1.0</version>
<scope>provided</scope>
</dependency>
With these all set you can enjoy durable subscriptions with "star" topology of your servers.