Large message lost for divert - activemq-artemis

Question

I have following queue setup in cluster:
server1 running on 61616
server2 running on 61617
I start two consumer with two different available master server
java -jar AmqJmsConsume.jar -duration 5 -queue IT-InputQueue -stats -log
/tmp/artemis/4 -verify -commitdelay 300 -url 'tcp://localhost:61616'
java -jar AmqJmsConsume.jar -duration 5 -queue IT-InputQueue -stats -log
/tmp/artemis/4 -verify -commitdelay 300 -url 'tcp://localhost:61617'
and run producer to server1 as below
java -jar AmqJmsProducer.jar -topic RFTopic -stats -log /tmp/artemis/4 -id
-count 500 -n 500 -ttl 3600000 -url 'tcp://localhost:61616' -outliers 100
-outliersize 500k
both consumer gets few message but after below exception server1 clears
queue with almost 50% of messages but rest of messages are stuck with
internal.sf.cluster queue and not received by any consumer.
java.lang.IllegalStateException: no queueIDs defined
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.beforeForward(ClusterConnectionBridge.java:182) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:653) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueImpl.java:3346) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2606) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2300(QueueImpl.java:117) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:3613) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_211]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.9.0.jar:2.9.0]
2019-07-29 10:21:38,503 WARN [org.apache.activemq.artemis.core.server.impl.QueueImpl] null: java.util.NoSuchElementException
at org.apache.activemq.artemis.utils.collections.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:172) [artemis-commons-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2627) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2300(QueueImpl.java:117) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:3613) [artemis-server-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.9.0.jar:2.9.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_211]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.9.0.jar:2.9.0]
server1-master broker
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<persistence-enabled>true</persistence-enabled>
<thread-pool-max-size>400</thread-pool-max-size>
<journal-type>NIO</journal-type>
<paging-directory>${data.dir}/paging</paging-directory>
<bindings-directory>${data.dir}/bindings</bindings-directory>
<journal-directory>${data.dir}/journal</journal-directory>
<large-messages-directory>${data.dir}/large-messages
</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>20</journal-min-files>
<journal-pool-files>20</journal-pool-files>
<journal-file-size>10M</journal-file-size>
<journal-compact-min-files>30</journal-compact-min-files>
<journal-buffer-timeout>23480000</journal-buffer-timeout>
<!-- When using ASYNCIO, this will determine the writing queue depth for
libaio. -->
<journal-max-io>1</journal-max-io>
<!-- You can verify the network health of a particular NIC by specifying
the <network-check-NIC> element. <network-check-NIC>theNicName</network-check-NIC> -->
<!-- Use this to use an HTTP server to validate the network <network-check-URL-list>http://www.apache.org</network-check-URL-list> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs it should
accept IPV6 Warning: Make sure you understand your network topology as this
is meant to validate if your network is valid. Using IPs that could eventually
disappear or be partially visible may defeat the purpose. You can use a list
of multiple IPs, and if any successful ping will make the server OK to continue
running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<!-- how often we are looking for how many bytes are being used on the
disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the
connection in certain protocols that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>LOG</critical-analyzer-policy>
<transaction-timeout>1800000</transaction-timeout>
<!-- the system will enter into page mode once you hit this limit. This
is an estimate in bytes of how much the messages are using in memory The
system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your
needs. <global-max-size>100Mb</global-max-size> -->
<connectors>
<connector name="netty-connector">tcp://localhost:61616</connector>
</connectors>
<!-- Acceptors -->
<acceptors>
<acceptor name="netty-acceptor">tcp://localhost:61616?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.</acceptor>
</acceptors>
<ha-policy>
<shared-store>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
</shared-store>
</ha-policy>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<broadcast-period>5000</broadcast-period>
<jgroups-file>idsk-jgroups.xml</jgroups-file>
<jgroups-channel>persistence-fs</jgroups-channel>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<jgroups-file>idsk-jgroups.xml</jgroups-file>
<jgroups-channel>persistence-fs</jgroups-channel>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group" />
</cluster-connection>
</cluster-connections>
<cluster-user>admin</cluster-user>
<cluster-password>admin</cluster-password>
<diverts>
<divert name="RF-Transform">
<routing-name>RFeeds-Transform</routing-name>
<address>RFTopic</address>
<forwarding-address>IT-InputQueue</forwarding-address>
<exclusive>false</exclusive>
</divert>
<divert name="RF-Output">
<routing-name>RFeeds-Output</routing-name>
<address>RFTopic</address>
<forwarding-address>T1-InputQueue</forwarding-address>
<exclusive>false</exclusive>
</divert>
</diverts>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq" />
<permission type="deleteNonDurableQueue" roles="amq" />
<permission type="createDurableQueue" roles="amq" />
<permission type="deleteDurableQueue" roles="amq" />
<permission type="createAddress" roles="amq" />
<permission type="deleteAddress" roles="amq" />
<permission type="consume" roles="amq" />
<permission type="browse" roles="amq" />
<permission type="send" roles="amq" />
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq" />
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be
auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>60000</redelivery-delay>
<max-delivery-attempts>5</max-delivery-attempts>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>50485760</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
</address-setting>
<!--default for catch all -->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>60000</redelivery-delay>
<max-delivery-attempts>5</max-delivery-attempts>
<redistribution-delay>10000</redistribution-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>50485760</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
<address name="IT-InputQueue">
<anycast>
<queue name="IT-InputQueue" />
</anycast>
</address>
<address name="T1-InputQueue">
<anycast>
<queue name="T1-InputQueue" />
</anycast>
</address>
<address name="RFTopic">
<multicast />
</address>
</addresses>
</core>
</configuration>
server-master broker
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<persistence-enabled>true</persistence-enabled>
<thread-pool-max-size>400</thread-pool-max-size>
<journal-type>NIO</journal-type>
<paging-directory>${data.dir}/paging</paging-directory>
<bindings-directory>${data.dir}/bindings</bindings-directory>
<journal-directory>${data.dir}/journal</journal-directory>
<large-messages-directory>${data.dir}/large-messages
</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>20</journal-min-files>
<journal-pool-files>20</journal-pool-files>
<journal-file-size>10M</journal-file-size>
<journal-compact-min-files>30</journal-compact-min-files>
<journal-buffer-timeout>23480000</journal-buffer-timeout>
<!-- When using ASYNCIO, this will determine the writing queue depth for
libaio. -->
<journal-max-io>1</journal-max-io>
<!-- You can verify the network health of a particular NIC by specifying
the <network-check-NIC> element. <network-check-NIC>theNicName</network-check-NIC> -->
<!-- Use this to use an HTTP server to validate the network <network-check-URL-list>http://www.apache.org</network-check-URL-list> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs it should
accept IPV6 Warning: Make sure you understand your network topology as this
is meant to validate if your network is valid. Using IPs that could eventually
disappear or be partially visible may defeat the purpose. You can use a list
of multiple IPs, and if any successful ping will make the server OK to continue
running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<!-- how often we are looking for how many bytes are being used on the
disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the
connection in certain protocols that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>LOG</critical-analyzer-policy>
<transaction-timeout>1800000</transaction-timeout>
<!-- the system will enter into page mode once you hit this limit. This
is an estimate in bytes of how much the messages are using in memory The
system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your
needs. <global-max-size>100Mb</global-max-size> -->
<connectors>
<connector name="netty-connector">tcp://localhost:61617</connector>
</connectors>
<!-- Acceptors -->
<acceptors>
<acceptor name="netty-acceptor">tcp://localhost:61617?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.</acceptor>
</acceptors>
<ha-policy>
<shared-store>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
</shared-store>
</ha-policy>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<broadcast-period>5000</broadcast-period>
<jgroups-file>idsk-jgroups.xml</jgroups-file>
<jgroups-channel>persistence-fs</jgroups-channel>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<jgroups-file>idsk-jgroups.xml</jgroups-file>
<jgroups-channel>persistence-fs</jgroups-channel>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group" />
</cluster-connection>
</cluster-connections>
<cluster-user>admin</cluster-user>
<cluster-password>admin</cluster-password>
<diverts>
<divert name="RF-Transform">
<routing-name>RFeeds-Transform</routing-name>
<address>RFTopic</address>
<forwarding-address>IT-InputQueue</forwarding-address>
<exclusive>false</exclusive>
</divert>
<divert name="RF-Output">
<routing-name>RFeeds-Output</routing-name>
<address>RFTopic</address>
<forwarding-address>T1-InputQueue</forwarding-address>
<exclusive>false</exclusive>
</divert>
</diverts>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq" />
<permission type="deleteNonDurableQueue" roles="amq" />
<permission type="createDurableQueue" roles="amq" />
<permission type="deleteDurableQueue" roles="amq" />
<permission type="createAddress" roles="amq" />
<permission type="deleteAddress" roles="amq" />
<permission type="consume" roles="amq" />
<permission type="browse" roles="amq" />
<permission type="send" roles="amq" />
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq" />
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be
auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>60000</redelivery-delay>
<max-delivery-attempts>5</max-delivery-attempts>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>50485760</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
</address-setting>
<!--default for catch all -->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>60000</redelivery-delay>
<max-delivery-attempts>5</max-delivery-attempts>
<redistribution-delay>10000</redistribution-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>50485760</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
<address name="IT-InputQueue">
<anycast>
<queue name="IT-InputQueue" />
</anycast>
</address>
<address name="T1-InputQueue">
<anycast>
<queue name="T1-InputQueue" />
</anycast>
</address>
<address name="RFTopic">
<multicast />
</address>
</addresses>
</core>
</configuration>
My Analysis:
I have gone thru the code of artemis-server module and below are my findings.
The org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl. route(message,context,direct,rejectDuplicates,bindingMove) method clean the internal properties from the Message. Is there any reason to cleanup those properties. As I am getting above error because the properties "_AMQ_ROUTE_TO$.artemis.internal.sf.my-cluster.68d83229-be54-11e9-824e-f8b156cb82da" is not present in message properties
We are cleaning up internal properties based on following condition.
package org.apache.activemq.artemis.core.message.impl
public class CoreMessage extends RefCountMessage implements ICoreMessage {
private static final Predicate<SimpleString> INTERNAL_PROPERTY_NAMES_PREDICATE =
name -> (name.startsWith(Message.HDR_ROUTE_TO_IDS) && !name.equals(Message.HDR_ROUTE_TO_IDS)) ||
(name.startsWith(Message.HDR_ROUTE_TO_ACK_IDS) && !name.equals(Message.HDR_ROUTE_TO_ACK_IDS));