WildFly 10.1.0: ActiveMQConnectionTimedOutException while the server is running - wildfly

Question

We have a WildFly 10 instance configured with ActiveMQ Artemis.
After the server is running for weeks We have found this error on putting a message in a queue:
WARN [com.arjuna.ats.jta] (default task-39) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_RMFAIL for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a4809a7:12ad14f2:613b6c2b:8c8976c, node_name=1, branch_uid=0:ffff0a4809a7:12ad14f2:613b6c2b:8c89779, subordinatenodename=null, eis_name=java:/JmsXA NodeId:560e8de4-ccea-11eb-a9ec-014175cacf74 >: javax.transaction.xa.XAException
at org.apache.activemq.artemis.ra.ActiveMQRAXAResource.start(ActiveMQRAXAResource.java:85)
at org.apache.activemq.artemis.service.extensions.xa.ActiveMQXAResourceWrapperImpl.start(ActiveMQXAResourceWrapperImpl.java:121)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:662)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:423)
...
Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119014: Timed out after waiting 30,000 ms for response when sending packet 44]
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:398)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:304)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.simpleRollback(ActiveMQSessionContext.java:299)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:542)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:513)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.resetIfNeeded(ClientSessionImpl.java:594)
at org.apache.activemq.artemis.ra.ActiveMQRAXAResource.start(ActiveMQRAXAResource.java:80)
... 131 more
This error appears on each operation on the queues until the server is restarted.
The exception was logged by a local client (a war deployed in the application server itself). The local client sends messages, there are remote consumers consuming them. When the error occurred the as load was not higher than usual. I did not collect a thread dump I can try to collect them if the error occurs again.
The subsystem configuration is:
<subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
<server name="default">
<management jmx-enabled="true" />
<security enabled="false" />
<bindings-directory
path="/data/activemq/bindings" />
<journal-directory path="/data/activemq/journal" />
<large-messages-directory
path="/data/activemq/largemessages" />
<paging-directory path="/data/activemq/pages" />
<security-setting name="#">
<role name="guest" delete-non-durable-queue="true"
create-non-durable-queue="true" consume="true" send="true" />
</security-setting>
<address-setting name="#"
message-counter-history-day-limit="10" page-size-bytes="2097152"
max-size-bytes="104857600" max-delivery-attempts="-1"
redelivery-delay="300000" expiry-address="jms.queue.ExpiryQueue"
dead-letter-address="jms.queue.DLQ" />
<address-setting
name="jms.queue.queue1" max-delivery-attempts="-1"
expiry-address="jms.queue.ExpiryQueue"
dead-letter-address="jms.queue.DLQ" redelivery-delay="120000" />
<http-connector name="http-connector"
endpoint="http-acceptor" socket-binding="messaging" />
<http-connector name="http-connector-throughput"
endpoint="http-acceptor-throughput" socket-binding="messaging">
<param name="batch-delay" value="50" />
</http-connector>
<in-vm-connector name="in-vm" server-id="0" />
<http-acceptor name="http-acceptor"
http-listener="default" />
<http-acceptor name="http-acceptor-throughput"
http-listener="default">
<param name="batch-delay" value="50" />
<param name="direct-deliver" value="false" />
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0" />
<jms-queue name="ExpiryQueue"
entries="java:/jms/queue/ExpiryQueue" />
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ" />
<jms-queue name="queue1"
entries="queue1 queue/queue1 jms/queue/queue1 java:jboss/exported/queue1" />
<!--
...
--->
<jms-queue name="queueN"
entries="queueN queue/queueN jms/queue/queueN java:jboss/exported/queueN" />
<connection-factory name="InVmConnectionFactory"
entries="java:/ConnectionFactory" connectors="in-vm" />
<connection-factory
name="RemoteConnectionFactory"
failover-on-initial-connection="true" reconnect-attempts="-1"
block-on-acknowledge="true" consumer-window-size="0"
client-failure-check-period="10000" ha="true"
entries="java:jboss/exported/jms/RemoteConnectionFactory"
connectors="http-connector" />
<pooled-connection-factory
name="activemq-ra" transaction="xa"
entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory"
connectors="in-vm" />
</server>
</subsystem>
Is there any configuration error? Can anyone suggest a solution for this error?

Answer 1

WildFly 10.1.0 was released in August 2016 and it uses ActiveMQ Artemis 1.1.0 which was release in October 2015, nearly 6 years ago now. WildFly is up to version 25 now and ActiveMQ Artemis is at 2.18.0 now. I strongly encourage you to upgrade to a later release of WildFly or even use the latest release of ActiveMQ Artemis standalone (i.e. not embedded in WildFly).
Even if a bug is identified in 1.1.0 there will be no bug-fix release for this version. In order to get any kind of fix you will be forced to upgrade to the latest version unless you back-port the fix and build it yourself. It's worth noting that many hundreds of bugs have been fixed in ActiveMQ Artemis since 1.1.0 was released. Your issue could very well be among them.