Artemis broker is not announcing itself as backup on RHEL machine - activemq-artemis

I have 5 Red Hat Enterprise Linux (RHEL) 7.7 system with ActiveMQ Artemis brokers installed on each of them. I have configured 1 master and the rest as slaves. The snippet of broker.xml for the master broker is as follows:
<connectors>
<connector name="mainBrokerConnector">tcp://10.1.144.200:61616</connector>
</connectors>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>10.1.144.200</local-bind-address>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>mainBrokerConnector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>10.1.144.200</local-bind-address>
<local-bind-port>5432</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>mainBrokerConnector</connector-ref>
</broadcast-group>
</broadcast-groups>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Three of the four backup brokers when started (./artemis run) announce themselves as backups. In the logs of the backup brokers I see something like this (hence I know that they are in sync):
2020-10-07 14:19:43,480 INFO [org.apache.activemq.artemis.core.server] AMQ221024: Backup server ActiveMQServerImpl::serverUUID=e2fbfd28-04ec-11eb-b49a-005056aded6a is synchronized with live-server.
2020-10-07 14:19:43,768 INFO [org.apache.activemq.artemis.core.server] AMQ221031: backup announced
However, the fourth broker starts but does not logs anything about syncing. The broker.xml snippet of the fourth broker is as follows
<connectors>
<connector name="brokerTwoConnector">tcp://10.1.144.202:61616</connector>
</connectors>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>10.1.144.202</local-bind-address>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>brokerTwoConnector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>10.1.144.202</local-bind-address>
<local-bind-port>5432</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>brokerTwoConnector</connector-ref>
</broadcast-group>
</broadcast-groups>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
Rest of the slave brokers have the same configuration except different host IP addresses and connector names. So what is not configured properly that this broker is not syncing with the master broker?
EDIT
When I start the culprit broker then following logs appear which shows that its part of a cluster:
2020-10-07 16:10:15,525 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-10-07 16:10:15,572 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-10-07 16:10:15,598 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokerTwo/brokerTwo/data/journal to /opt/brokerTwo/brokerTwo/data/journal/oldreplica.2
2020-10-07 16:10:15,631 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-10-07 16:10:15,740 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-10-07 16:10:15,749 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-10-07 16:10:15,964 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-10-07 16:10:15,966 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-10-07 16:10:15,967 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-10-07 16:10:15,968 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-10-07 16:10:15,968 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-10-07 16:10:15,969 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-10-07 16:10:16,203 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-10-07 16:10:16,279 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-10-07 16:10:17,173 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-10-07 16:10:17,201 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-10-07 16:10:17,205 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-10-07 16:10:17,210 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/brokerTwo/brokerTwo/tmp/uploads
2020-10-07 16:10:17,239 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-10-07 16:10:17,293 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/brokerTwo/brokerTwo/etc/jolokia-access.xml]
2020-10-07 16:10:17,355 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control

Related

ActiveMQ Artemis rolling upgrade fails with AMQ214013

I've got two EC2 instances running Artemis in the master-slave replication setup and I always perform rolling upgrade by shutting down the slave, upgrading it and starting it again. Then I do the same with the master.
This no longer works for me while upgrading from 2.17 to 2.18. After upgrading one of the nodes I always get this message on that node:
AMQ214013: Failed to decode packet: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
No matter what I do. I even tried to setup a fresh new cluster only adding replication related bits of configuration:
Master
<connectors>
<connector name="broker-master">tcp://10.35.4.16:61616</connector>
<connector name="broker-slave">tcp://10.35.4.211:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="cluster">
<connector-ref>broker-master</connector-ref>
<check-period>5000</check-period>
<static-connectors>
<connector-ref>broker-slave</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<cluster-name>cluster</cluster-name>
<group-name>rs1</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave
<connectors>
<connector name="broker-master">tcp://10.35.4.16:61616</connector>
<connector name="broker-slave">tcp://10.35.4.211:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="cluster">
<connector-ref>broker-slave</connector-ref>
<check-period>5000</check-period>
<static-connectors>
<connector-ref>broker-master</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<cluster-name>cluster</cluster-name>
<group-name>rs1</group-name>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
Is there anything wrong with my configuration, or is it just not possible to from 2.17 to 2.18 like this?
Full log from master
...
2022-04-07 10:00:17,610 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
...
2022-04-07 10:00:18,726 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2022-04-07 10:00:18,728 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2022-04-07 10:00:18,759 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2022-04-07 10:00:18,760 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2022-04-07 10:00:19,072 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2022-04-07 10:00:19,075 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5445 for protocols [HORNETQ,STOMP]
2022-04-07 10:00:19,079 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5672 for protocols [AMQP]
2022-04-07 10:00:19,082 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1883 for protocols [MQTT]
2022-04-07 10:00:19,085 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61613 for protocols [STOMP]
2022-04-07 10:00:19,086 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2022-04-07 10:00:19,086 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.17.0 [broker-master, nodeID=eb08fe05-b640-11ec-b6e2-06e66cf4b718]
2022-04-07 10:00:19,470 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2022-04-07 10:00:19,590 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2022-04-07 10:00:20,032 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2022-04-07 10:00:20,123 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2022-04-07 10:00:20,126 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.11.0
2022-04-07 10:00:20,137 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2022-04-07 10:00:20,171 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2022-04-07 10:00:20,182 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/broker-master/etc/jolokia-access.xml]
2022-04-07 10:00:20,342 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://10.35.4.16:8161
2022-04-07 10:00:20,343 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://10.35.4.16:8161/console/jolokia
2022-04-07 10:00:20,343 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://10.35.4.16:8161/console
2022-04-07 10:00:52,429 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile{activemq-data-49.amq, opened=false, pendingClose=false, pendingCallbacks=org.apache.activemq.artemis.utils.AutomaticLatch#2907ccd6} (size=10,485,760) to replica.
2022-04-07 10:00:52,931 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-34.bindings (size=1,048,576) to replica.
2022-04-07 10:00:52,963 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-35.bindings (size=1,048,576) to replica.
2022-04-07 10:00:53,011 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-1.bindings (size=1,048,576) to replica.
2022-04-07 10:00:53,073 WARN [org.apache.activemq.artemis.core.server] AMQ222092: Connection to the backup node failed, removing replication now: ActiveMQRemoteDisconnectException[errorType=REMOTE_DISCONNECT message=null]
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:582) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor$Listener.connectionDestroyed(NettyAcceptor.java:942) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.lambda$channelInactive$0(ActiveMQChannelHandler.java:89) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.17.0.jar:2.17.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
2022-04-07 10:01:23,032 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,035 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,045 WARN [org.apache.activemq.artemis.core.server] AMQ222013: Error when trying to start replication: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,058 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected): java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1848) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:648) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:433) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.17.0.jar:2.17.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
2022-04-07 10:01:23,297 INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter
2022-04-07 10:01:23,303 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2022-04-07 10:01:23,325 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2022-04-07 10:01:23,336 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2022-04-07 10:01:23,384 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.17.0 [eb08fe05-b640-11ec-b6e2-06e66cf4b718] stopped, uptime 1 minute
Full log from slave
2022-04-07 10:00:50,645 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2022-04-07 10:00:50,703 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2022-04-07 10:00:50,747 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/broker-slave/data/bindings/oldreplica.7
2022-04-07 10:00:50,750 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/broker-slave/data/bindings to /opt/artemis/broker-slave/data/bindings/oldreplica.9
2022-04-07 10:00:50,753 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/broker-slave/data/journal/oldreplica.8
2022-04-07 10:00:50,755 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/broker-slave/data/journal to /opt/artemis/broker-slave/data/journal/oldreplica.10
2022-04-07 10:00:50,888 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2022-04-07 10:00:51,031 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2022-04-07 10:00:51,124 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2022-04-07 10:00:51,323 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2022-04-07 10:00:51,326 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2022-04-07 10:00:51,328 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2022-04-07 10:00:51,329 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2022-04-07 10:00:51,330 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2022-04-07 10:00:51,330 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2022-04-07 10:00:51,538 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2022-04-07 10:00:51,685 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2022-04-07 10:00:52,052 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.18.0 [null] started, waiting live to fail before it gets active
2022-04-07 10:00:52,224 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2022-04-07 10:00:52,337 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2022-04-07 10:00:52,341 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.13.5
2022-04-07 10:00:52,351 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2022-04-07 10:00:52,383 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2022-04-07 10:00:52,392 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/broker-slave/etc/jolokia-access.xml]
2022-04-07 10:00:52,570 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8161
2022-04-07 10:00:52,571 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://localhost:8161/console/jolokia
2022-04-07 10:00:52,571 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8161/console
2022-04-07 10:00:53,029 ERROR [org.apache.activemq.artemis.core.client] AMQ214013: Failed to decode packet: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readByte(ChannelBufferWrapper.java:300) [artemis-commons-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ReplicationStartSyncMessage.decodeRest(ReplicationStartSyncMessage.java:158) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.18.0.jar:2.18.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.18.0.jar:2.18.0]
2022-04-07 10:00:53,031 ERROR [org.apache.activemq.artemis.core.client] AMQ214031: Failed to decode buffer, disconnect immediately.: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:401) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.18.0.jar:2.18.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.18.0.jar:2.18.0]
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readByte(ChannelBufferWrapper.java:300) [artemis-commons-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ReplicationStartSyncMessage.decodeRest(ReplicationStartSyncMessage.java:158) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388) [artemis-core-client-2.18.0.jar:2.18.0]
... 20 more
2022-04-07 10:00:53,052 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57) [code=GENERIC_EXCEPTION]
2022-04-07 10:00:53,092 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2022-04-07 10:00:53,093 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2022-04-07 10:00:53,093 INFO [org.apache.activemq.artemis.core.server] AMQ221083: ignoring quorum vote as max cluster size is 1.
2022-04-07 10:00:53,094 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2022-04-07 10:00:53,113 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:302) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4271) [artemis-server-2.18.0.jar:2.18.0]
2022-04-07 10:01:23,105 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2022-04-07 10:01:23,107 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2022-04-07 10:01:42,243 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-04-07 10:01:42,274 INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter
2022-04-07 10:01:42,278 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2022-04-07 10:01:42,296 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2022-04-07 10:01:42,304 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
This bug was introduced via ARTEMIS-3340. It should be fixed in the next release (i.e. 2.22.0). Until then there's no work-around unfortunately.
You may consider creating a completely separate new live/backup pair and then using a DNS update to direct clients from the old pair to the new pair.

Artemis slave node does not go live start in full cluster

I'm trying to run a 6 node Apache Artemis static cluster as 3 live-backup pairs. In order not to lose data during full cluster restart I've also created a mechanism figuring out which of the pair has newer data, and based on that the server automatically decides whether it's safe to start that particular node. During full cluster restart the servers are started simultaneously, and nodes with newer data start before nodes that were lately acting as backups.
This scenario has been working flawlessly with single live-backup pair.
Shutdown:
Master node is live, slave node is backup
Master node => slave node goes live
Slave node
Start:
Both nodes are dead
Slave node => goes live
Master node => goes live, slave node goes backup
However in the 6 node cluster the backup node does not go live at all:
3 live nodes, 3 backup nodes; pairs bound together in separate groups (rs1, rs2, rs3)
Stop rs1 master node => rs1 slave node goes live
Stop rs1 slave node
Start rs1 slave node => never goes live
Must not start rs1 master node because of potential data loss
Any idea what's wrong with my setup?
Master HA configuration (the group name varies):
<?xml version="1.0" encoding="UTF-8"?>
<ha-policy xmlns="urn:activemq:core">
<replication>
<master>
<cluster-name>clouedi-mq-dev</cluster-name>
<group-name>rs1</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave HA configuration (the group name varies):
<?xml version="1.0" encoding="UTF-8"?>
<ha-policy xmlns="urn:activemq:core">
<replication>
<slave>
<cluster-name>clouedi-mq-dev</cluster-name>
<group-name>rs1</group-name>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
Thank you.
Before the issue
Master
2020-11-30 12:34:54,021 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:34:54,082 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:34:59,425 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:34:59,429 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:34:59,435 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:34:59,487 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
...address deployment...
2020-11-30 12:35:00,123 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,STOMP]
2020-11-30 12:35:00,124 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-11-30 12:35:00,124 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.15.0 [edimq-broker-master-dev-az1-1, nodeID=6937ad47-309c-11eb-811e-0275ba22b614]
2020-11-30 12:35:00,315 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#58d025c9 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#7c398410 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#58d025c9 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#7c398410 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1779219567[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-master-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-96, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:35:00,318 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#77158457 [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#2f988822 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#77158457 [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#2f988822 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1779219567[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-master-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-96, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:35:00,688 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:35:00,766 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:35:01,187 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:35:01,196 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:35:01,199 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:35:01,203 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-master-dev-az1-1/tmp/uploads
2020-11-30 12:35:01,217 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:35:01,240 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-master-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:35:01,269 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:35:01,389 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.96, edimq-broker-master-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:35:01,722 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:35:01,722 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:35:01,723 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-11-30 12:35:18,072 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/artemis/edimq-broker-master-dev-az1-1/data/journal/activemq-data-45.amq (size=10,485,760) to replica.
2020-11-30 12:35:18,687 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-2.bindings (size=1,048,576) to replica.
2020-11-30 12:35:18,697 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-50.bindings (size=1,048,576) to replica.
2020-11-30 12:35:18,709 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-37.bindings (size=1,048,576) to replica.
Slave
2020-11-30 12:35:16,773 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:35:16,847 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:35:16,976 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.44
2020-11-30 12:35:16,978 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal to /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.46
2020-11-30 12:35:17,039 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:35:17,101 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:35:17,107 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:35:17,263 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:35:17,356 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:35:17,405 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-11-30 12:35:17,863 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [null] started, waiting live to fail before it gets active
2020-11-30 12:35:17,974 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:35:17,985 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:35:17,987 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:35:17,992 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-slave-dev-az1-1/tmp/uploads
2020-11-30 12:35:18,021 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:35:18,058 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-slave-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:35:18,089 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:35:18,216 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.101, edimq-broker-slave-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:35:18,581 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:35:18,582 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:35:18,582 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-11-30 12:35:18,964 INFO [org.apache.activemq.artemis.core.server] AMQ221024: Backup server ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614 is synchronized with live-server.
2020-11-30 12:35:18,981 INFO [org.apache.activemq.artemis.core.server] AMQ221031: backup announced
Shutting down
Master
2020-11-30 12:38:48,621 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:38:48,632 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:38:48,644 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe
2020-11-30 12:38:48,644 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4
2020-11-30 12:38:48,796 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-11-30 12:38:48,800 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-11-30 12:38:48,849 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-11-30 12:38:48,853 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-11-30 12:38:48,877 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.15.0 [6937ad47-309c-11eb-811e-0275ba22b614] stopped, uptime 3 minutes
Slave goes live
2020-11-30 12:38:48,621 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2020-11-30 12:38:48,622 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 2 quorum votes
2020-11-30 12:38:48,623 INFO [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results.
2020-11-30 12:38:48,637 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.2.96/10.35.2.96:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-11-30 12:38:48,644 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.2.96/10.35.2.96:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-11-30 12:38:48,666 INFO [org.apache.activemq.artemis.core.server] AMQ221060: Sending quorum vote request to 10.35.2.20/10.35.2.20:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=false]
2020-11-30 12:38:48,667 INFO [org.apache.activemq.artemis.core.server] AMQ221060: Sending quorum vote request to 10.35.2.206/10.35.2.206:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=false]
2020-11-30 12:38:48,675 INFO [org.apache.activemq.artemis.core.server] AMQ221061: Received quorum vote response from 10.35.2.206/10.35.2.206:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=true]
2020-11-30 12:38:48,676 INFO [org.apache.activemq.artemis.core.server] AMQ221061: Received quorum vote response from 10.35.2.20/10.35.2.20:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=true]
2020-11-30 12:38:48,677 INFO [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes.
2020-11-30 12:38:48,682 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2020-11-30 12:38:48,705 INFO [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614 to become 'live'
2020-11-30 12:38:48,722 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
...address deployment...
2020-11-30 12:38:49,217 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-11-30 12:38:49,238 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,STOMP]
2020-11-30 12:38:49,269 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#50794972 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#13fa6cc targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#50794972 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#13fa6cc targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1854054128[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-slave-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-101, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:38:49,270 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#39f7703c [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#66eabf5 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#39f7703c [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#66eabf5 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1854054128[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-slave-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-101, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]] is connected
Shutting down slave
2020-11-30 12:39:56,693 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:39:56,694 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:39:56,704 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe
2020-11-30 12:39:56,711 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4
2020-11-30 12:39:56,859 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-11-30 12:39:56,863 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-11-30 12:39:56,914 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-11-30 12:39:56,918 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-11-30 12:39:56,946 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.15.0 [6937ad47-309c-11eb-811e-0275ba22b614] stopped, uptime 4 minutes
Starting up
Slave
2020-11-30 12:40:34,153 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:40:34,209 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:40:34,240 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings/oldreplica.19
2020-11-30 12:40:34,246 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings to /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings/oldreplica.21
2020-11-30 12:40:34,248 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.45
2020-11-30 12:40:34,248 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal to /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.47
2020-11-30 12:40:34,249 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging/oldreplica.19
2020-11-30 12:40:34,257 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging to /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging/oldreplica.21
2020-11-30 12:40:34,321 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:40:34,402 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:40:34,413 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:40:34,607 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-11-30 12:40:34,774 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:40:34,888 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:40:34,927 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [null] started, waiting live to fail before it gets active
2020-11-30 12:40:35,301 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:40:35,310 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:40:35,312 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:40:35,320 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-slave-dev-az1-1/tmp/uploads
2020-11-30 12:40:35,339 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:40:35,365 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-slave-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:40:35,392 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:40:35,511 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.101, edimq-broker-slave-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
The above shows that the slave does not start at all.
When ActiveMQ Artemis uses replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network.
Upon start-up the backup server will first need to synchronize all existing data from the live server before becoming capable of replacing the live server should it fail. So unlike when using shared storage, a replicating backup will not be a fully operational backup right after start-up, but only after it finishes synchronizing the data with its live server.
Why does a single live-backup pair look working?
The backup node at start-up executes the following steps at SharedNothingBackupActivation:
Initialize (it looks running)
Waiting on cluster connection (it is starving here)
Starting backup manager
So in a scenario with a single live-backup pair the backup node doesn't complete the initialization.

Artemis Master-Slave failover and sync error message in logs

After struggling with Artemis 2.11 and an older Java version I decided to update my whole system to the "latest greatest software" that is currently available. So I am using Artemis 2.14 and Java 14.0.2 on two Ubuntu 18.04 VM with 4 Cores an 16 GB RAM.
I configured the master-slave ha-policy like this:
MASTER:
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
SLAVE
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
And I am using the cluster-connection like this:
MASTER
<cluster-connections>
<cluster-connection name="artemis-cluster">
<connector-ref>Artemis-Node-A-Sync</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>Node-B-Sync</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
SLAVE
<cluster-connections>
<cluster-connection name="artemis-cluster">
<connector-ref>Node-B-Sync</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>Node-A-Sync</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
My problem is that I get this ERROR message from at SLAVE...
2020-08-07 12:45:37,548 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-08-07 12:45:37,550 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-08-07 12:45:37,552 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/mybroker-broker/tmp/uploads
2020-08-07 12:45:37,565 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-08-07 12:45:37,585 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/mybroker-broker/etc/jolokia-access.xml]
2020-08-07 12:45:37,605 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-08-07 12:45:37,703 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, *.*.*.*, *.*.*.*, localhost.localdomain]
2020-08-07 12:45:37,966 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-08-07 12:45:37,967 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-08-07 12:45:37,967 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-08-07 12:45:46,905 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 192.168.144.75/192.168.144.75:22522 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-07 12:45:50,678 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2020-08-07 12:45:50,678 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2020-08-07 12:45:50,679 INFO [org.apache.activemq.artemis.core.server] AMQ221083: ignoring quorum vote as max cluster size is 1.
2020-08-07 12:45:50,679 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2020-08-07 12:45:50,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:310) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3946) [artemis-server-2.14.0.jar:2.14.0]
The logfile at master is:
2020-08-07 12:44:58,292 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 21% loaded
2020-08-07 12:44:58,540 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 42% loaded
2020-08-07 12:44:59,020 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 64% loaded
2020-08-07 12:44:59,416 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 85% loaded
2020-08-07 12:45:12,143 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2020-08-07 12:45:12,145 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2020-08-07 12:45:12,151 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2020-08-07 12:45:12,152 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2020-08-07 12:45:12,382 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1883 for protocols [MQTT]
2020-08-07 12:45:12,385 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:22522 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2020-08-07 12:45:12,387 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:8883 for protocols [MQTT]
2020-08-07 12:45:12,388 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:18884 for protocols [MQTT]
2020-08-07 12:45:12,388 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-08-07 12:45:12,389 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.14.0 [0.0.0.0, nodeID=95f808d9-d641-11ea-9c48-005056073c33]
2020-08-07 12:45:12,650 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-08-07 12:45:12,686 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-08-07 12:45:13,281 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-08-07 12:45:13,304 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-08-07 12:45:13,306 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-08-07 12:45:13,311 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/mybroker-broker/tmp/uploads
2020-08-07 12:45:13,325 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-08-07 12:45:13,344 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/mybroker-broker/etc/jolokia-access.xml]
2020-08-07 12:45:13,373 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-08-07 12:45:13,472 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, *.*.*.*, 192.168.144.75, localhost.localdomain]
2020-08-07 12:45:13,680 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-08-07 12:45:13,681 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-08-07 12:45:13,681 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-08-07 12:45:46,608 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page, message=/opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:156) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:98) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.Page.open(Page.java:483) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.openNewPage(PagingStoreImpl.java:1136) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.forceAnotherPage(PagingStoreImpl.java:606) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.getPageInformationForSync(JournalStorageManager.java:738) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:665) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:179) [artemis-server-2.14.0.jar:2.14.0]
at java.base/java.lang.Thread.run(Thread.java:832) [java.base:]
Caused by: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
at java.base/java.io.RandomAccessFile.open0(Native Method) [java.base:]
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:347) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:261) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) [java.base:]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:143) [artemis-journal-2.14.0.jar:2.14.0]
... 8 more
2020-08-07 12:45:46,615 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
at java.base/java.io.RandomAccessFile.open0(Native Method) [java.base:]
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:347) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:261) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) [java.base:]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:143) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:98) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.Page.open(Page.java:483) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.openNewPage(PagingStoreImpl.java:1136) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.forceAnotherPage(PagingStoreImpl.java:606) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.getPageInformationForSync(JournalStorageManager.java:738) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:665) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:179) [artemis-server-2.14.0.jar:2.14.0]
at java.base/java.lang.Thread.run(Thread.java:832) [java.base:]
2020-08-07 12:45:46,693 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected): java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1844) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:644) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:545) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:524) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:433) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.14.0.jar:2.14.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.14.0.jar:2.14.0]
2020-08-07 12:45:47,407 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-08-07 12:45:47,409 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 2020-08-07 12:45:47,442 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-08-07 12:45:47,444 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-08-07 12:45:47,476 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.14.0 [95f808d9-d641-11ea-9c48-005056073c33] stopped, uptime 1 minute
tail: /opt/mybroker-broker/log/artemis.log: file truncated
2020-08-07 12:46:07,212 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-08-07 12:46:07,242 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-08-07 12:46:09,516 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-08-07 12:46:09,542 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 5,368,709,120
2020-08-07 12:46:09,589 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-08-07 12:46:09,590 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-08-07 12:46:09,591 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-08-07 12:46:09,592 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-08-07 12:46:09,592 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-08-07 12:46:09,593 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-08-07 12:46:09,776 WARN [org.apache.activemq.artemis.core.server] AMQ222035: Directory /opt/mybroker-broker/data/paging/9fd77264-d6db-11ea-bb8e-005056073c33 did not have an identification file address.txt
2020-08-07 12:46:13,695 WARN [org.apache.activemq.artemis.core.server] AMQ222035: Directory /opt/mybroker-broker/data/paging/9fd77264-d6db-11ea-bb8e-005056073c33 did not have an identification file address.txt
...when I do the following:
Stop the SLAVE
Restart the MASTER
Start the SLAVE
Using Jolokia I see that the SLAVE is connected to the MASTER after some time.
Questions:
What does the ERROR message mean?
Is there an CLI command to get some information about synchronization?
Is there an CLI command to get some system/cluster status information?
The fundamental problem is noted in the log from the master broker:
2020-08-07 12:45:46,615 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
This is an environmental error indicating that the operating system will not allow the process (i.e. the broker) to open any more files.
Since the broker can't open all the files it needs then it can't properly synchronize its data with the backup. Furthermore, the inability to open additional files is interpreted as a "critical" error which forces the broker to shut itself down. When the master broker shuts down the slave activates, but since the master was not able to complete the initial replication with the slave the slave states:
2020-08-07 12:45:50,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
The slave is simply logging the fact that it didn't complete the initial synchronization/replication process with the master before the master was shut down.
Ultimately you need to increase the file limit for the user running the broker. A quick web search should help you with the specifics of that task for your particular operating system.

Cant access Artemis console

I have just install a new artemis in my new server, able to run the service normally but not able to access from outside.
I also have followed this question that seems have the similar issue with me, but
seems not able to work.
Artemis-2.6.3 console: Service Unavailable
my current jolokia-access :
<restrict>
<remote>
<host>localhost</host>
<host>10.85.102.148</host><!--This is my computer ip-->
</remote>
<!--
<cors>
<allow-origin>*://localhost*</allow-origin>
<strict-checking/>
</cors>
-->
</restrict>
current logging
2019-04-25 20:16:07,013 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2019-04-25 20:16:07,046 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2019-04-25 20:16:07,080 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2019-04-25 20:16:07,133 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2019-04-25 20:16:07,162 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2019-04-25 20:16:07,163 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2019-04-25 20:16:07,164 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2019-04-25 20:16:07,164 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2019-04-25 20:16:07,165 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2019-04-25 20:16:07,166 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2019-04-25 20:16:07,222 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock
2019-04-25 20:16:07,223 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock
2019-04-25 20:16:08,143 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2019-04-25 20:16:08,513 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61619 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2019-04-25 20:16:08,521 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5446 for protocols [HORNETQ,STOMP]
2019-04-25 20:16:08,536 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5673 for protocols [AMQP]
2019-04-25 20:16:08,540 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1884 for protocols [MQTT]
2019-04-25 20:16:08,545 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61614 for protocols [STOMP]
2019-04-25 20:16:08,547 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2019-04-25 20:16:08,548 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.4 [0.0.0.0, nodeID=c4f07cfd-6745-11e9-bfbc-005056b0ef9d]
2019-04-25 20:16:09,647 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2019-04-25 20:16:09,749 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2019-04-25 20:16:10,199 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2019-04-25 20:16:10,269 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2019-04-25 20:16:10,273 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2019-04-25 20:16:10,298 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/CMS/dev/artemisCMSDEV2.6.4/tmp/uploads
2019-04-25 20:16:10,318 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2019-04-25 20:16:10,362 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/CMS/dev/artemisCMSDEV2.6.4/etc/jolokia-access.xml]
2019-04-25 20:16:10,403 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2019-04-25 20:16:10,581 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.92.202.52, mask.myserver.com, 192.168.46.52, mask-p1.myserver.com]
2019-04-25 20:16:10,882 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8162
2019-04-25 20:16:10,883 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://localhost:8162/console/jolokia
2019-04-25 20:16:10,883 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8162/console
According to the log the console is bound to http://localhost:8162 therefore not accessible from outside localhost. Please check the bind entry in bootstrap.xml.
Also note that you have a restriction with localhost in cors section of jolokia-access.xml what you also should adjust.
You need just to modify these two files us below :
$[broker_artemis_path]/etc/bootstrap.xml
<web bind="http://0.0.0.0:8161" path="web">
<app url="activemq-branding" war="activemq-branding.war"/>
<app url="artemis-plugin" war="artemis-plugin.war"/>
<app url="console" war="console.war"/>
</web>
and $[broker_artemis_path]/etc/jolokia-access.xml
<cors>
<!-- Allow cross origin access from localhost ... -->
<allow-origin>*://*</allow-origin>
<!-- Options from this point on are auto-generated by Create.java from the Artemis CLI -->
<!-- Check for the proper origin on the server side, too -->
<strict-checking/>
</cors>
For more information you can check this link :
https://medium.com/#hasnat.saeed/setup-activemq-artemis-on-ubuntu-18-04-76bb4975308b

Artemis crashed and messages/queues lost

Any ideas what happened here? Can messages and queues be restored? How to use ActiveMQ Artemis data? All queues/messages are lost. Only DLQ/ExpiryQueue.
We had to do a configuration change. So we started updating the Slave.
To failover to the Slave to fix the Master we did a ./artemis-service stop #16:32
Some warnings on Master but not much else.
On the Slave we can see some timeouts and connection failures. Not sure if Slave took over the Queues and worked ok or not.
#16:46:27 we did a ./artemis-service start on the Master
Seems like Artemis started moving/removing bindings/journals
Master seems to fail to start with Connections issues #16:46:33
Slave tries to restart again and take control #16:46:37
Slave cleans out journals/bindings
And now all queues are gone. Only DLQ and ExpiryQueue exists on Slave. Master is down.
Can we restore from oldreplica? Might be wiped already?
We have not yet ried to restore from Replicas. Is there a manual or KnowledgeBase?
Master:
A few of these:
16:32:15,847 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17:1:-1
16:32:15,847 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17:1:-1
...
16:32:16,088 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
16:32:16,093 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
16:32:16,104 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056926b6e] stopped, uptime 4 days 3 hours
Meanwhile on the Slave:
16:32:15,867 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
16:32:15,868 INFO [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results.
16:32:15,869 INFO [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes.
16:32:15,869 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
16:32:15,889 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
16:32:15,944 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
16:32:15,999 INFO [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=06926557-2906-11e8-a15f-005056926b6e to become 'live'
16:32:16,450 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
16:32:34,825 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue DLQ on address DLQ
16:32:34,825 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue ExpiryQueue on address ExpiryQueue
16:32:35,156 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
16:33:51,161 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:53781 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:34:10,061 INFO [io.hawt.web.LoginServlet] hawtio login is using 1800 sec. HttpSession timeout
16:34:11,891 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:34:52,912 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:35:01,183 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:47171 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:36:01,191 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:54700 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:36:12,705 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:36:53,104 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:37:11,203 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:16751 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:38:11,209 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20634 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:38:12,182 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:39:21,216 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:61541 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:40:21,225 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:49708 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:41:18,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred.
16:45:51,271 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20748 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
16:46:32,151 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-167.amq (size=10,485,760) to replica.
16:46:33,100 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-168.amq (size=10,485,760) to replica.
16:46:33,182 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1
16:46:33,188 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1
16:46:33,199 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380d4fc6-4f13-11e9-b0fb-00505692a0af
16:46:33,288 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
16:46:33,290 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
16:46:33,391 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
16:46:33,397 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
16:46:33,408 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056926b6e] stopped, uptime 4 days 3 hours
16:46:37,163 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
16:46:37,266 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging)
16:46:37,288 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.45
16:46:37,292 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/bindings to /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.47
16:46:37,295 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.49
16:46:37,597 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/journal to /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.51
16:46:37,628 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.45
16:46:37,852 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
16:46:38,056 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/paging to /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.47
16:46:38,091 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
16:46:38,157 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 2,147,483,648
16:46:38,252 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
16:46:38,252 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
16:46:38,254 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
16:46:38,328 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
16:46:42,502 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
16:46:42,572 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/brokers/ActiveMQServer2//etc/jolokia-access.xml]
16:46:42,617 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
16:46:42,788 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, aaa.bbb.ccc.50, nodep02.domain.local]
16:46:43,202 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://nodep02:8161
16:46:43,202 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://nodep02:8161/console/jolokia
Back on the Master
16:46:27,913 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
16:46:27,999 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging)
16:46:28,440 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/bindings/oldreplica.18
16:46:28,447 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/bindings to /opt/brokers/ActiveMQServer1/./data/bindings/oldreplica.20
16:46:28,450 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/journal/oldreplica.18
16:46:28,478 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/journal to /opt/brokers/ActiveMQServer1/./data/journal/oldreplica.20
16:46:28,482 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer1/./data/paging/oldreplica.18
16:46:28,517 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/paging to /opt/brokers/ActiveMQServer1/./data/paging/oldreplica.20
16:46:28,580 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
16:46:33,209 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
16:46:33,266 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
16:46:33,273 WARN [org.apache.activemq.artemis.journal] File not opened, file=null: java.lang.NullPointerException: File not opened, file=null
at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.checkOpened(AIOSequentialFile.java:328) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:242) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4.0]
16:46:33,318 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ119026: Backup Server was not yet in sync with live]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:314) [artemis-server-2.4.0.jar:2.4.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4.0]
16:46:33,995 INFO [io.hawt.web.keycloak.KeycloakServlet] Keycloak integration is disabled
16:46:40,486 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped
16:46:50,033 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped
16:47:00,034 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped
Broker.xml (Master)
<?xml version="1.0"?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
<name>ActiveMQ1</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>./data/paging</paging-directory>
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>-1</journal-pool-files>
<journal-file-size>10M</journal-file-size>
<journal-buffer-size>33554432</journal-buffer-size>
<!-- Size in bytes -->
<journal-buffer-timeout>128000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<connectors>
<connector name="artemis">tcp://nodep01:61616</connector>
<connector name="ActiveMQ2-Connector">tcp://nodep02:61616</connector>
</connectors>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<acceptors>
<acceptor name="artemis">tcp://nodep01:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
<acceptor name="amqp">tcp://nodep01:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
<acceptor name="stomp">tcp://nodep01:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<acceptor name="hornetq">tcp://nodep01:5445?protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<acceptor name="artemis+ssl">tcp://nodep01:61443?sslEnabled=true;keyStorePath=/opt/brokers/ActiveMQServer1/activemq.keystore;keyStorePassword=CCCCC;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
</acceptors>
<cluster-user>XXXXXXXX</cluster-user>
<cluster-password>YYYYYYY</cluster-password>
<cluster-connections>
<cluster-connection name="ActiveMQClusterPROD">
<connector-ref>artemis</connector-ref>
<min-large-message-size>33554432</min-large-message-size>
<!-- Size should be eqvialent to journal-buffer-size -->
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>ActiveMQ2-Connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq" />
<permission type="deleteNonDurableQueue" roles="amq" />
<permission type="createDurableQueue" roles="amq" />
<permission type="deleteDurableQueue" roles="amq" />
<permission type="createAddress" roles="amq" />
<permission type="deleteAddress" roles="amq" />
<permission type="consume" roles="amq" />
<permission type="browse" roles="amq" />
<permission type="send" roles="amq" />
<!-- we need this otherwise ./artemis data imp wouldn't work-->
<permission type="manage" roles="amq" />
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
</core>
</configuration>
Broker.xml (Slave)
<?xml version="1.0"?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
<name>ActiveMQ2</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>./data/paging</paging-directory>
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>-1</journal-pool-files>
<journal-file-size>10M</journal-file-size>
<journal-buffer-size>33554432</journal-buffer-size>
<!-- size in bytes -->
<journal-buffer-timeout>104000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://nodep02:61616</connector>
<connector name="ActiveMQ1-Connector">tcp://nodep01:61616</connector>
</connectors>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<acceptors>
<acceptor name="artemis">tcp://nodep02:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
<acceptor name="amqp">tcp://nodep02:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
<acceptor name="stomp">tcp://nodep02:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<acceptor name="hornetq">tcp://nodep02:5445?protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<acceptor name="mqtt">tcp://nodep02:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
<acceptor name="artemis+ssl">tcp://nodep02:61443?sslEnabled=true;keyStorePath=/opt/brokers/ActiveMQServer2/activemq.keystore;keyStorePassword=CCCCC;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
</acceptors>
<cluster-user>XXXXXXXX</cluster-user>
<cluster-password>YYYYYYY</cluster-password>
<cluster-connections>
<cluster-connection name="ActiveMQClusterPROD">
<connector-ref>artemis</connector-ref>
<min-large-message-size>33554432</min-large-message-size>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>ActiveMQ1-Connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq" />
<permission type="deleteNonDurableQueue" roles="amq" />
<permission type="createDurableQueue" roles="amq" />
<permission type="deleteDurableQueue" roles="amq" />
<permission type="createAddress" roles="amq" />
<permission type="deleteAddress" roles="amq" />
<permission type="consume" roles="amq" />
<permission type="browse" roles="amq" />
<permission type="send" roles="amq" />
<!-- we need this otherwise ./artemis data imp wouldn't work-->
<permission type="manage" roles="amq" />
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
</core>
</configuration>
You can restore the full journal from the "oldreplica" backups. That's what they're there for. You simply need to copy the files from the backup location to the original location. As always, be careful when copying data so that you don't overwrite something you might need later.