After struggling with Artemis 2.11 and an older Java version I decided to update my whole system to the "latest greatest software" that is currently available. So I am using Artemis 2.14 and Java 14.0.2 on two Ubuntu 18.04 VM with 4 Cores an 16 GB RAM.
I configured the master-slave ha-policy like this:
MASTER:
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
SLAVE
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
And I am using the cluster-connection like this:
MASTER
<cluster-connections>
<cluster-connection name="artemis-cluster">
<connector-ref>Artemis-Node-A-Sync</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>Node-B-Sync</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
SLAVE
<cluster-connections>
<cluster-connection name="artemis-cluster">
<connector-ref>Node-B-Sync</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>Node-A-Sync</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
My problem is that I get this ERROR message from at SLAVE...
2020-08-07 12:45:37,548 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-08-07 12:45:37,550 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-08-07 12:45:37,552 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/mybroker-broker/tmp/uploads
2020-08-07 12:45:37,565 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-08-07 12:45:37,585 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/mybroker-broker/etc/jolokia-access.xml]
2020-08-07 12:45:37,605 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-08-07 12:45:37,703 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, *.*.*.*, *.*.*.*, localhost.localdomain]
2020-08-07 12:45:37,966 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-08-07 12:45:37,967 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-08-07 12:45:37,967 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-08-07 12:45:46,905 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 192.168.144.75/192.168.144.75:22522 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-07 12:45:50,678 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2020-08-07 12:45:50,678 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2020-08-07 12:45:50,679 INFO [org.apache.activemq.artemis.core.server] AMQ221083: ignoring quorum vote as max cluster size is 1.
2020-08-07 12:45:50,679 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2020-08-07 12:45:50,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:310) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:3946) [artemis-server-2.14.0.jar:2.14.0]
The logfile at master is:
2020-08-07 12:44:58,292 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 21% loaded
2020-08-07 12:44:58,540 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 42% loaded
2020-08-07 12:44:59,020 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 64% loaded
2020-08-07 12:44:59,416 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 85% loaded
2020-08-07 12:45:12,143 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2020-08-07 12:45:12,145 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2020-08-07 12:45:12,151 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2020-08-07 12:45:12,152 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2020-08-07 12:45:12,382 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1883 for protocols [MQTT]
2020-08-07 12:45:12,385 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:22522 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2020-08-07 12:45:12,387 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:8883 for protocols [MQTT]
2020-08-07 12:45:12,388 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:18884 for protocols [MQTT]
2020-08-07 12:45:12,388 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-08-07 12:45:12,389 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.14.0 [0.0.0.0, nodeID=95f808d9-d641-11ea-9c48-005056073c33]
2020-08-07 12:45:12,650 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-08-07 12:45:12,686 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-08-07 12:45:13,281 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-08-07 12:45:13,304 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-08-07 12:45:13,306 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-08-07 12:45:13,311 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/mybroker-broker/tmp/uploads
2020-08-07 12:45:13,325 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-08-07 12:45:13,344 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/mybroker-broker/etc/jolokia-access.xml]
2020-08-07 12:45:13,373 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-08-07 12:45:13,472 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, *.*.*.*, 192.168.144.75, localhost.localdomain]
2020-08-07 12:45:13,680 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-08-07 12:45:13,681 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-08-07 12:45:13,681 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-08-07 12:45:46,608 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page, message=/opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:156) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:98) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.Page.open(Page.java:483) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.openNewPage(PagingStoreImpl.java:1136) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.forceAnotherPage(PagingStoreImpl.java:606) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.getPageInformationForSync(JournalStorageManager.java:738) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:665) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:179) [artemis-server-2.14.0.jar:2.14.0]
at java.base/java.lang.Thread.run(Thread.java:832) [java.base:]
Caused by: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
at java.base/java.io.RandomAccessFile.open0(Native Method) [java.base:]
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:347) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:261) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) [java.base:]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:143) [artemis-journal-2.14.0.jar:2.14.0]
... 8 more
2020-08-07 12:45:46,615 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
at java.base/java.io.RandomAccessFile.open0(Native Method) [java.base:]
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:347) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:261) [java.base:]
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) [java.base:]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:143) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:98) [artemis-journal-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.Page.open(Page.java:483) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.openNewPage(PagingStoreImpl.java:1136) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.forceAnotherPage(PagingStoreImpl.java:606) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.getPageInformationForSync(JournalStorageManager.java:738) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:665) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:179) [artemis-server-2.14.0.jar:2.14.0]
at java.base/java.lang.Thread.run(Thread.java:832) [java.base:]
2020-08-07 12:45:46,693 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected): java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1844) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:644) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:545) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:524) [artemis-core-client-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:433) [artemis-server-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.14.0.jar:2.14.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.14.0.jar:2.14.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.14.0.jar:2.14.0]
2020-08-07 12:45:47,407 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-08-07 12:45:47,409 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 2020-08-07 12:45:47,442 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-08-07 12:45:47,444 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-08-07 12:45:47,476 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.14.0 [95f808d9-d641-11ea-9c48-005056073c33] stopped, uptime 1 minute
tail: /opt/mybroker-broker/log/artemis.log: file truncated
2020-08-07 12:46:07,212 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-08-07 12:46:07,242 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-08-07 12:46:09,516 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-08-07 12:46:09,542 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 5,368,709,120
2020-08-07 12:46:09,589 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-08-07 12:46:09,590 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-08-07 12:46:09,591 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-08-07 12:46:09,592 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-08-07 12:46:09,592 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-08-07 12:46:09,593 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-08-07 12:46:09,776 WARN [org.apache.activemq.artemis.core.server] AMQ222035: Directory /opt/mybroker-broker/data/paging/9fd77264-d6db-11ea-bb8e-005056073c33 did not have an identification file address.txt
2020-08-07 12:46:13,695 WARN [org.apache.activemq.artemis.core.server] AMQ222035: Directory /opt/mybroker-broker/data/paging/9fd77264-d6db-11ea-bb8e-005056073c33 did not have an identification file address.txt
...when I do the following:
Stop the SLAVE
Restart the MASTER
Start the SLAVE
Using Jolokia I see that the SLAVE is connected to the MASTER after some time.
Questions:
What does the ERROR message mean?
Is there an CLI command to get some information about synchronization?
Is there an CLI command to get some system/cluster status information?
The fundamental problem is noted in the log from the master broker:
2020-08-07 12:45:46,615 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: java.io.FileNotFoundException: /opt/mybroker-broker/data/paging/9fbeba62-d6db-11ea-bb8e-005056073c33/000000003.page (Too many open files)
This is an environmental error indicating that the operating system will not allow the process (i.e. the broker) to open any more files.
Since the broker can't open all the files it needs then it can't properly synchronize its data with the backup. Furthermore, the inability to open additional files is interpreted as a "critical" error which forces the broker to shut itself down. When the master broker shuts down the slave activates, but since the master was not able to complete the initial replication with the slave the slave states:
2020-08-07 12:45:50,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
The slave is simply logging the fact that it didn't complete the initial synchronization/replication process with the master before the master was shut down.
Ultimately you need to increase the file limit for the user running the broker. A quick web search should help you with the specifics of that task for your particular operating system.
Related
Environment:
OS: Ubuntu 20.04
Java: OpenJDK 11.0.12
ActiveMQ Artemis: 2.25.0
My MQ has a big data as below:
root#mdmcs22p:~# du -h -d 1 /data/artemis/data
4.4G /data/artemis/data/large-messages
104G /data/artemis/data/journal
2.1M /data/artemis/data/bindings
500G /data/artemis/data/paging
608G /data/artemis/data
root#mdmcs22p:~#
For some reason I need to restart it, so I run below command in bin folder of the broker:
./artemis-service restart
After a while, artemis dumps some exception logs. Eventually, it still completes the start action.
Does this exception matter ? And how to give Artemis more time to start JournalStorageManager without this exception ?
2022-09-30 06:06:30,680 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue mdms.lpr.kma in AddressSettings
2022-09-30 06:06:30,681 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue mdms.lpi.bduser in AddressSettings
2022-09-30 06:06:30,681 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue mdms.lpi.bduser in AddressSettings
2022-09-30 06:06:30,681 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue mdms.lpr.sadms in AddressSettings
2022-09-30 06:06:30,682 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue mdms.lpr.sadms in AddressSettings
2022-09-30 06:06:30,682 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue mdms.lpi.edwh in AddressSettings
2022-09-30 06:06:30,682 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue mdms.lpi.edwh in AddressSettings
2022-09-30 06:09:30,003 WARN [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager is expired on path 0
2022-09-30 06:09:30,004 WARN [org.apache.activemq.artemis.core.server] AMQ224116: The component org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager#219f4597 is not responsive during start up. The Server may be taking too long to start
2022-09-30 06:09:30,816 WARN [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump: *******************************************************************************
Complete Thread dump
"main" Id=1 RUNNABLE
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:612)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:2136)
- locked org.apache.activemq.artemis.core.journal.impl.JournalImpl#36073cbf
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:2429)
- locked org.apache.activemq.artemis.core.journal.impl.JournalImpl#36073cbf
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1739)
- locked org.apache.activemq.artemis.core.journal.impl.JournalImpl#36073cbf
at org.apache.activemq.artemis.core.journal.Journal.load(Journal.java:278)
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:908)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:3650)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3313)
- locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl#5930b0c4
at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:76)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:671)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:579)
- locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl#5930b0c4
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:64)
- locked org.apache.activemq.artemis.integration.FileBroker#3fec39f6
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:127)
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:160)
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:108)
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:135)
at java.base#11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base#11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base#11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base#11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
at app//org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134)
at app//org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50)
"Reference Handler" Id=2 RUNNABLE
at java.base#11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
at java.base#11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
at java.base#11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock#4d2d1b79
at java.base#11.0.16/java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock#4d2d1b79
at java.base#11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
at java.base#11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
at java.base#11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
....
....
===============================================================================
End Thread dump
*******************************************************************************
2022-09-30 06:11:42,593 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 25% loaded
2022-09-30 06:11:53,330 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 50% loaded
2022-09-30 06:12:01,382 INFO [org.apache.activemq.artemis.core.server] AMQ221014: 76% loaded
2022-09-30 06:12:11,906 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2022-09-30 06:12:11,907 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2022-09-30 06:12:11,912 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2022-09-30 06:12:11,912 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2022-09-30 06:12:11,913 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address IT_RECV_QUEUE supporting [ANYCAST]
2022-09-30 06:12:11,913 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue IT_RECV_QUEUE on address IT_RECV_QUEUE
2022-09-30 06:12:11,913 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address IT_SEND_QUEUE supporting [ANYCAST]
2022-09-30 06:12:11,914 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue IT_SEND_QUEUE on address IT_SEND_QUEUE
2022-09-30 06:12:11,914 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address IT_APIC_REQUEST_QUEUE supporting [ANYCAST]
2022-09-30 06:12:11,914 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue IT_APIC_REQUEST_QUEUE on address IT_APIC_REQUEST_QUEUE
2022-09-30 06:12:11,914 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address IT_APIC_RESPONSE_QUEUE supporting [ANYCAST]
2022-09-30 06:12:11,914 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue IT_APIC_RESPONSE_QUEUE on address IT_APIC_RESPONSE_QUEUE
2022-09-30 06:12:11,915 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.CustnoTransRelation.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,915 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.CustnoTransRelationTrans.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,915 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.Feeder.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,915 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.PlannedOutage.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,916 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.PlanOutageStatusChange.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,916 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.Transformer.to.MAM supporting [MULTICAST]
2022-09-30 06:12:11,916 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address OMS.OutageRestoreQueryReply.to.STM supporting [MULTICAST]
2022-09-30 06:12:11,916 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address STM.NonPlanOutage.to.OMS supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address STM.OutageRestoreQuery.to.OMS supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address STM.TransformerOutage.to.OMS supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address STM.OutageRestore.to.OMS supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.all supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.all_to_ot supporting [MULTICAST]
2022-09-30 06:12:11,917 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.dreams supporting [ANYCAST]
2022-09-30 06:12:11,918 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.dreams on address mdms.lpi.dreams
2022-09-30 06:12:11,918 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.bduser supporting [ANYCAST]
2022-09-30 06:12:11,918 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.bduser on address mdms.lpi.bduser
2022-09-30 06:12:11,918 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.edwh supporting [ANYCAST]
2022-09-30 06:12:11,918 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.edwh on address mdms.lpi.edwh
2022-09-30 06:12:11,919 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.bduser2 supporting [ANYCAST]
2022-09-30 06:12:11,919 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.bduser2 on address mdms.lpi.bduser2
2022-09-30 06:12:11,919 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.gems supporting [ANYCAST]
2022-09-30 06:12:11,919 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.gems on address mdms.lpi.gems
2022-09-30 06:12:11,919 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.nbs supporting [ANYCAST]
2022-09-30 06:12:11,920 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.nbs on address mdms.lpi.nbs
2022-09-30 06:12:11,920 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.nbs2 supporting [ANYCAST]
2022-09-30 06:12:11,920 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.nbs2 on address mdms.lpi.nbs2
2022-09-30 06:12:11,920 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.daps supporting [ANYCAST]
2022-09-30 06:12:11,921 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.daps on address mdms.lpi.daps
2022-09-30 06:12:11,921 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.kma supporting [ANYCAST]
2022-09-30 06:12:11,921 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.kma on address mdms.lpi.kma
2022-09-30 06:12:11,921 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.sadms supporting [ANYCAST]
2022-09-30 06:12:11,922 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.sadms on address mdms.lpi.sadms
2022-09-30 06:12:11,922 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.test_it supporting [ANYCAST]
2022-09-30 06:12:11,922 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.test_it on address mdms.lpi.test_it
2022-09-30 06:12:11,922 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.test_ot supporting [ANYCAST]
2022-09-30 06:12:11,923 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.test_ot on address mdms.lpi.test_ot
2022-09-30 06:12:11,923 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.amtfi supporting [ANYCAST]
2022-09-30 06:12:11,923 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.amtfi on address mdms.lpi.amtfi
2022-09-30 06:12:11,923 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ttu.sadms supporting [ANYCAST]
2022-09-30 06:12:11,923 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ttu.sadms on address ttu.sadms
2022-09-30 06:12:11,924 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address fci.sadms supporting [ANYCAST]
2022-09-30 06:12:11,924 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue fci.sadms on address fci.sadms
2022-09-30 06:12:11,924 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpi.sadms supporting [ANYCAST]
2022-09-30 06:12:11,924 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpi.sadms on address mdms.lpi.sadms
2022-09-30 06:12:11,925 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.event.sadms supporting [ANYCAST]
2022-09-30 06:12:11,925 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.event.sadms on address mdms.event.sadms
2022-09-30 06:12:11,925 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpr.all supporting [MULTICAST]
2022-09-30 06:12:11,925 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpr.kma supporting [ANYCAST]
2022-09-30 06:12:11,925 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpr.kma on address mdms.lpr.kma
2022-09-30 06:12:11,926 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.lpr.sadms supporting [ANYCAST]
2022-09-30 06:12:11,926 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue mdms.lpr.sadms on address mdms.lpr.sadms
2022-09-30 06:12:11,926 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.event.public supporting [MULTICAST]
2022-09-30 06:12:11,926 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.ds.LoadProfile.22.Delivered supporting [MULTICAST]
2022-09-30 06:12:11,926 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.ds.LoadProfile.22.Received supporting [MULTICAST]
2022-09-30 06:12:11,927 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.ds.MeterAssets supporting [MULTICAST]
2022-09-30 06:12:11,927 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address mdms.ds.MeterEvents supporting [MULTICAST]
2022-09-30 06:12:11,927 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address test.TestTopic supporting [MULTICAST]
2022-09-30 06:12:11,927 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address test.TestQueue supporting [ANYCAST]
2022-09-30 06:12:11,927 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue test.TestQueue on address test.TestQueue
2022-09-30 06:12:12,241 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 192.168.11.64:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2022-09-30 06:12:12,246 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 192.168.11.64:5672 for protocols [AMQP]
2022-09-30 06:12:12,250 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2022-09-30 06:12:12,251 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.25.0 [broker_1164, nodeID=f6119a4a-3fc1-11ed-9fe0-00505694e350]
2022-09-30 06:12:12,262 INFO [org.apache.activemq.artemis] AMQ241003: Starting embedded web server
2022-09-30 06:12:12,562 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2022-09-30 06:12:12,606 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2022-09-30 06:12:12,858 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2022-09-30 06:12:12,870 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2022-09-30 06:12:12,873 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.15.0
2022-09-30 06:12:12,879 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2022-09-30 06:12:12,884 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
2022-09-30 06:12:12,895 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2022-09-30 06:12:12,900 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/broker_1164/etc/jolokia-access.xml]
2022-09-30 06:12:12,979 INFO [org.apache.activemq.artemis] AMQ241001: Embedded web server started at http://0.0.0.0:8161
2022-09-30 06:12:12,981 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2022-09-30 06:12:12,981 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
If you know you have a large journal that's going to take a long time to load then this WARN message and thread dump should not be a concern. The WARN message is emitted by the critical analyzer which monitors critical broker functions and emits WARN messages in some circumstances and can even stop the broker completely if the problem is critical enough. As noted in the documentation you can tune how the critical analyzer functions. In your case you may want to increase critical-analyzer-timeout in broker.xml.
That said, you have a lot of data on disk. The broker is designed for data to flow through it. It is not meant to be a data repository like a relational database. The production and consumption of messages should be relatively balanced so that messages don't accumulate. The broker can certainly buffer messages if consumption falls behind for a bit, but eventually consumption must resume and catch up to eliminate any backlogs.
I've got two EC2 instances running Artemis in the master-slave replication setup and I always perform rolling upgrade by shutting down the slave, upgrading it and starting it again. Then I do the same with the master.
This no longer works for me while upgrading from 2.17 to 2.18. After upgrading one of the nodes I always get this message on that node:
AMQ214013: Failed to decode packet: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
No matter what I do. I even tried to setup a fresh new cluster only adding replication related bits of configuration:
Master
<connectors>
<connector name="broker-master">tcp://10.35.4.16:61616</connector>
<connector name="broker-slave">tcp://10.35.4.211:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="cluster">
<connector-ref>broker-master</connector-ref>
<check-period>5000</check-period>
<static-connectors>
<connector-ref>broker-slave</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<cluster-name>cluster</cluster-name>
<group-name>rs1</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave
<connectors>
<connector name="broker-master">tcp://10.35.4.16:61616</connector>
<connector name="broker-slave">tcp://10.35.4.211:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="cluster">
<connector-ref>broker-slave</connector-ref>
<check-period>5000</check-period>
<static-connectors>
<connector-ref>broker-master</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<cluster-name>cluster</cluster-name>
<group-name>rs1</group-name>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
Is there anything wrong with my configuration, or is it just not possible to from 2.17 to 2.18 like this?
Full log from master
...
2022-04-07 10:00:17,610 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
...
2022-04-07 10:00:18,726 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2022-04-07 10:00:18,728 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2022-04-07 10:00:18,759 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2022-04-07 10:00:18,760 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2022-04-07 10:00:19,072 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2022-04-07 10:00:19,075 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5445 for protocols [HORNETQ,STOMP]
2022-04-07 10:00:19,079 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5672 for protocols [AMQP]
2022-04-07 10:00:19,082 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1883 for protocols [MQTT]
2022-04-07 10:00:19,085 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61613 for protocols [STOMP]
2022-04-07 10:00:19,086 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2022-04-07 10:00:19,086 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.17.0 [broker-master, nodeID=eb08fe05-b640-11ec-b6e2-06e66cf4b718]
2022-04-07 10:00:19,470 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2022-04-07 10:00:19,590 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2022-04-07 10:00:20,032 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2022-04-07 10:00:20,123 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2022-04-07 10:00:20,126 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.11.0
2022-04-07 10:00:20,137 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2022-04-07 10:00:20,171 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2022-04-07 10:00:20,182 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/broker-master/etc/jolokia-access.xml]
2022-04-07 10:00:20,342 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://10.35.4.16:8161
2022-04-07 10:00:20,343 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://10.35.4.16:8161/console/jolokia
2022-04-07 10:00:20,343 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://10.35.4.16:8161/console
2022-04-07 10:00:52,429 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile{activemq-data-49.amq, opened=false, pendingClose=false, pendingCallbacks=org.apache.activemq.artemis.utils.AutomaticLatch#2907ccd6} (size=10,485,760) to replica.
2022-04-07 10:00:52,931 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-34.bindings (size=1,048,576) to replica.
2022-04-07 10:00:52,963 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-35.bindings (size=1,048,576) to replica.
2022-04-07 10:00:53,011 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/broker-master/data/bindings/activemq-bindings-1.bindings (size=1,048,576) to replica.
2022-04-07 10:00:53,073 WARN [org.apache.activemq.artemis.core.server] AMQ222092: Connection to the backup node failed, removing replication now: ActiveMQRemoteDisconnectException[errorType=REMOTE_DISCONNECT message=null]
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:582) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor$Listener.connectionDestroyed(NettyAcceptor.java:942) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.lambda$channelInactive$0(ActiveMQChannelHandler.java:89) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.17.0.jar:2.17.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
2022-04-07 10:01:23,032 WARN [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,035 WARN [org.apache.activemq.artemis.core.server] AMQ222251: Unable to start replication: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,045 WARN [org.apache.activemq.artemis.core.server] AMQ222013: Error when trying to start replication: ActiveMQReplicationTimeooutException[errorType=REPLICATION_TIMEOUT_ERROR message=AMQ229114: Replication synchronization process timed out after waiting 30,000 milliseconds]
at org.apache.activemq.artemis.core.replication.ReplicationManager.sendSynchronizationDone(ReplicationManager.java:660) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:717) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:180) [artemis-server-2.17.0.jar:2.17.0]
at java.base/java.lang.Thread.run(Thread.java:829) [java.base:]
2022-04-07 10:01:23,058 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected): java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1848) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:648) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:433) [artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.17.0.jar:2.17.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
2022-04-07 10:01:23,297 INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter
2022-04-07 10:01:23,303 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2022-04-07 10:01:23,325 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2022-04-07 10:01:23,336 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2022-04-07 10:01:23,384 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.17.0 [eb08fe05-b640-11ec-b6e2-06e66cf4b718] stopped, uptime 1 minute
Full log from slave
2022-04-07 10:00:50,645 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2022-04-07 10:00:50,703 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2022-04-07 10:00:50,747 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/broker-slave/data/bindings/oldreplica.7
2022-04-07 10:00:50,750 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/broker-slave/data/bindings to /opt/artemis/broker-slave/data/bindings/oldreplica.9
2022-04-07 10:00:50,753 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/broker-slave/data/journal/oldreplica.8
2022-04-07 10:00:50,755 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/broker-slave/data/journal to /opt/artemis/broker-slave/data/journal/oldreplica.10
2022-04-07 10:00:50,888 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2022-04-07 10:00:51,031 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2022-04-07 10:00:51,124 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2022-04-07 10:00:51,323 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2022-04-07 10:00:51,326 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2022-04-07 10:00:51,328 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2022-04-07 10:00:51,329 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2022-04-07 10:00:51,330 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2022-04-07 10:00:51,330 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2022-04-07 10:00:51,538 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2022-04-07 10:00:51,685 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2022-04-07 10:00:52,052 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.18.0 [null] started, waiting live to fail before it gets active
2022-04-07 10:00:52,224 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2022-04-07 10:00:52,337 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2022-04-07 10:00:52,341 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.13.5
2022-04-07 10:00:52,351 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2022-04-07 10:00:52,383 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2022-04-07 10:00:52,392 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/broker-slave/etc/jolokia-access.xml]
2022-04-07 10:00:52,570 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8161
2022-04-07 10:00:52,571 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://localhost:8161/console/jolokia
2022-04-07 10:00:52,571 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8161/console
2022-04-07 10:00:53,029 ERROR [org.apache.activemq.artemis.core.client] AMQ214013: Failed to decode packet: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readByte(ChannelBufferWrapper.java:300) [artemis-commons-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ReplicationStartSyncMessage.decodeRest(ReplicationStartSyncMessage.java:158) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.18.0.jar:2.18.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.18.0.jar:2.18.0]
2022-04-07 10:00:53,031 ERROR [org.apache.activemq.artemis.core.client] AMQ214031: Failed to decode buffer, disconnect immediately.: java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:401) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.18.0.jar:2.18.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.18.0.jar:2.18.0]
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57)
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) [netty-all-4.1.66.Final.jar:4.1.66.Final]
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readByte(ChannelBufferWrapper.java:300) [artemis-commons-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ReplicationStartSyncMessage.decodeRest(ReplicationStartSyncMessage.java:158) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371) [artemis-core-client-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388) [artemis-core-client-2.18.0.jar:2.18.0]
... 20 more
2022-04-07 10:00:53,052 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: java.lang.IndexOutOfBoundsException: readerIndex(57) + length(1) exceeds writerIndex(57): PooledUnsafeDirectByteBuf(ridx: 57, widx: 57, cap: 57) [code=GENERIC_EXCEPTION]
2022-04-07 10:00:53,092 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2022-04-07 10:00:53,093 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2022-04-07 10:00:53,093 INFO [org.apache.activemq.artemis.core.server] AMQ221083: ignoring quorum vote as max cluster size is 1.
2022-04-07 10:00:53,094 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2022-04-07 10:00:53,113 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ229026: Backup Server was not yet in sync with live]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:302) [artemis-server-2.18.0.jar:2.18.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4271) [artemis-server-2.18.0.jar:2.18.0]
2022-04-07 10:01:23,105 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2022-04-07 10:01:23,107 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.4.16/10.35.4.16:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2022-04-07 10:01:42,243 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-04-07 10:01:42,274 INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter
2022-04-07 10:01:42,278 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2022-04-07 10:01:42,296 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2022-04-07 10:01:42,304 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
This bug was introduced via ARTEMIS-3340. It should be fixed in the next release (i.e. 2.22.0). Until then there's no work-around unfortunately.
You may consider creating a completely separate new live/backup pair and then using a DNS update to direct clients from the old pair to the new pair.
Short version
When I run ActiveMQ Artemis in docker I see this basically empty screen:
That doesn't look right... I was expecting this, like I get when using the zip file:
Regardless of whether I use docker or the zip file, it doesn't matter what username or password I enter, I just get logged in regardless, which is a little concerning...
What am I doing wrong?
Longer Version
I'm attempting a "Hello World" style installation of ActiveMQ. It sounds like ActiveMQ Artemis is what I should be using. We'll be using this on Kubernetes, so I found and have followed https://artemiscloud.io/. There is a Quickly deploy a basic Container image that runs the broker right there on the front page. It suggests:
docker run -e AMQ_USER=admin -e AMQ_PASSWORD=admin -p80:8161 --name artemis quay.io/artemiscloud/activemq-artemis-broker:dev.latest
I changed the port to -p8161:8161 and run it using Docker for Windows. Visiting http://localhost:8161/console/ shows me the first screenshot above.
If I instead download from https://activemq.apache.org/components/artemis/download/ and follow https://activemq.apache.org/components/artemis/documentation/latest/using-server.html I get the second screenshot.
Regardless of whether I use docker or the zip file, it doesn't matter what username or password I enter, I just get logged in regardless.
Full docker console log
ยป docker run -e AMQ_USER=admin -e AMQ_PASSWORD=admin -p8161:8161 --name artemis quay.io/artemiscloud/activemq-artemis-broker:latest
Creating Broker with args --user XXXXX --password XXXXX --role admin --name broker --allow-anonymous --http-host 172.17.0.3 --host 172.17.0.3 --force
Creating ActiveMQ Artemis instance at: /home/jboss/broker
Auto tuning journal ...
done! Your system can make 1.44 writes per millisecond, your journal-buffer-timeout will be 696000
You can now start the broker by executing:
"/home/jboss/broker/bin/artemis" run
Or you can run the broker in the background using:
"/home/jboss/broker/bin/artemis-service" start
Running Broker
_ _ _
/ \ ____| |_ ___ __ __(_) _____
/ _ \| _ \ __|/ _ \ \/ | |/ __/
/ ___ \ | \/ |_/ __/ |\/| | |\___ \
/_/ \_\| \__\____|_| |_|_|/___ /
Apache ActiveMQ Artemis 2.17.0
2021-05-27 21:23:44,556 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2021-05-27 21:23:44,605 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2021-05-27 21:23:44,650 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2021-05-27 21:23:44,694 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2021-05-27 21:23:44,712 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2021-05-27 21:23:44,712 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2021-05-27 21:23:44,713 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2021-05-27 21:23:44,713 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2021-05-27 21:23:44,713 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2021-05-27 21:23:44,714 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2021-05-27 21:23:44,759 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock
2021-05-27 21:23:44,760 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock
2021-05-27 21:23:44,832 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2021-05-27 21:23:44,839 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2021-05-27 21:23:44,880 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2021-05-27 21:23:44,882 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2021-05-27 21:23:45,148 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 172.17.0.3:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2021-05-27 21:23:45,152 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 172.17.0.3:5445 for protocols [HORNETQ,STOMP]
2021-05-27 21:23:45,155 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 172.17.0.3:5672 for protocols [AMQP]
2021-05-27 21:23:45,158 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 172.17.0.3:1883 for protocols [MQTT]
2021-05-27 21:23:45,162 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 172.17.0.3:61613 for protocols [STOMP]
2021-05-27 21:23:45,167 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2021-05-27 21:23:45,168 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.17.0 [broker, nodeID=d1043ae4-bf31-11eb-8c49-0242ac110003]
2021-05-27 21:23:45,379 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2021-05-27 21:23:45,413 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2021-05-27 21:23:45,625 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2021-05-27 21:23:45,636 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2021-05-27 21:23:45,638 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.11.0
2021-05-27 21:23:45,643 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "admin" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2021-05-27 21:23:45,656 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2021-05-27 21:23:45,660 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/home/jboss/broker/etc/jolokia-access.xml]
2021-05-27 21:23:45,727 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://172.17.0.3:8161
2021-05-27 21:23:45,727 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://172.17.0.3:8161/console/jolokia
2021-05-27 21:23:45,728 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://172.17.0.3:8161/console
2021-05-27 21:23:54,289 INFO [io.hawt.web.auth.LoginServlet] Hawtio login is using 1800 sec. HttpSession timeout
2021-05-27 21:23:54,968 INFO [io.hawt.web.auth.keycloak.KeycloakServlet] Keycloak integration is disabled
2021-05-27 21:24:00,270 INFO [io.hawt.web.auth.LoginServlet] Logging in user: admin
The ArtemisCloud container image for ActiveMQ Artemis is designed to run inside a container so the container IP address should be used to access to the console or to other resources.
The container IP address can be obtained by using the command docker inspect or by reading the container log, ie:
[org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://172.17.0.3:8161/console
I'm trying to run a 6 node Apache Artemis static cluster as 3 live-backup pairs. In order not to lose data during full cluster restart I've also created a mechanism figuring out which of the pair has newer data, and based on that the server automatically decides whether it's safe to start that particular node. During full cluster restart the servers are started simultaneously, and nodes with newer data start before nodes that were lately acting as backups.
This scenario has been working flawlessly with single live-backup pair.
Shutdown:
Master node is live, slave node is backup
Master node => slave node goes live
Slave node
Start:
Both nodes are dead
Slave node => goes live
Master node => goes live, slave node goes backup
However in the 6 node cluster the backup node does not go live at all:
3 live nodes, 3 backup nodes; pairs bound together in separate groups (rs1, rs2, rs3)
Stop rs1 master node => rs1 slave node goes live
Stop rs1 slave node
Start rs1 slave node => never goes live
Must not start rs1 master node because of potential data loss
Any idea what's wrong with my setup?
Master HA configuration (the group name varies):
<?xml version="1.0" encoding="UTF-8"?>
<ha-policy xmlns="urn:activemq:core">
<replication>
<master>
<cluster-name>clouedi-mq-dev</cluster-name>
<group-name>rs1</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave HA configuration (the group name varies):
<?xml version="1.0" encoding="UTF-8"?>
<ha-policy xmlns="urn:activemq:core">
<replication>
<slave>
<cluster-name>clouedi-mq-dev</cluster-name>
<group-name>rs1</group-name>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
Thank you.
Before the issue
Master
2020-11-30 12:34:54,021 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:34:54,082 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:34:59,425 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:34:59,429 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:34:59,435 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:34:59,487 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
...address deployment...
2020-11-30 12:35:00,123 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,STOMP]
2020-11-30 12:35:00,124 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-11-30 12:35:00,124 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.15.0 [edimq-broker-master-dev-az1-1, nodeID=6937ad47-309c-11eb-811e-0275ba22b614]
2020-11-30 12:35:00,315 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#58d025c9 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#7c398410 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#58d025c9 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#7c398410 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1779219567[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-master-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-96, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:35:00,318 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#77158457 [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#2f988822 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#77158457 [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#2f988822 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1779219567[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-master-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-96, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:35:00,688 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:35:00,766 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:35:01,187 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:35:01,196 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:35:01,199 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:35:01,203 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-master-dev-az1-1/tmp/uploads
2020-11-30 12:35:01,217 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:35:01,240 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-master-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:35:01,269 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:35:01,389 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.96, edimq-broker-master-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:35:01,722 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:35:01,722 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:35:01,723 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-11-30 12:35:18,072 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/artemis/edimq-broker-master-dev-az1-1/data/journal/activemq-data-45.amq (size=10,485,760) to replica.
2020-11-30 12:35:18,687 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-2.bindings (size=1,048,576) to replica.
2020-11-30 12:35:18,697 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-50.bindings (size=1,048,576) to replica.
2020-11-30 12:35:18,709 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending NIOSequentialFile /opt/artemis/edimq-broker-master-dev-az1-1/data/bindings/activemq-bindings-37.bindings (size=1,048,576) to replica.
Slave
2020-11-30 12:35:16,773 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:35:16,847 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:35:16,976 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.44
2020-11-30 12:35:16,978 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal to /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.46
2020-11-30 12:35:17,039 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:35:17,101 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:35:17,107 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:35:17,263 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:35:17,356 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:35:17,405 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-11-30 12:35:17,863 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [null] started, waiting live to fail before it gets active
2020-11-30 12:35:17,974 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:35:17,985 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:35:17,987 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:35:17,992 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-slave-dev-az1-1/tmp/uploads
2020-11-30 12:35:18,021 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:35:18,058 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-slave-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:35:18,089 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:35:18,216 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.101, edimq-broker-slave-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:35:18,581 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:35:18,582 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:35:18,582 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2020-11-30 12:35:18,964 INFO [org.apache.activemq.artemis.core.server] AMQ221024: Backup server ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614 is synchronized with live-server.
2020-11-30 12:35:18,981 INFO [org.apache.activemq.artemis.core.server] AMQ221031: backup announced
Shutting down
Master
2020-11-30 12:38:48,621 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:38:48,632 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:38:48,644 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe
2020-11-30 12:38:48,644 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4
2020-11-30 12:38:48,796 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-11-30 12:38:48,800 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-11-30 12:38:48,849 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-11-30 12:38:48,853 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-11-30 12:38:48,877 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.15.0 [6937ad47-309c-11eb-811e-0275ba22b614] stopped, uptime 3 minutes
Slave goes live
2020-11-30 12:38:48,621 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2020-11-30 12:38:48,622 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 2 quorum votes
2020-11-30 12:38:48,623 INFO [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results.
2020-11-30 12:38:48,637 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.2.96/10.35.2.96:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-11-30 12:38:48,644 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to 10.35.2.96/10.35.2.96:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-11-30 12:38:48,666 INFO [org.apache.activemq.artemis.core.server] AMQ221060: Sending quorum vote request to 10.35.2.20/10.35.2.20:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=false]
2020-11-30 12:38:48,667 INFO [org.apache.activemq.artemis.core.server] AMQ221060: Sending quorum vote request to 10.35.2.206/10.35.2.206:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=false]
2020-11-30 12:38:48,675 INFO [org.apache.activemq.artemis.core.server] AMQ221061: Received quorum vote response from 10.35.2.206/10.35.2.206:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=true]
2020-11-30 12:38:48,676 INFO [org.apache.activemq.artemis.core.server] AMQ221061: Received quorum vote response from 10.35.2.20/10.35.2.20:61616: ServerConnectVote [nodeId=6937ad47-309c-11eb-811e-0275ba22b614, vote=true]
2020-11-30 12:38:48,677 INFO [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes.
2020-11-30 12:38:48,682 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2020-11-30 12:38:48,705 INFO [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614 to become 'live'
2020-11-30 12:38:48,722 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
...address deployment...
2020-11-30 12:38:49,217 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-11-30 12:38:49,238 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,STOMP]
2020-11-30 12:38:49,269 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#50794972 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#13fa6cc targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#50794972 [name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#13fa6cc targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1854054128[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-slave-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-101, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-2, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-206], discoveryGroupConfiguration=null]] is connected
2020-11-30 12:38:49,270 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge#39f7703c [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#66eabf5 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#39f7703c [name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, queue=QueueImpl[name=$.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614], temp=false]#66eabf5 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1854054128[nodeUUID=6937ad47-309c-11eb-811e-0275ba22b614, connector=TransportConfiguration(name=edimq-broker-slave-dev-az1-1, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-101, address=, server=ActiveMQServerImpl::serverUUID=6937ad47-309c-11eb-811e-0275ba22b614])) [initialConnectors=[TransportConfiguration(name=edimq-broker-master-dev-az1-3, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=10-35-2-20], discoveryGroupConfiguration=null]] is connected
Shutting down slave
2020-11-30 12:39:56,693 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:39:56,694 WARN [org.apache.activemq.artemis.core.server] AMQ222294:
**************************************************************************************************************************************************************************************************************************************************************
There is a possible split brain on nodeID 6937ad47-309c-11eb-811e-0275ba22b614, coming from connectors 6937ad47-309c-11eb-811e-0275ba22b614. Topology update ignored.
**************************************************************************************************************************************************************************************************************************************************************
2020-11-30 12:39:56,704 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.88c93073-309c-11eb-9fb4-02ca1af23dbe
2020-11-30 12:39:56,711 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.clouedi-mq-dev.793500b2-309c-11eb-998e-02436e93e5b4
2020-11-30 12:39:56,859 INFO [io.hawt.HawtioContextListener] Destroying hawtio services
2020-11-30 12:39:56,863 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter
2020-11-30 12:39:56,914 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2020-11-30 12:39:56,918 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2020-11-30 12:39:56,946 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.15.0 [6937ad47-309c-11eb-811e-0275ba22b614] stopped, uptime 4 minutes
Starting up
Slave
2020-11-30 12:40:34,153 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-11-30 12:40:34,209 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2020-11-30 12:40:34,240 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings/oldreplica.19
2020-11-30 12:40:34,246 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings to /opt/artemis/edimq-broker-slave-dev-az1-1/data/bindings/oldreplica.21
2020-11-30 12:40:34,248 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.45
2020-11-30 12:40:34,248 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal to /opt/artemis/edimq-broker-slave-dev-az1-1/data/journal/oldreplica.47
2020-11-30 12:40:34,249 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging/oldreplica.19
2020-11-30 12:40:34,257 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging to /opt/artemis/edimq-broker-slave-dev-az1-1/data/paging/oldreplica.21
2020-11-30 12:40:34,321 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-11-30 12:40:34,402 WARN [org.apache.activemq.artemis.core.server] AMQ222007: Security risk! Apache ActiveMQ Artemis is running with the default cluster admin user and default password. Please see the cluster chapter in the ActiveMQ Artemis User Guide for instructions on how to change this.
2020-11-30 12:40:34,413 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-11-30 12:40:34,607 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-11-30 12:40:34,774 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2020-11-30 12:40:34,888 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-11-30 12:40:34,927 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [null] started, waiting live to fail before it gets active
2020-11-30 12:40:35,301 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2020-11-30 12:40:35,310 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-11-30 12:40:35,312 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-11-30 12:40:35,320 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/artemis/edimq-broker-slave-dev-az1-1/tmp/uploads
2020-11-30 12:40:35,339 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-11-30 12:40:35,365 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/artemis/edimq-broker-slave-dev-az1-1/etc//jolokia-access.xml]
2020-11-30 12:40:35,392 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-11-30 12:40:35,511 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.35.2.101, edimq-broker-slave-dev-az1-1.dc01.clouedi.local]
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-11-30 12:40:35,819 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
The above shows that the slave does not start at all.
When ActiveMQ Artemis uses replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network.
Upon start-up the backup server will first need to synchronize all existing data from the live server before becoming capable of replacing the live server should it fail. So unlike when using shared storage, a replicating backup will not be a fully operational backup right after start-up, but only after it finishes synchronizing the data with its live server.
Why does a single live-backup pair look working?
The backup node at start-up executes the following steps at SharedNothingBackupActivation:
Initialize (it looks running)
Waiting on cluster connection (it is starving here)
Starting backup manager
So in a scenario with a single live-backup pair the backup node doesn't complete the initialization.
I have just install a new artemis in my new server, able to run the service normally but not able to access from outside.
I also have followed this question that seems have the similar issue with me, but
seems not able to work.
Artemis-2.6.3 console: Service Unavailable
my current jolokia-access :
<restrict>
<remote>
<host>localhost</host>
<host>10.85.102.148</host><!--This is my computer ip-->
</remote>
<!--
<cors>
<allow-origin>*://localhost*</allow-origin>
<strict-checking/>
</cors>
-->
</restrict>
current logging
2019-04-25 20:16:07,013 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2019-04-25 20:16:07,046 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2019-04-25 20:16:07,080 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2019-04-25 20:16:07,133 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2019-04-25 20:16:07,162 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2019-04-25 20:16:07,163 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2019-04-25 20:16:07,164 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2019-04-25 20:16:07,164 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2019-04-25 20:16:07,165 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2019-04-25 20:16:07,166 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2019-04-25 20:16:07,222 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock
2019-04-25 20:16:07,223 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock
2019-04-25 20:16:08,143 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2019-04-25 20:16:08,144 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2019-04-25 20:16:08,513 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61619 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE]
2019-04-25 20:16:08,521 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5446 for protocols [HORNETQ,STOMP]
2019-04-25 20:16:08,536 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:5673 for protocols [AMQP]
2019-04-25 20:16:08,540 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:1884 for protocols [MQTT]
2019-04-25 20:16:08,545 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61614 for protocols [STOMP]
2019-04-25 20:16:08,547 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2019-04-25 20:16:08,548 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.4 [0.0.0.0, nodeID=c4f07cfd-6745-11e9-bfbc-005056b0ef9d]
2019-04-25 20:16:09,647 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2019-04-25 20:16:09,749 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2019-04-25 20:16:10,199 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2019-04-25 20:16:10,269 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2019-04-25 20:16:10,273 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2019-04-25 20:16:10,298 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/CMS/dev/artemisCMSDEV2.6.4/tmp/uploads
2019-04-25 20:16:10,318 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2019-04-25 20:16:10,362 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/CMS/dev/artemisCMSDEV2.6.4/etc/jolokia-access.xml]
2019-04-25 20:16:10,403 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2019-04-25 20:16:10,581 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.92.202.52, mask.myserver.com, 192.168.46.52, mask-p1.myserver.com]
2019-04-25 20:16:10,882 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8162
2019-04-25 20:16:10,883 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://localhost:8162/console/jolokia
2019-04-25 20:16:10,883 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8162/console
According to the log the console is bound to http://localhost:8162 therefore not accessible from outside localhost. Please check the bind entry in bootstrap.xml.
Also note that you have a restriction with localhost in cors section of jolokia-access.xml what you also should adjust.
You need just to modify these two files us below :
$[broker_artemis_path]/etc/bootstrap.xml
<web bind="http://0.0.0.0:8161" path="web">
<app url="activemq-branding" war="activemq-branding.war"/>
<app url="artemis-plugin" war="artemis-plugin.war"/>
<app url="console" war="console.war"/>
</web>
and $[broker_artemis_path]/etc/jolokia-access.xml
<cors>
<!-- Allow cross origin access from localhost ... -->
<allow-origin>*://*</allow-origin>
<!-- Options from this point on are auto-generated by Create.java from the Artemis CLI -->
<!-- Check for the proper origin on the server side, too -->
<strict-checking/>
</cors>
For more information you can check this link :
https://medium.com/#hasnat.saeed/setup-activemq-artemis-on-ubuntu-18-04-76bb4975308b