Apache ActiveMQ Artemis Slow performance after paging starts - activemq-artemis

Question

We have an ActiveMQ Artemis test deployment and we noticed very slow performance after broker having a large number of messages. This is when paging starts. I hope this is normal. To mitigate this after testing we doubled the xmx for the broker. Now the paging (and performance drop) is delayed. My question is are there any other parameters beside memory which can address this.
My broker.xml is:
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core">
<ha-policy>
<replication>
<master>
<group-name>master</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<global-max-size>-1</global-max-size>
<bindings-directory>/opt/broker/broker-data/bindings</bindings-directory>
<journal-directory>/opt/broker/broker-data/journal</journal-directory>
<large-messages-directory>/opt/broker/broker-data/largemessages</large-messages-directory>
<paging-directory>/opt/broker-data/paging</paging-directory>
<journal-min-files>25</journal-min-files>
<journal-type>ASYNCIO</journal-type>
<journal-max-io>5000</journal-max-io>
<journal-sync-transactional>false</journal-sync-transactional>
<journal-sync-non-transactional>false</journal-sync-non-transactional>
<journal-buffer-timeout>750000</journal-buffer-timeout>
<connectors>
<connector name="netty-connector">tcp://node1:61616?tcpSendBufferSize=307200;tcpReceiveBufferSize=307200;writeBufferHighWaterMark=1228800;useEpoll=true;useNio=true</connector>
</connectors>
<acceptors>
<acceptor name="netty-acceptor">tcp://node1:61616?tcpSendBufferSize=307200;tcpReceiveBufferSize=307200;writeBufferHighWaterMark=1228800;useEpoll=true;useNio=true</acceptor>
</acceptors>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9875</group-port>
<broadcast-period>100</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9875</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<connection-ttl>130000</connection-ttl>
<call-timeout>120000</call-timeout>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<!-- address section -->
</core>
</configuration>
EDIT:
Most critical issue is once paging starts broker won't recover to original performance even-though majority of messages are consumed.

Answer 1

Consider that paged messages need to be synchronized to disk similarly to durable ones and the parameter to be set to control the frequency of flushes is page-sync-timeout. If no value is set, the default one is used (see the documentation for an explanation about what that setting is for).
By looking at your journal-buffer-timeout (and assuming this is correctly set) your disk seems quite slow so it's expected that paged messages won't perform great as the disk doesn't have enough IOPS.
I would first check what's the expected IOPS for random writes for your disk and will set page-sync-timeout accordingly (1/IOPS in nanoseconds), but don't expect any improvement if the disk isn't fast enough.
Additional note: If you don't care about power failure durability you can still disable journal-datasync and it should let any disk write to be able to survive just to process failures (i.e. no power failure guarantees). It should be ok if you are using shared-nothing replication, given that a backup is able to take the role in case of failure.