Paging started before reaching max size configured in ActiveMQ Artemis 2.17.0 version.
2021-05-31 08:59:30,293 WARN [org.apache.activemq.artemis.core.server] AMQ222038: Starting paging on address 'consumer_queue'; size is currently: 208,250 bytes; max-size-bytes: 524,288,000; global-size-bytes: 1,073,744,320
Address settings configured for queue consumer_queue.
<address-setting match="consumer_queue">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>500Mb</max-size-bytes>
<page-size-bytes>10Mb</page-size-bytes>
<max-delivery-attempts>5</max-delivery-attempts>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>false</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
<auto-create-jms-queues>false</auto-create-jms-queues>
<auto-create-jms-topics>false</auto-create-jms-topics>
</address-setting>
global-max-size set as -1. The JVM is using -Xms36G and the broker is automatically configuring the global-max-size to 18G, confirmed via this log:
AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 19,327,352,832
This looks like it may be a bug. Please file an issue here, and include a minimal reproducible example.
Related
I have many queues, i.e
my.queue.no.1
my.queue.no.2
my.queue.no.3
my.queue.no.4
And I want to redirect unsuccessful messages to DLQ, but I don't want to mix messages from all queues to a one DLQ.
Is it possible to have multiple DLQs?
i.e
my.queue.no.1
my.queue.no.dlq.1
my.queue.no.2
my.queue.no.dlq.2
my.queue.no.3
my.queue.no.dlq.3
my.queue.no.4
my.queue.no.dlq.4
P.S I'm using Artemis 2.16.0
ActiveMQ Artemis can automatically create the defined dead-letter-address and a corresponding dead-letter queue when a message is undeliverable enabling the auto-create-dead-letter-resources address setting. The dead-letter-queue-prefix address setting can be used to define a prefix used for automatically created dead-letter queues, i.e. to create DLQ.my.queue.no.NNN queues under the DLA address:
<address-settings>
<address-setting match="my.queue.no.#">
<dead-letter-address>DLA</dead-letter-address>
<auto-create-dead-letter-resources>true</auto-create-dead-letter-resources>
<dead-letter-queue-prefix>DLQ.</dead-letter-queue-prefix>
...
we are currently experiencing that after some addresses/queues finished being paged, our application does not get any messages anymore. It is like as if the messages are gone. We are expecting that paging modus should not cause the disfunctionality of the cluster. Our configuration of address-settings looks as follow:
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq#">
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
<auto-delete-queues>true</auto-delete-queues>
<auto-delete-addresses>true</auto-delete-addresses>
<config-delete-addresses>FORCE</config-delete-addresses>
</address-setting>
<address-setting match="MyQueueEins">
<address-full-policy>BLOCK</address-full-policy>
<max-size-bytes>100Mb</max-size-bytes>
</address-setting>
<address-setting match="MyQueueZwei">
<address-full-policy>BLOCK</address-full-policy>
<max-size-bytes>50Mb</max-size-bytes>
</address-setting>
<address-setting match="MyQueueDrei">
<address-full-policy>BLOCK</address-full-policy>
<max-size-bytes>1Mb</max-size-bytes>
</address-setting>
<address-setting match="Onlinequeue.#">
<address-full-policy>BLOCK</address-full-policy>
<max-size-bytes>50Mb</max-size-bytes>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>10000000</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>false</auto-create-queues>
<auto-create-addresses>false</auto-create-addresses>
<auto-create-jms-queues>false</auto-create-jms-queues>
<auto-create-jms-topics>false</auto-create-jms-topics>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
<config-delete-addresses>FORCE</config-delete-addresses>
</address-setting>
</address-settings>
Are we missing something in broker.xml configuration?
What could be the cause why the messages are gone/ not being consumed after paging modus?
An address experiencing this issue is BoxDeliveryQueue. In broker.xml, this queue would have the default address-setting with # as match-regex. We are experiencing, that after paging mode on this address, the messages are not being consumed anymore. Today I found the file system we are using to store the paged messages:
<paging-directory>./data/paging</paging-directory>
I will check in the file system, whether messages exist in the defined directory for addresses. If that is the case, it would mean that the system is unable to consume the messages from the file system as stated in the documentation:
The system will navigate on the files as needed, and it will remove the page file as soon as all the messages are acknowledged up to that point.
Could it be the case, that somehow because of our configuration in broker.xml the messages directly get acknowledged after being paged, so that the are not being consumed anymore?
BoxDeliveryQueue is created also in broker.xml:
</address>
<address name="BoxDeliveryQueue">
<anycast>
<queue name="BoxDeliveryQueue">
<durable>true</durable>
</queue>
</anycast>
</address>
The consumer is a Message-Driven-Bean, pointing to the BoxDeliveryQueue
#TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
#MessageDriven(mappedName = "BoxDeliveryQueue", name = "BoxDeliveryMessageBean", activationConfig = {
#ActivationConfigProperty(propertyName = "clientId", propertyValue = "BoxDeliveryQueue"),
#ActivationConfigProperty(propertyName = "destination", propertyValue = "BoxDeliveryQueue"),
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "rebalanceConnections", propertyValue = "true"),
#ActivationConfigProperty(propertyName = "resourceAdapterJndiName", propertyValue = "delivery_ra") })
public class BoxDeliveryMessageBean implements MessageListener {
This is so wierd. The messages are unable to be consumed from the MDB only after the address finsihed being paged. If the address not being paged, everything is fine. Regarding protocols, it is configured like this:
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis"
>tcp://localhost:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;nioRemotingThreads=24</acceptor>
</acceptors>
Our JavaEE-Server is Weblogic 12c. I havent checked the thread-dumps regarding stuckthread. I assumed that it is the issue in my configuration of Artemis-System, as I assume that the MDBs are unaware whether the message directly from memory or from paged storage. I will check if there are stuck threads in application server.
Thank you,
Hadi
I have two components communicating over an jms queue in a wildfly instance. As soon as the consumer of the queue disconnects or gets stopped, the messages are forwarded to the DLQ (at least when wildfly is restarted).
Is it possible to configure wildfly to automatically redeliver the messages from DLQ as soon as a consumer reconnects to the queue?
Some details
Wildfly version: 8.2.0
standalone.xml - As far as I can tell, nothing special
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="java:/jms/queue/ExpiryQueue"/>
<durable>false</durable>
</jms-queue>
<jms-queue name="DLQ">
<entry name="java:/jms/queue/DLQ"/>
<durable>false</durable>
</jms-queue>
...
<jms-queue name="Q1-Producer-to-Consumer">
<entry name="java:/queue/Q1-Producer-to-Consumer"/>
<entry name="java:jboss/exported/queue/Q1-Producer-to-Consumer"/>
<durable>false</durable>
</jms-queue>
</jms-destinations>
Thanks.
The DLQ only gets messages that have thrown an exception during message processing. If a consumer disconnects, the messages will just still be sitting there awaiting delivery
If you are seeing an Issue whereby during a server restart messages hit the DLQ, this would suggest that your consumer is consuming messages before the resources it requires are available, so is erroring when processing the messages. You would be better to fix your consumer to not start consuming messages to early, rather than trying to fish the failed messages back from DLQ
I have an application that needs to send messages to ~30k address. I used to have OpenMQ as my broker and use JMS. The JMS client code that I have was capable of sending ~30k messages to the broker within few seconds. Then I switched to Artemis and the send performance degraded drastically. Now it takes up to 10 minutes to send all the messages to Artemis broker.
I played with different configuration settings both on the server (broker.xml) and also via connection URL.
Address setting on broker.xml
<address-setting match="#">
<!-- <dead-letter-address>DLQ</dead-letter-address> -->
<!-- <expiry-address>ExpiryQueue</expiry-address> -->
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>DROP</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
Sending
private static final String BROKER_URL = "tcp://localhost:61616?minLargeMessageSize=10485760;compressLargeMessages=true;producerWindowSize=-1";
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(BROKER_URL);
factory.setDupsOKBatchSize(50 * 1024 * 1024);
factory.setProducerWindowSize(-1);
factory.setBlockOnDurableSend(false);
factory.setBlockOnNonDurableSend(false);
factory.setCacheDestinations(true);
System.out.println("Connection factory " + factory.toString());
Connection connection = factory.createConnection();
Session producerSession = connection.createSession(false, ActiveMQJMSConstants.PRE_ACKNOWLEDGE);
MessageProducer producer = producerSession.createProducer(null);
producer.setDisableMessageID(true);
producer.setDisableMessageTimestamp(true);
// Create a JMS message of type ByteMessage, and send
producer.send(destination, jmsMessage, DeliveryMode.NON_PERSISTENT, JMS_MESSAGE_PRIORITY, DEFAULT_TTL);
With above code (simplified for readability) I only get 20 to 40 messages per second.
Am I missing something? Or is it some performance penalty introduced by JMS to Core conversion from Artemis client library?
Is there anything I can do to improve the message rate?
I would like to enable message redistribution on my 2-nodes cluster with static hosts. But it does not seem to work.
1) I have 10 producers that write to the queue "MyTestQueue" on node 1 (but no consumers).
2) I have 1 consumer on node 2 (but no producers) that consumes messages from node 2.
I expect that node 1 will redistribute the messages to node 2 where the consumer exists, but it does not. The message count on node 1 is still equal the amount of messages that was sent to node 1.
I have the following configuration in my broker.xml that sets forward-when-no-consumers to false.
I also have set redistribution-delay to a value of zero.
<jms xmlns="urn:activemq:jms">
<queue name="MyTestQueue"/>
</jms>
...
<cluster-connections>
<cluster-connection name="my-test-cluster">
<address>jms</address>
<connector-ref>server0-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<confirmation-window-size>1024</confirmation-window-size>
<static-connectors>
<connector-ref>server1-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
...
<address-settings>
<address-setting match="#">
<redelivery-delay>5000</redelivery-delay>
<redelivery-delay-multiplier>3</redelivery-delay-multiplier>
<max-redelivery-delay>10000</max-redelivery-delay>
<max-delivery-attempts>10</max-delivery-attempts>
<max-size-bytes>104857600</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<address-full-policy>PAGE</address-full-policy>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>
How can I get the message redistribution to work?
This might be related to a known issue. There is a situation on which the broker fails to load balance the messages if they don't contain the application properties field.
Could you, please, try with that?
Couple of things...
What version of Artemis are you using? Have you tried reproducing this with version 2.2.0?
What kind of client are you using (e.g. JMS, AMQP, STOMP, etc.)?
Do you have a reproducible test-case (e.g. a modified version of one of the examples shipped with Artemis)?
The configuration element <forward-when-no-consumers> is not valid in Artemis (although it was in older versions of HornetQ).
Remove the <address>jms</address> from cluster connection configuration - each cluster connection only applies to addresses that match the specified address. And make sure that you're using compatible client because messages from 1.x clients to 2.x cluster are lost when the are load balanced to nodes with matching consumers.
Here's official, working example of ActiveMQ Artemis configuration with symmetric cluster, on demand load balancing and message redistribution