I'm using the Virtual Topics concepts of Activemq 5.14.5 (https://activemq.apache.org/virtual-destinations) with MQTT protocol.
ActiveMQ will pick up the messages written to the topic and write them to a queue (or multiple queues or topics). See the ${ACTIVEMQ_HOME}/conf/activemq.xml configuration below:
<beans>
<broker>
...
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="MY.QUEUE">
<forwardTo>
<queue physicalName="FOO" />
<topic physicalName="BAR" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
...
</broker>
</beans>
Using Mqtt.fx software (https://mqttfx.jensd.de/) I'm only able to dequeue from a Topic (BAR). How can I dequeue from a Queue (FOO) to see the messages arrived on it?
I am new tot this and learning about the protocol MQTT and broker Activemq.
Related
I have many queues, i.e
my.queue.no.1
my.queue.no.2
my.queue.no.3
my.queue.no.4
And I want to redirect unsuccessful messages to DLQ, but I don't want to mix messages from all queues to a one DLQ.
Is it possible to have multiple DLQs?
i.e
my.queue.no.1
my.queue.no.dlq.1
my.queue.no.2
my.queue.no.dlq.2
my.queue.no.3
my.queue.no.dlq.3
my.queue.no.4
my.queue.no.dlq.4
P.S I'm using Artemis 2.16.0
ActiveMQ Artemis can automatically create the defined dead-letter-address and a corresponding dead-letter queue when a message is undeliverable enabling the auto-create-dead-letter-resources address setting. The dead-letter-queue-prefix address setting can be used to define a prefix used for automatically created dead-letter queues, i.e. to create DLQ.my.queue.no.NNN queues under the DLA address:
<address-settings>
<address-setting match="my.queue.no.#">
<dead-letter-address>DLA</dead-letter-address>
<auto-create-dead-letter-resources>true</auto-create-dead-letter-resources>
<dead-letter-queue-prefix>DLQ.</dead-letter-queue-prefix>
...
I am new to KAFKA and would like help
I have 2 applications (springboot), they are identical/copies only with different ports.
http://localhost:8080/
http://localhost:8081/
They are both consumers
the two listen to the topic XXX
I have another APP that plays the role of producer.
whenever I send something to topic XXX.
only one of them consumes the message and the other does not.
I have tested both individually and they listen normally if they are listening alone, but if they are listening together only one of them will listen.
I'm using
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
<version>3.0.0.RELEASE</version>
</dependency>
spring.cloud.stream.kafka.binder.autoCreateTopics=true
spring.cloud.stream.kafka.binder.headers=type
spring.cloud.stream.kafka.default.consumer.ackEachRecord=true
spring.cloud.stream.kafka.default.consumer.enableDlq=true
spring.cloud.stream.kafka.default.consumer.standardHeaders=both
spring.cloud.stream.kafka.default.consumer.dlqName=api***.d**.api***
my listener
#StreamListener(target=SomeString.TOPIC, condition = "headers['type']=='***' or headers['type']=='***'")
public void handle(GenericMessage<String> message) throws BusinessException {
***
}
The Apache Kafka Binder implementation maps each destination to an
Apache Kafka topic. The consumer group maps directly to the same
Apache Kafka concept. Partitioning also maps directly to Apache Kafka
partitions as well.
check kafka consumer group behavior -> https://kafka.apache.org/documentation/#consumerconfigs_group.id
if kafka consumers have same groupId just one of listen message, you should give them different groupId
app in 8080 -> spring.cloud.stream.bindings.<channelName>.group=consumer-group-1
app in 8081 -> spring.cloud.stream.bindings.<channelName>.group=consumer-group-2
I am sending all my application logs to the Kafka using Log4j2 Kafka appender and it works. But in a situation, when I purposefully bring down the broker, the application gets hung-up and the kafka appender keeps on retrying to establish the connection.
How can I stop writing into Kafka when the broker(s) are down? and resume once it is available?
Following is the appender configuration I have used.
<Kafka name="KafkaServiceStatInfo" topic="testKafkaLogs">
<PatternLayout pattern="%m"/>
<Property name="bootstrap.servers">localhost:9092</Property>
<Property name="acks">0</Property>
</Kafka>
<Async name="Async">
<AppenderRef ref="KafkaServiceStatInfo"/>
</Async>
I'm configuring an Apache Artemis message-broker. The broker will accept big files and downstream consumers access the topic to process the latest files. Now I'm wondering how to make the latest files available for dev-runs. Because the messages only arrive a few times a day, the test runs would need to access the last few sent messages and can't wait for the next.
For production and staging-systems, I found that durable subscriptions work fine. I've adapted an Apache Camel config to serve as an illustration. Here are two consumers that receive messages, each using a durable subscription:
<route id="inbox">
<from uri="file:inbox"/>
<to uri="activemq:topic:testing"/>
</route>
<route id="outbox-staging">
<from uri="activemq:topic:testing?clientId=staging&durableSubscriptionName=staging"/>
<to uri="file:outbox-staging"/>
</route>
<route id="outbox-production">
<from uri="activemq:topic:testing?clientId=production&durableSubscriptionName=production"/>
<to uri="file:outbox-production"/>
</route>
This is fine. If a consumer is offline it will pick up messages when it comes back online. Now if another consumer joins for testing;
<route id="outbox-testing" streamCache="true">
<from uri="activemq:topic:testing?clientId=my-local-consumer&durableSubscriptionName=my-local-consumer"/>
<to uri="file:outbox-local"/>
</route>
because the subscription didn't exist before, the consumer will have to wait for new messages. What I'm looking for is new subscribers to be immediately primed with the available messages. I found different names for the concept such as prefetchPolicy, consumerWindowSize, or "retroactive consumer". But it's unclear to me which terms apply to Apache Artemis and how to set them up because the examples mostly refer to Apache ActiveMQ.
How can a configure Artemis so that a consumer joining on a new subscription gets past messages?
The prefetchPolicy doesn't apply to ActiveMQ Artemis. It's for ActiveMQ 5.x.
The consumerWindowSize does apply to ActiveMQ Artemis.
However, neither prefetchPolicy nor consumerWindowSize apply to this situation as they're both related to "flow control" and have nothing to do with putting "missed" messages onto a JMS topic subscription.
The "retroactive consumer" feature is for ActiveMQ 5.x. A similar feature (called "retroactive address") will be available in ActiveMQ Artemis 2.11. It was implemented as part of ARTEMIS-2504.
Therefore you have a few options:
Wait for ActiveMQ Artemis 2.11 to be released (should be released in January).
Build your own version of ActiveMQ Artemis based on the master branch which includes the retroactive address feature.
Modify your test environment so that new subscribers don't have to wait so long for messages (e.g. send them more frequently).
I have two components communicating over an jms queue in a wildfly instance. As soon as the consumer of the queue disconnects or gets stopped, the messages are forwarded to the DLQ (at least when wildfly is restarted).
Is it possible to configure wildfly to automatically redeliver the messages from DLQ as soon as a consumer reconnects to the queue?
Some details
Wildfly version: 8.2.0
standalone.xml - As far as I can tell, nothing special
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="java:/jms/queue/ExpiryQueue"/>
<durable>false</durable>
</jms-queue>
<jms-queue name="DLQ">
<entry name="java:/jms/queue/DLQ"/>
<durable>false</durable>
</jms-queue>
...
<jms-queue name="Q1-Producer-to-Consumer">
<entry name="java:/queue/Q1-Producer-to-Consumer"/>
<entry name="java:jboss/exported/queue/Q1-Producer-to-Consumer"/>
<durable>false</durable>
</jms-queue>
</jms-destinations>
Thanks.
The DLQ only gets messages that have thrown an exception during message processing. If a consumer disconnects, the messages will just still be sitting there awaiting delivery
If you are seeing an Issue whereby during a server restart messages hit the DLQ, this would suggest that your consumer is consuming messages before the resources it requires are available, so is erroring when processing the messages. You would be better to fix your consumer to not start consuming messages to early, rather than trying to fish the failed messages back from DLQ