Handling connection failures in apache-camel - scala

I am writing an apache-camel RabbitMQ consumer. I would like to react somehow to connection problems (i.e. try to reconnect). Is it possible to configure apache-camel to automatically reconnect?
If not, how can I find out that a connection to the queue was interrupted? I've done the following test:
start the queue (and some producer)
start my consumer (it was getting messages as expected)
stop the queue (the messages stopped arriving, as expected, but no exception was thrown)
start the queue (no new messages were received)
I am using camel in Scala (via akka-camel), but a Java solution would be probably also OK

You can pass in the flag automaticRecoveryEnabled=true to the URI, Camel will reconnect if the connection is lost.

For automatic RabbitMQ resource recovery (Connections/Channels/Consumers/Queues/Exchanages/Bindings) when failures occur, check out Lyra (which I authored). Example usage:
Config config = new Config()
.withRecoveryPolicy(new RecoveryPolicy()
.withMaxAttempts(20)
.withInterval(Duration.seconds(1))
.withMaxDuration(Duration.minutes(5)));
ConnectionOptions options = new ConnectionOptions().withHost("localhost");
Connection connection = Connections.create(options, config);
The rest of the API is just the amqp-client API, except your resources are automatically recovered when failures occur.
I'm not sure about camel-rabbitmq specifically, but hopefully there's a way you can swap in your own resource creation via Lyra.

Current camel-rabbitmq just create a connection and the channel when the consumer or producer is started. So it don't have a chance to catch the connection exception :(.

Related

How to configure an ActiveMQ Artemis queue not be created again

I have an embedded ActiveMQ Artemis 2.17.0 broker that has no pre-configured queues. I'd like for the client to connect to the broker and have its queue created automatically if it doesn't exist. This works just fine the 1st time when I run the client but on the second time I connect again to the broker the client throws an error :
errorType=QUEUE_EXISTS message=AMQ229019: Queue hornetq already exists on address router
Is it possible to configure the client or the server in such a way that if a queue is already in existence it should not attempt to recreate it again?
Below are the programs I am running:
Server
try {
ActiveMQServer server = new ActiveMQServerImpl(new ConfigurationImpl()
.setPersistenceEnabled(true)
.setBindingsDirectory("./router/data/bindings")
.setLargeMessagesDirectory("./router/data/large")
.setPagingDirectory("./router/data/paging")
.setJournalDirectory("./router/data/journal")
.setSecurityEnabled(false)
.addAcceptorConfiguration("tcp", "tcp://0.0.0.0:61617?protocols=CORE,AMQP"));
server.start();
} catch (Exception ex) {
System.err.println(ex);
}
Client
ServerLocator serverLocator = ActiveMQClient.createServerLocator("tcp://127.0.0.1.3:61617");
ClientSessionFactory factory = serverLocator.createSessionFactory();
ClientSession session = factory.createSession();
session.createQueue(new QueueConfiguration("router::hornetq")
.setAutoCreateAddress(Boolean.FALSE)
.setAutoCreated(Boolean.FALSE)
.setRoutingType(RoutingType.ANYCAST));
ClientProducer producer = session.createProducer("router::hornetq");
ClientMessage message = session.createMessage(true);
message.getBodyBuffer().writeString("Core Queue Message");
producer.send(message);
session.start();
ClientConsumer consumer = session.createConsumer("router::hornetq");
ClientMessage msgReceived = consumer.receive();
System.out.println("message = " + msgReceived.getBodyBuffer().readString());
session.close();
I'm using the fully qualified queue name here (i.e. router::hornetq) because I have multiple queues on the router address.
Your client is using the core API which is a low-level API that does not support auto-queue creation. Your client is creating (or attempting to create) the queue manually, e.g.:
session.createQueue(new QueueConfiguration("router::hornetq")
.setAutoCreateAddress(Boolean.FALSE)
.setAutoCreated(Boolean.FALSE)
.setRoutingType(RoutingType.ANYCAST));
The exception you're seeing is no doubt being thrown here if the queue exists already. Creating a queue is not idempotent so you have 3 options from what I can see:
Simply catch the ActiveMQQueueExistsException that is being thrown here and ignore it.
Use ClientSession.queueQuery to see if the queue exists before you attempt to create it. If it exists then don't try to create it again. If it doesn't exist then create it. That said, if you have lots of clients like this running concurrently it's still possible you could get an ActiveMQQueueExistsException due to race conditions between the clients.
Use a client/protocol that supports auto-creation like the core JMS client or AMQP.
A few other things are worth mentioning:
Since you're not explicitly calling ClientSession.createAddress() you probably want to use setAutoCreateAddress(Boolean.TRUE).
Your consumer doesn't have to use router::hornetq. It can just use hornetq and it will receive any messages sent to the hornetq queue. Queue names are universally unique in the broker.

TIMED_WAITING on message listener thread

I use ActiveMQ Artemis 2.10.1 and getting message listener thread hanging issue.
Thread is going into TIMED_WAITING and recover only after client JVM restart. This is an indeterminate issue and not able to reproduce easily. Client library version is 2.16.0.
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:301)
- locked <0x000000050cd4e4f0> (a org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:275)
- locked <0x000000050cd4e4f0> (a org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:159)
at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:91)
at org.apache.activemq.artemis.jms.client.ActiveMQBytesMessage.readBytes(ActiveMQBytesMessage.java:220)
at com.eu.jms.JMSEventBus.onMessage(JMSEventBus.java:385)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:746)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:684)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748) ```
The client is waiting in LargeMessageControllerImpl.waitCompletion. This wait will not block forever. The code waits in a loop for packets of a large message. As long as packets of the large message are still arriving the client will continue to wait until all the packets have arrived or if a packet doesn't arrive within the given timeout it will throw an error. The timeout is based on the callTimeout which is configured on the client's URL. The default callTimeout is 30000 (i.e. 30 seconds).
My guess is that your client is receiving a very large message or the network has slown down or perhaps a combination of these things. You can turn on TRACE logging for org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl to see the individual large message packets arriving at the client if you want more insight into what's happening.
To be clear, it's not surprising that thread dumps show your client waiting here as this is the most likely place for the code to be waiting while it receives a large message. It doesn't mean the client is stuck.
Keep in mind that if there is an actual network error or loss of connection the client will throw an error. Also, the client maintains an independent thread which sends & receives "ping" packets to & from the broker respectively. If the client doesn't get the expected ping response then it will throw an error as well. The fact that none of this happened with your client indicates the connection is valid.
I would recommend checking the size of the message at the head of the queue. The broker supports arbitrarily large messages so it could potentially be many gigs which the client will happily sit and receive as long as the connection is valid.

ActiveMQ Artemis publish message loss during HA fail-over

I use ActiveMQ Artemis 2.17.0, and I'm looking to avoid message loss in a producer during fail-over.
Message publish loss handled during Artemis active to passive switch by catching ActiveMQUnBlockedException and sending the message again.
The brokers are configured as active/passive HA shared-store. Active node configured in host1 and passive node configured in host2.
Url is:
(tcp://host1:61616,tcp://host2:61616)?ha=true&reconnectAttempts=-1&blockOnDurableSend=false
blockOnDurableSend set to false for high throughput.
During active to passive switch publishing code throws ActiveMQUnBlockedException but not during passive to active switching.
We're using Spring 4.2.5 and CachingConnectionFactory for connection factory.
I'm using the following code to send messages:
private void sendMessageInternal(final ConnectionFactory connectionFactory, final Destination queue, final String message)
throws JMSException {
try (final Connection connection = connectionFactory.createConnection();) {
connection.start();
try (final Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
final MessageProducer producer = session.createProducer(queue);) {
final TextMessage textMessage = session.createTextMessage(message);
producer.send(textMessage);
}
} catch (JMSException thr) {
if (thr.getCause() instanceof ActiveMQUnBlockedException) {
// consider as fail-over disconnection, send message again.
} else {
throw thr;
}
}
}
In host1 machine, Artemis deployed as master - node1.
In host2 machine, Artemis deployed as slave - node2.
following steps I did to simulate fail-over
node1 and node2 started
node1 started as live server and node2 started as backup server
killed node1, node2 become live server
client publish code threw ActiveMQUnBlockedException and handled to send message again
started node1 again. node1 become live server and node2 become backup again
client publish code did not throw ActiveMQUnBlockedException and loss in message.
Getting following error stack during step #3. ( Killed node1 and node2 become Live server).
javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:540)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:434)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:470)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1121)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:1110)
at org.apache.activemq.artemis.jms.client.ActiveMQSession.stop(ActiveMQSession.java:1244)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection.stop(ActiveMQConnection.java:339)
at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.localStop(SingleConnectionFactory.java:644)
at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:577)
at com.sun.proxy.$Proxy5.close(Unknown Source)
at com.eu.amq.failover.test.ProducerNodeTest.sendMessageInternal(ProducerNodeTest.java:133)
at com.eu.amq.failover.test.ProducerNodeTest.sendMessage(ProducerNodeTest.java:110)
at com.eu.amq.failover.test.ProducerNodeTest.main(ProducerNodeTest.java:90)
The ActiveMQUnBlockedException you're getting is coming from Spring's invocation of javax.jms.Connection#stop. It's not related to sending a message. Re-sending a message when you get this specific exception could result in a duplicate message.
Ultimately your problem is directly related to setting blockOnDurableSend=false. This tells the client to "fire and forget." In other words the client won't wait for a response from the broker to ensure the message actually made it successfully. This lack of waiting increases throughput but decreases reliability.
If you really want to mitigate potential message loss you have two main options.
Set blockOnDurableSend=true. This will reduce message throughput, but it's the simplest way to guarantee the message arrived at the broker successfully.
Use a CompletionListener. This will allow you to keep blockOnDurableSend=false, but the application will still be informed if there are problems sending the message although the information will be provided asynchronously. This feature was added in JMS 2 specifically for this kind of scenario. See the JavaDoc for more details.

JMS-HornetQ JBoss AS6 configuration

Sometimes getting below exception
javax.jms.JMSException: Could not create a session: Unable to get managed connection for JmsXA
at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:881)
at org.hornetq.ra.HornetQRASessionFactoryImpl.createQueueSession(HornetQRASessionFactoryImpl.java:237)
While creating QueueSession, below is the snippet used
connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
We are using java:JmsXA connection factory which uses INVM .
AFAIK there is no use of setting parameters in nettyconnectionfactory and INVMconnectionfactory in hornetq-jms.xml
Either we should set it some parameters in jms-ds.xml(JMS Queue Configuration File) or ra.xml(MDB configuration file)
I know some parameters can be set to
1. <reconnect-attempts>1000</reconnect-attempts>
this will try to reconnect 1000 times after it gets disconnected
2. <call-timeout>10800000</call-timeout>
also there is no use of setting
as it is default to -1 and will try to connect unlimited no. of times
I am confused as to what parameters can be set and at what level ..i.e. either on queue level (in jms-ds.xml) or at MDB level (ra.xml) as some parameters are same e.g. call-timeout,retry-interval,etc
Try with increasing max-pool-size of pooled-connection-factory JmsXA.

Atmosphere Jersey - Server messages

Is it possible to let the server send messages to all connected clients without waiting for any action from them? Let me explain it :-) I've been reading the docs/examples and I have found nothing that satifies my needs: the flow is always the same; a client connects (e.g: a GET call to a Rest API), the connection is suspendend and until a new API call is received (e.g.: a POST call) the server simply waits (or at least this is what I have understood). My use case is pretty different: I want the server to send some "notifications" once new data become available. This would be my use case (pretty simplifed):
Client A connects to server
Connection is suspended since no new data is available at the moment
The server gets notified new data is available from an external
source and broadcasts it to client A
Go to step 2
What I have achieved so far is getting the connection successfully established. The next step is to solve this server issue. I must say this technology is completely new to me so it is possible I misunderstood how something works. If that's the case, let me know!
This is my stack:
Spring 3.2.0 RELEASE
Jersey 1.8
Atmosphere Jersey 1.0.13
Tomcat 7.0.40
Thank you all in advance!
UPDATE: After following this I get this warning, which I have no idea how to get rid of:
2013-06-04 09:40:36,284 WARN [org.atmosphere.cpr.AtmosphereFramework] - Failed using comet support: org.atmosphere.container.Tomcat7AsyncSupportWithWebSocket, error: Tomcat failed to detect this is a Comet application because context.xml is missing or the Http11NioProtocol Connector is not enabled.
If that's not the case, you can also remove META-INF/context.xml and WEB-INF/lib/atmosphere-compat-tomcat.jar Is the Nio or Apr Connector enabled?
2013-06-04 09:40:36,285 WARN [org.atmosphere.cpr.AtmosphereFramework] - Using org.atmosphere.container.Tomcat7BIOSupportWithWebSocket
I followed the app structure commented here, so this should not be a problem. I have noticed that by changing the transport to "websocket" instead of "long-polling" shows no errors. The server finally sends data tough :)
I followed your link and modified the code a little.
When you are in the step 3 "The server gets notified new data is available from an external source and broadcasts it to client A", you have to write a line like this:
BroadcasterFactory.getDefault().lookup("/*").broadcast(response);
At first I used the TextMessage received from my ActiveMQ Queue but I get this error, so I used a Jackson class as an object response and everything worked fine.
SEVERE: A message body writer for Java class org.apache.activemq.command.ActiveMQTextMessage, and Java type class org.apache.activemq.command.ActiveMQTextMessage, and MIME media type application/json was not found
jun 03, 2014 11:32:21 AM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: The registered message body writers compatible with the MIME media type are:
application/json (JSONJAXBElementProvider, JSONArrayProvider, JSONObjectProvider, JSONRootElementProvider, JSONListElementProvider, ...)