I'm receiving messages from MSMQ in async fashion. How can I limit processing of messages upto n messages and then reenable the event to start processing again? (I don't want to limit the thread numbers)
You can't do this with MSMQ. You will either have to implement this behavior in your sending application or receiving application.
Related
I've started to use Kafka. And I have a question about it.
If Kafka is not running because of network problem, kafka crash etc. how can I eliminate this problem? And, What happens to messages that was sent to kafka?
If all brokers in the cluster are unavailable your producer will not get any acknowledgements (note that the actual send happens in a background thread, not the thread that calls send - that is an async call).
If you have acks=0 then you have lost the message but acks=1 or acks=all then it depends on retry configuration. By default the producer thread retries pretty much indefinitely which means at some point the send buffer will fill up and then the async send method will fail synchronously, but if your client fails in the meantime then the messages in the buffer are lost as that is just in memory.
If you are wondering about behaviour when some but not all brokers are down, I wrote about it here
I want a Weblogic queue to receive a message, but I don't want to process that message further. I want the messages I've sent to the queue to stay there before they are consumed.
So I think I need to pause Production and Consumption but leave Insertion to run so every message sent to that Queue will stay there, and I will be able to read each message created there. Am I right?
Based on the Weblogic documentation on this subject you should only pause consumption. If you pause production then producers will not be able to send messages to the queue. As the documentation states:
When a JMS destination is "paused for production," new and existing producers attached to that destination are unable to produce new messages for that destination. A producer that attempts to send a message to a paused destination receives an exception that indicates that the destination is paused.
Also, if you pause insertion then any in-flight messages will not appear on the queue either. Again, from the documentation:
When a JMS destination is paused for "insertion," both messages inserted as a result of in-flight work and new messages sent by producers are prevented from appearing on the destination. Use insertion pause to stop all messages from appearing on a destination.
That said, if consumption is paused then you won't be able to consume the messages either, although you should be able to use a JMS browser to inspect them.
I'm publishing MQTT messages from an Arduino, and subscribing to those from a Raspberry Pi. Sometimes the publishing goes faster than the Raspberry can receive (and process).
I'm looking for a way of checking how many messages are queued on the Raspberry side. I'm using Paho-MQTT. I only see it is possible to set a max queue size, but how can I check the current queue size? (If possible.)
There is no queue in the broker, all messages are delivered as they are published.
The Paho client is singled threaded and the Message received call back is handled on the network thread, so messages may back up on the network stack (for QOS0 messages). QOS1/2 messages will back up in the broker until the QOS handshake for the current message completes.
The max_queued message setting is about how many QOS 1/2 messages the client will accept to publish before blocking, not how many incoming messages it will queue up.
If you want to queue messages in a measurable way then have the Message received callback place the messages on to a local queue and have a second thread (or pool of threads if they can be handled in parallel) take messages from the local queue.
With RabbitMQ, is there a way to "push" messages from a queue TO a consumer as opposed to having a consumer "poll and pull" messages FROM a queue?
This has been the cause of some debate on a current project i'm on. The argument from one side is that having consumers (i.e. a windows service) "poll" a queue until a new message arrives is somewhat inefficient and less desirable than the idea having the message "pushed" automatically from the queue to the subscriber(s)/consumer(s).
I can only seem to find information supporting the idea of consumers "polling and pulling" messages off of a queue (e.g. using a windows service to poll the queue for new messages). There isn't much information on the idea of "pushing" messages to a consumer/subscriber...
Having the server push messages to the client is one of the two ways to get messages to the client, and the preferred way for most applications. This is known as consuming messages via a subscription.
The client is connected. (The AMQP/RabbitMQ/most messaging systems model is that the client is always connected - except for network interruptions, of course.)
You use the client API to arrange that your channel consume messages by supplying a callback method. Then whenever a message is available the server sends it to the client over the channel and the client application gets it via an asynchronous callback (typically one thread per channel). You can set the "prefetch count" on the channel which controls the amount of pipelining your client can do over that channel. (For further parallelism an application can have multiple channels running over one connection, which is a common design that serves various purposes.)
The alternative is for the client to poll for messages one at a time, over the channel, via a get method.
You "push" messages from Producer to Exchange.
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
BTW this is fitting very well IoT scenarios. Devices produce messages and sends them to an exchange. Queue is handling persistence, FIFO and other features, as well as delivery of messages to subscribers.
And, by the way, you never "Poll" the queue. Instead, you always subscribe to publisher. Similar to observer pattern. Generally, I would say genius principle.
So it is similar to post box or post office, except it sends you a notification when message is available.
Quoting from the docs here:
AMQP brokers either deliver messages to consumers subscribed to
queues, or consumers fetch/pull messages from queues on demand.
And from here:
Storing messages in queues is useless unless applications can consume
them. In the AMQP 0-9-1 Model, there are two ways for applications to
do this:
Have messages delivered to them ("push API")
Fetch messages as needed ("pull API")
With the "push API", applications have to indicate interest in
consuming messages from a particular queue. When they do so, we say
that they register a consumer or, simply put, subscribe to a queue. It
is possible to have more than one consumer per queue or to register an
exclusive consumer (excludes all other consumers from the queue while
it is consuming).
Each consumer (subscription) has an identifier called a consumer tag.
It can be used to unsubscribe from messages. Consumer tags are just
strings.
RabbitMQ broker is like server that wont send data to consumer without consumer client getting registering itself to server. but then question comes like below
Can RabbitMQ keep client consumer details and connect to client when packet comes?
Answer is no. so what is alternative well then write plugin by yourself that maintain client information in some kind of config. Plugin will pull from RabbitMQ Queue and push to client.
Please give look at this plugin might help.
https://www.rabbitmq.com/shovel.html
Frankly speaking Client need to implement AMQP protocol to receive so and should listen connection on some port for that. This sound like another server.
Regards,
Vishal
I have a transactional private message queue (among other message queues on which I have not seen this problem) on a Windows Server 2008 R2 server.
This particular queue has a recurring problem happening every few weeks where the console shows a nonzero count of messages in the queue, but it does not have any messages in the queue itself or any subqueue. Queue Explorer shows the same thing. Performance counters indicate there are messages like the count in the built-in msmq console and queue explorer.
I cannot find any messages. I understand that I could see a situation like this for outgoing queues with dead letter tracking such that it may have been delivered to a remote machine but not yet processed. This is not an outgoing queue, though. Messages are sourced from remote machines and have landed here on this machine.
Also, I am certain that the count I'm seeing are not journal messages or subqueues.
Does this make any sense? Is there a logical explanation for this and under some circumstance this is expected? If so, what is it?
EDIT: Removed info about purging queue removing the count - that was incorrect. Purging actually does nothing and leaves me in the same state as before with a count reflected, but no messages showing.
As you noted, you can see a message count on an outgoing queue if source journaling is in use. The invisible messages are there in case they need to be moved to the DLQ.I would expect your problem to be similar - there should be a visible message in the outgoing queue and an invisible message in the destination queue because delivery hasn't completed. I assume a handshaking or storage acknowledgement has been lost along the way. Or maybe the message has been processed and removed from the queue but MSMQ couldn't update the sender of the fact. Check the outgoing queues on the remote machines sending TO this queue.