I currently have a console application that has this algorithm:
Consume from MSMQ using Peek
Get message out from MSMQ , do sql insert statement
if insert is successful, get message out from MSMQ using Receive()
Using the above consumes 20 messages from MSMQ /sec(20 sql tx per sec), with 1 consume thread
Now , I would want to increase the threads consuming MSMQ , however I discovered that messages get inserted 2 times compared to 1, using peek.
I use Peek() to get the message and process the sql statement initially to deal with server power down suddenly.
If I use Receive() , and the sql statement is not performed , this will result in a loss of data
Please advise.
Related
Our use case -> Using Remote partitioning - the job is devided into multiple partitions and using active MQ workers are processing these partitions.
Job is failing with memory issue at MessageChannelPartitionHandler handle method where it is holding more number of StepExecution in memory.(we have around 20K StepExecutions/partitions in this case)
we override message channel partition handler for submitting controlled messages to ActiveMQ and even when we try to poll replies from DB it is having database connection timeout issues and when we increased idle connection this approach as well failing to hold all those StepExecutions in memory.
Either case of our Custom/MessageChannelPartitionHandler we are facing similar issues and these step executions are required to aggregate at master. Do we have any alternative way of achieving this.
Can someone help us to understand better way of handling these long running/huge data processing scenarios?
I have a jsr223 preprocessor in the concurrency thread group which creates data to send it to Kafka producer. and I have a JSR sampler that uses Kafka client 2.7.0 to send messages to Kafka procedure.
The message sent to Kafka should be different each time for e.g. it has device information which should be different and events with time (which is the current time). These are been generated without any issues as I tested it with few
(50) threads. The problem I am having is when I want to send more messages like 6000 messages per second. How to resolve this issue
below is my setup
You're showing us a screenshot of the Concurrency Thread Group configured to start 6000 threads (virtual users) and hold them for 20 seconds.
It will result in 6000 messages per second only if your JSR223 PreProcessor and Sampler cumulative response time is exactly 1 second. If it will be less - you will generate more messages per second and vice versa.
For example:
if PreProcessor and Sampler execution time is 500ms - you will end with 12000 messages per second
if PreProcessor and Sampler execution time is 2000ms - you will send 3000 messages per second
If you're sending less messages than you need - consider following JMeter Best Practices, at least disable all the Listeners and run your test in non-GUI mode. Still not enough? Increase the concurrency. Increased concurrency, lacking resources and still not enough - go for Distributed Testing
If you're sending more messages than 6000 per second you can limit JMeter sampler execution rate to the desired throughput using Throughput Shaping Timer
You can see your current throughput using i.e. Transactions per Second listener
I am not able to use parallel threads in my application(C# Application) as each msmq msg is a data for sql statement
The C# Application consumes messages in the msmq , then processes it.
I am using msg.Recoverable=true, to prevent data loss when server powers down.
Currently, it is transactional message.
The msmq messages(sql statements) must be processed in order in the sequence(FIFO)
Using multiple threads to consume is a no-no as unable to control which thread will consume and perform which sql statement.
Currently , the speed of Receive() for a msmq message of 490 KB is about 0.016 -0.032 sec, which makes about 30-50 Tx/sec, which is unacceptable.
Currently, MSMQ receive is the bottleneck of the whole application
Server is on Windows Server 2008 R2
and the application is getting(Receive()) from the Local server MSMQ, not remote
I am not an expert, but i think you can try to use batching - http://msdn.microsoft.com/en-us/library/ms788973.aspx
This blog discusses creating multiple "QueuePeekCompleted" event handlers. It worked for me.
http://code-ronin.blogspot.com/2008/09/msmq-transactional-message-processing.html
I am using the Azure REST API to read messages from an Azure Queue using Peek-Lock Message. Is there any way I can read the last message that was posted in the queue rather than reading from a queue based mechanism (FIFO)?
Also, is there a faster way to process messages from Azure other than using the Peek-Lock Message REST API?
Thanks!
Is there any way I can read the last message that was posted in the
queue rather than reading from a queue based mechanism (FIFO)?
Using the REST API, unfortunately there's no way to process the last message first. You would have to implement something on your own. If you know that your queue can't have more than 32 messages at a time, you could possibly get all 32 messages in one go and sort them on the client side based on the message insertion time. Yet another (crazy) idea would be to create a new queue for each message and name the queue using the following pattern: "q"-(DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks). Now list queues and get only the 1st queue. This will give you the message you last inserted.
Also, is there a faster way to process messages from Azure other than
using the Peek-Lock Message REST API?
One possibility could be to fetch more than one messages from a queue and process them in parallel on the client side.
This is my proposed architecture. Process A would create items and add it to a queue A on the local machine and I plan to have multiple instances of windows service( running on different machines) reading from this queue A .Each of these windows service would read a set of messages and then process that batch.
What I want to make sure is that a particular message will not get processed multiple times ( by the different windows service). Does MSMQ by default guarantee single delivery?
Should I make the queue transactional? or would a regular queue suffice.
If you need to make sure that the message is delivered only once, you would want to use a transactional queue. However, when a service reads a message from the queue it is removed from the queue and can only be received once.