I would like to add a message to a broker without the broker publishing it to subscribers.
I then want to, at a later date, tell the broker to publish the message.
I want to do this so I can set a one-off predefined task that can only be executed by calling it.
An alternative I have tried that doesnt work is to do:
task = tasks.send_message.apply_async(['hello'], countdown=60)
revoke(task.task_id, terminate=True)`
But this doesn't revoke the task - the task executes.
This can be done with the RabbitMQ dead-letter exchange extension. When you publish the task you put it on a queue with no consumers and declare the consumer queue as the dead letter queue. When the message TTL expires on the original exchange it will be dead letter to the consumer queue where it will then be consumed.
To accomplish this with Celery you would declare a queue
from kombu import Exchange, Queue
DEAD_LETTER_OPTIONS = {
'x-message-ttl': 60 * 10 * 1000, # 10 mins
'x-dead-letter-exchange': 'default',
'x-expires': (60 * 10 + 1) * 1000,
}
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('wait', Exchange('wait', arguments=DEAD_LETTER_OPTIONS ), routing_key='wait'),
)
Then you would call your task and put it on the wait queue.
tasks.send_message.apply_async(['hello'], queue='wait')
You can also see this example of a general dead letter based countdown https://gist.github.com/dgouldin/3485236
Related
I have set ActiveMQ Artemis' configuration to redeliver an unsuccessful message after a period of time with a delay, like this
attempt no 1. unsuccessful delivery wait for 5 secs
attempt no 2. unsuccessful delivery wait for 10 secs
...
attempt no nnn. unsuccessful delivery wait for 5 hours
The problem is that I don't see messages on the queue that are scheduled and I don't know how to cancel 5 hours waiting period and redeliver the message right now
My questions
Why can't I see that message on the queue when I execute browse() function on the Artemis GUI Console? I can only see that message when I execute listScheduledMessages(). Had I not tried listScheduledMessages() I would be wondering why have I lost a message.
Is there any way to repeat a message without waiting for the next 5 hours?
You can't see scheduled messages when you use the browse() management method because technically scheduled messages are not on the queue. If they were on the queue they would be delivered to consumers.
There is currently no way to repeat a message without waiting for the scheduled time to arrive. However, you could get the message's ID using listScheduledMessages() and then pass that ID to removeMessage(long) to delete the message and then resend it with a different (or no) schedule.
I have a function:
public async static Task Run([QueueTrigger("efs-api-call-last-datetime", Connection = "StorageConnectionString")]DateTime queueItem,
[Queue("efs-api-call-last-datetime", Connection = "StorageConnectionString")]CloudQueue inputQueue,
TraceWriter log)
{
Then I have long process for processing message from queue. Problem is the message will be readded to queue after 30 seconds, while I process this message. I don't need to add this message and process it twice.
I would like to have code like:
try
{
// long operation
}
catch(Exception ex)
{
// something wrong. Readd this message in 1 minute
await inputQueue.AddMessageAsync(new CloudQueueMessage(
JsonConvert.SerializeObject(queueItem)),
timeToLive: null,
initialVisibilityDelay: TimeSpan.FromMinutes(1),
options: null,
operationContext: null
);
}
and prevent to readd it automatically. Any way to do it?
There are couple of things here.
1) When there are multiple queue messages waiting, the queue trigger retrieves a batch of messages and invokes function instances concurrently to process them. By default, the batch size is 16. But this is configurable in Host.json. You can set the batch size to 1 if you want to minimize the parallel execution. Microsoft document explains this.
2) As it is long running process so it seems your messages are not complete and the function might timeout and message are visible again. You should try to break down your function into smaller functions. Then you can use durable function which will chain the work you have to do.
Yes, you can dequeue same message twice.
Reasons:
1.Worker A dequeues Message B and invisibility timeout expires. Message B becomes visible again and Worker C dequeues Message B, invalidating Worker A's pop receipt. Worker A finishes work, goes to delete Message B and error is thrown. This is most common.
2.The lock on the original message that triggers the first Azure Function to execute is likely expiring. This will cause the Queue to assume that processing the message failed, and it will then use that message to trigger the Function to execute again.
3.In certain conditions (very frequent queue polling) you can get the same message twice on a GetMessage. This is a type of race condition that while rare does occur. Worker A and B are polling very quickly and hit the queue simultaneously and both get same message. This used to be much more common (SDK 1.0 time frame) under high polling scenarios, but it has become much more rare now in later storage updates (can't recall seeing this recently).
1 and 3 only happen when you have more than 1 worker.
Workaround:
Install azure-webjobs-sdk 1.0.11015.0 version (visible in the 'Settings' page of the Functions portal). For more details, you could refer to fixing queue visibility renewals
TL;DR: If my producer process crashes after sending some work to the consumers, how can it resume waiting for the consumers to complete their work once it restarts?
producer.py dispatches work items to a group of consumers (registered Celery tasks), like so:
from celery import group, signature
job = group(
signature(task_name, args=(x,)) for x in xrange(100)
)
group_result = job.apply_async()
group_result.join() # blocks until tasks complete
The consumers take a long time to complete, so it's possible/expected that the producer will sometimes die during the call to join(). When the producer dies, it is restarted.
When the produces restarts, is there a way to resume the join?
I'm a Celery newbie; have combed through docs and examples but haven't found an answer to this. Hoping the experts can help point me in the right direction.
For the record, my solution was to store the task ids (in some external file/db/whatever).
job = group(
signature(task_name, args=(x,)) for x in xrange(100)
)
group_result = job.apply_async()
# store group_result.results somewhere durable
persist(some_db, group_result.results)
group_result.join() # blocks until tasks complete
Now, if the join above fails, we can recover like this:
# read tasks from persistent storage
previous_pending_results = read_previously_persisted_results(some_db)
for result in previous_pending_results:
if result.status != 'SUCCESS':
...
I have a use case where i consume certain logs from a queue and hit some third party API's with some info from that log , in case the third party system is not responding properly i wish to implement a retry logic for that particular log .
I can add a time field and repush the message to the same queue and this message will get again consumed if its time field is valid i.e less than current time and if not then get pushed again into queue.
But this logic will add same log again and again until retry time is correct and the queue will grow unnecessarily.
Is there is better way to implement retry logic in Kafka ?
You can create several retry topics and push failed task there. For instance you can create 3 topics with different delays in mins and rotate the single failed task till the max attempt limit reached.
‘retry_5m_topic’ — for retry in 5 minutes
‘retry_30m_topic’ — for retry in 30 minutes
‘retry_1h_topic’ — for retry in 1 hour
See more for details: https://blog.pragmatists.com/retrying-consumer-architecture-in-the-apache-kafka-939ac4cb851a
In consumer, if it throws an exception, produce another message with attempt number 1. so next time when it is consumed, it has the property of attempt no 1. Handle it in the producer that, if it attempts more than your retry count, then stop producing it.
Yes, this could be one straight solution that I also thought of. But with this, we will end up in creating many topics as it is possible that message processing will fail again.
I solved this problem by mapping this use case to Rabbit MQ. In rabbit MQ we have the concept of retry exchange where if a message processing fails from an exchange then u can send it to a retry exchange with a TTL. Once TTL gets expired the message will move back to the main exchange and is ready to be processed again.
I can post some examples explaining how we can implement an exponential backoff message processing using Rabbit MQ.
I am kafka newbie and as I was reading the docs, I had this design related question related to kafka consumer.
A kafka consumer reads messages from the kafka stream which is made up
of one or more partitions from one or more servers.
Lets say one of the incoming messages is corrupt and as a result the consumer fails to process. But when processing event logs you don't want to drop any events, as a result you do infinite retries to avoid transient errors during processing. In such cases of infinite retries, how can the consumer move forward. Is there a way to blacklist this message for next retry?
I'd think it needs manual intervention. Where we log some message metadata (don't know what exactly yet) to look at which message is failing and have logic in place where each consumer checks redis (or someplace else?) after n reties to see if this message needs to be skipped. The blacklist doesn't have to be stored forever in the redis either, only until the consumer can skip it. Here's a pseudocode of what i just described:
while (errorState) {
if (msg in blacklist) {
//skip
commitOffset()
} else {
errorState = processMessage(msg);
if (!errorState) {
commitOffset();
} else {
// log this msg so that we can add to blacklist
logger.info(msg)
}
}
}
I'd like to hear from more experienced folks to see if there are better ways to do this.
We had a requirement in our project where the processing of an incoming message to update a record was dependent on the record being present. Due to some race condition, sometimes update arrived before the insert. In such cases, we implemented couple of approaches.
A. Manual retry with a predefined delay. The code checks if the insert has arrived. If so, processing goes as normal. Otherwise, it would sleep for 500ms, then try again. This would repeat 10 times. At the end, if the message is still not processed, the code logs the message, commits the offset and moves forward. The processing of message is always done in a thread from a pool, so it doesn't block the main thread either. However, in the worst case each message would take 5 seconds of application time.
B. Recently, we refined the above solution to use a message scheduler based on kafka. So now if insert has not arrived before the update, system sends it to a separate scheduler which operates on kafka. This scheduler would replay the message after some time. After 3 retries, we again log the message and stop scheduling or retrying. This gives us the benefit of not blocking the application threads and manage when we would like to replay the message again.