Recently we have been moved to Azure Service Bus from RabitMQ, from there we were getting many complaints about duplicate reminders/reports from customer.
Cause: As defer message is getting duplicated on restart app/component, due to that it is sending multiple times.
Why was working with RabitMQ? It was working because in initialization of bus it can be used configurer.Timeouts(x => x.StoreInMemory()) this feature, but with Azure Service Bus it can't be. On every restart defer (schedule list) will get cleared as it is stored in memory and with Azure Service Bus t gets persistence due to that it is increased request on every restart.
So, what is best way to handle it?
Is it possible to clear defer message on startup or is it possible to use Timeouts (in memory) feature with Azure Service Bus?
Related
I have a Service Hook setup that sends an entry to an Azure Service Bus Queue. After experiencing delays on processing, I noticed the items were not getting processed and remained in Queued status. After several hours, new items were processed successfully.
No errors found on Service Bus Queue message processing for those that went through.
Will the queued items eventually get processed?
Will the queued items eventually get processed?
I'm afraid they won't be dealt with eventually.
Any that are still showing a Queued status will not be picked up again by the service and will not be delivered at this point. So there is no method could send the request again.
As far as I know, Service Hooks doesn't use jobs to create or send notifications, only AT threads, and it runs entirely in memory. Hence if the AT machine restarts or loses memory, the notification is lost for that event and will not be sent. This could be the cause of this issue. But it is hard to determine the specific cause
On the other hand, you could also submit the suggestion ticket about resending the Queued/Failed event (Because there is no such feature now). It will help deal with this situation.
I'd like to have more technical details about underling mechanisms of calling Actors in Azure Service Fabric, which I can't easily find online. Actors are know for their single-threaded scope, so unless any of its method execution is fully completed, no other clients are allowed to call it.
To be more specific, I need to know what happens if Actor is stuck for a while with its job initiated by one client call. How long are other clients supposed to wait until the job gets done? seconds, minutes, hours?
Is there any time-out mechanism, and if so is it somehow
configurable?
What happens if node where actor is located crashes,
would client receives immediate error, or ActorProxy somehow handles this situation and redirects call to newly created instance of Actor on healthy node?
There are quite a few SO answers with details about actor mechanisms and also in docs, I can point you a few:
This one does not exactly answer your question, but I described a bit how the locking works: Start a thread inside an Azure Service Fabric actor?
Q: Is there any time-out mechanism, and if so is it somehow configurable?
yes, there is a timeout, I have answered here: Acquisition of turn based concurrency lock for actor '{actorName}' timed out after {time}
The configuration docs are located in here
Q: What happens if node where actor is located crashes, would client receives immediate error, or ActorProxy somehow handles this situation and redirects call to newly created instance of Actor on healthy node?
Generally there is always a replica available when one replica goes down, new requests will start moving to the new replica when SF promotes a secondary replica to Primary.
Regarding the communication, by default, SF Actors use .Net Remoting for communication the same way as the reliable services, the behaviour is described very well in here, In summary, it retries transient failures, if the client can't connect to the service(Actor) it will retry until it reaches the connection timeout.
From the docs:
The service proxy handles all failover exceptions for the service partition it is created for. It re-resolves the endpoints if there are failover exceptions (non-transient exceptions) and retries the call with the correct endpoint. The number of retries for failover exceptions is indefinite. If transient exceptions occur, the proxy retries the call.
The Actor docs, has more info, in summary there are two points to keep in mind:
Message delivery is best effort.
Actors may receive duplicate messages from the same client.
That means, in case a transient failure occurs while delivering a message, it will retry, even though the message has been already delivered, causing duplicate messages.
I have a method on ServiceA that I need to call from ServiceB. The method takes upwards of 5 minutes to execute and I don't care about its return value. (Output from the method is handled another way)
I have setup my method in IServiceA like this:
[OneWay]
Task LongRunningMethod(int param1);
However that doesn't appear to run, because I am getting System.TimeoutException: This can happen if message is dropped when service is busy or its long running operation and taking more time than configured Operation Timeout.
One choice is to increase the timeout, but it seems that there should be a better way.
Is there?
For fire and forget or long running operations the best solution is using a message bus as a middle-ware that will handle this dependency between both process.
To do what you want without a middle-ware, your caller would have to worry about many things, like: Timeouts (like in your case), delivery guarantee(confirmation), Service availability, Exceptions and so on.
With the middle-ware the only worry your application logic need is the delivery guarantee, the rest should be handled by the middle-ware and the receiver.
There are many options, like:
Azure Service Bus
Azure Storage Queue
MSMQ
Event Hub
and so on.
I would not recommend using the SF Communication, Task.Run(), Threads workarounds as many places suggests, because they will just bring you extra work and wont run as smooth as the middle-ware approach.
I'm trying to figure out a portable way to develop a custom but scalable task queue in my cluster for google container engine . This is the scenario I have a front end that captures users details in my node js instance ,these details are sent to the api system which in turn contacts the db ,saves the user details and is expected to send a welcome mail .
My issue is this i don't want to use the same api endpoint method to process the sending mail requests ,I need another process to handle that how do I handle that with my kubernetes infrastructure?.Do I need to implement a pub sub type of system to publish to another container ?.If I do this it means all subscribes will be notified of my update but what if I have 2 instances of my sub system running it means they will all observe the changes and send the mail twice. Any thoughts or ideas on this would be appreciated.
I see two reasonable ways to approach this.
1: have a service that takes in mailing events by means of an API and returns immediately after receiving to process mailing asynchornously. Using kube service you will hit only one such service and one mail will be sent in a non blocking way for the calling service, but it has downsides - ie. what happens if something fails, the mail might not be generated at all.
2: I would go for some MQ probably (Kafka, Rabbit etc.), have a message queue consumed by any number of mailing service instances, make sure that only one can pick up the message, and require an ack for the message or return it to processing if no ack in N min
I've got a MassTransit ServiceBus running over MSMQ. It appears that the first message sent over the Bus doesn't arrive, but subsequent messages do?
Is there some initialization that needs performing on the queue or bus before the message is sent?
This depends on a few settings in how much time the system needs to setup before everything will correctly route. If only first message is failing to end up in the right location, then likely the subscription data isn't propagated everywhere yet. http://readthedocs.org/docs/masstransit/en/develop/overview/subscriptions.html
Using Multicast subscriptions, the easiest choice, will require a few seconds after a endpoint has come up and register a subscriber with all other endpoints. If you can control the order of services starting up, then this can often be avoided by started back to front in the flow.
If you are using the subscription service, then that can also take a couple seconds to get data everywhere. It has to go through the subscription service but the subscription is send to everyone on the bus. This is tied to a SQL db, and latency to the db can effect this timing.
Lastly, if you are using static routing, then that should work immediately, because the subscription is setup upon startup.