I want to try out Azure Functions with the following project:
Triggered by time (every 30 minutes) my initial function1 puts some data in a queue1.
This queue1 triggers another function2 that calls an external REST API, modifies the response and puts the results in another queue3.
This queue3 starts another functions3 doing the rest.
My problem is that the REST API has a rate limiting. So if my function1 puts 100 items in the queue1 and the function2 is called 100 times parallel, my API calls will be blocked. I therefore need some kind of throttling.
How would you achieve that? I could tell function2 to wait a specific time and then add the item back to queue1, but since everything is parallel I might run in to a deadlock?
Thanks in advance for ideas!
I would recommend that you take a look at Azure Durable Functions here. The Durable Functions framework allows you to orchestrate complex workflows and manage state. In your example, you could use Durable Functions to work around the rate limiting issue
is called 100 times parallel
You can limit this (to some extent) by configuring host.json:
batchSize for Storage Queues
maxConcurrentCalls for Service Bus
If that's not enough, you could do something more sofisticated:
Function 1 knows how many items it has to process, so it could calculate the "ideal" distribution of those for the next 30 minutes
When adding messages to queue1, it could set the time when the message should be picked up (ScheduledEnqueueTimeUtc for Service Bus or initialVisibilityDelay for CloudQueue)
Function2 will be called "on schedule" which should prevent throttling, if the total amount of messages is not too high
Related
I am wondering how to process one message at a time using Googles pub/sub functionality in Go. I am using the official library for this, https://pkg.go.dev/cloud.google.com/go/pubsub#section-readme. The event is being consumed by a service that runs with multiple instances, so any in memory locking mechanism will not work.
I realise that it's an anti-pattern to do this, so let me explain my use-case. Using mongoDB I store an array of objects as an embedded document for each entity. The event being published is modifying parts of this array and saves it. If I receive more than one event at a time and they start processing exactly at the same time, one of the saves will override the other. So I was thinking a solution for this is to make sure that only one message will be processed at a time, and it would be nice to use any built-in functionality in cloud pub/sub to do so. Otherwise I was thinking of implementing some locking mechanism in the DB but i'd like to avoid that.
Any help would be appreciated.
You can imagine 2 things:
You can use ordering key in PubSub. Like that, all the message in relation with the same object will be delivered in order and one by one.
You can use a PUSH subscription to PubSub, to push to Cloud Run or Cloud Functions. With Cloud Run, set the concurrency to 1 (it's by default with Cloud Functions gen1), and set the max instance to 1 also. Like that you can process only one message at a time, all the other message will be rejected (429 HTTP error code) and will be requeued to PubSub. The problem is that you can parallelize the processing as before with ordering key
A similar thing, and simpler to implement, is to use Cloud Tasks instead of PubSub. With Cloud Tasks you can set a rate limit on a queue, and set the maxConcurrentDispatches to 1 (and you haven't to do the same with Cloud Functions max instances or Cloud Run max instances and concurrency)
I am building an application that will need to run hundreds of short running tasks every minute. These functions are not doing anything special other than making calls to an HTTP endpoint. I need a reliable mechanism for scheduling these invocations every minute indefinitely. Failures to run at the scheduled time cannot be tolerated. I have considered the following options for the scheduler:
AWS Lambda
Mesosphere Chronos
Cron
Python Celery
Obviously there is a trade off between cost, maintainability (I will need to update the logic of these functions every once in a while), and reliability.
My question is, which of these options would be the most appropriate if I am most concerned about consistency/reliability? Are there options I'm missing that I should consider?
As you already mentioned, there are multiple technologies that could help you do this, I would say that the trick is more to find the logic flow/model to use.
For example, If the number of tasks are not fixed, a publish/subscribe pattern could apply, for this something like rabbitMQ or AWS SQS could be used.
There are multiple ways about how to submit a task to the queue and also how to de-queue, you could have multiple workers reading/waiting for events in where they could read one by one or by chunks (based on the num of cores per server) all this bound to the speed and precision you may want.
Scaling I would say is easier since if need more speed (precision to do all tasks every minute) just need to add more workers.
For more ideas check this article Using AWS Lambda with Amazon DynamoDB it covers a stream-based model / event-sourcing.
For a test, I created a new function app. I added two functions, one was an http trigger that when invoked, pushed 500 messages to a queue. The other, a queue trigger to read the messages. The queue trigger function code, was setup to read a message and randomly sleep from 1 to 30 seconds. This was intended to simulate longer running tasks.
I invoked the http trigger to create the messages, then watched the que fill up (messages were processed by the other trigger). I also wired up app insights to this function app, but I did not see is scale beyond 1 server.
Do Azure functions scale up soley on the # of messages in the que?
Also, I implemented these functions in Powershell.
If you're running in the Azure Functions consumption plan, we monitor both the length and the throughput of your queue to determine whether additional VM resources are needed.
Note that a single function app instance can process multiple queue messages concurrently without needing to scale across multiple VMs. So if all 500 messages can be consumed relatively quickly (again, in the consumption plan), then it's possible that you won't scale at all.
The exact algorithm for scaling isn't published (it's subject to lots of tweaking), but generally speaking you can expect the system to automatically scale you out if messages are getting added to the queue faster than your functions can process them. Your app will also scale out if the latency of the first message in the queue is continuously increasing (meaning, messages are sitting idle and not getting processed). The time between VMs getting added is usually in the tens of seconds.
There are some thresholds based on queue count as well. For example, the system tries to ensure that there is at least 1 VM for every 1K queue messages, but usually the scale decisions are based on message throughput as I described earlier.
I think #Chris Gillum put it well, it's hard for us to push the limits of the server to the point that things will start to scale.
Some other options available are:
Use durable functions and scale with Threading:
https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-cloud-backup
Another method could be to use Event Hubs which are designed for massive scale. Instead of queues, have Function #1 trigger an Event, and your Function #2 subscribed to that Event Hub trigger. Adding Streaming Analytics, could also be an option to more fully expand on capabilities if needed.
I have an application that queues requests received from callers. Each request results in a call to an external web service that has restrictions on how many calls we can make to it. E.g. we can only make X calls per minute.
Each request is added to Quartz.NET scheduler and I need to be able to schedule jobs in such a way as to not violate the terms of the external web service.
I've considered keeping track somehow of the last time a job was added to scheduler and making sure jobs are triggered N milliseconds apart (i.e. each job coming in is set to trigger at LastJobTime + N), where N = (60000/X). However, I'm not sure if this is reasonable.
Is there a better way to accomplish this? If not, must I keep track of LastJobTime myself or can Quartz.NET provide some help here?
Thanks
You can create a TriggerListener that implements the ITriggerListener interface. The ITriggerListener give you the possibility to veto a job. Just count how many calls do you made in this minute and if you over your call contingent, veto the Job.
I have defined some simple BPM flows (F1) and deployed in activiti-rest.war. For simplicity, I have take a simple start-end flow.
I have written a REST client to execute the flow (F1) in parallel threads (20) with its required parameters for 1000 http requests.
Problem: I can see the flows are running sequentially, one by one response for the 20 parallel threads. It took a time of around 60 secs to complete with 20 threads (even when increased to 50 threads) it is the same.
Activiti Version : 5.15
What should be the problem here ?. Any help will be really useful.
activiti-rest/service/runtime/process-instances - Rest URL used to start the instance
Thanks,
Yoka
At last i found the solution.
It could be of two reasons
1) Make sure task's "Exclusive" property is set to false. But it needs more analysis on how your process task will be running. Refer the below link for further information
http://www.activiti.org/userguide/#exclusiveJobs
2) If you run the activity rest application and the client process on a dual-core machine. It might be difficult to assess the response time.
Thanks,
Yoka