I'm planning a mechanism whose usage scenarios would be like cron's. It's a clock-ish mechanism that attempts task execution at prespecified times. Cron doesn't seem suitable, because these tasks trigger Scala method calls and the queue stored on a cloud database.
I imagine it like this: every x minutes, tasks' execution dates are retrieved from the database, and compared against current time, if the task is over-due it is executed and removed from queue.
My question is: how do I run the aforementioned check every x minutes on a distributed environment?
All advice encouraged.
I think the Akka scheduler might be what you are looking for. Here's a link to the Akka documentation and here's another link describing how to use Akka in Play.
Update: as Viktor Klang points out Akka is not a scheduler, however it does allow you to run a task periodically. I've used it in this mode quite successfully.
The best known library for this is Quartz Scheduler.
Related
I would like schedule a task that gets executed periodically after the completion of task. Previously I used ScheduledExecutorService.scheduleWithFixedDelay but now in vertx I am thinking whether it may cause any issue since vertx already uses thread for event loop and worker verticles.
I checked Vertx.setPeriodic, but that just executes periodically without checking or waiting for the task to complete before scheduling other.
With all options explored currently I have a workaround where I use Vertx.setTimer to schedule the task and on completion I am calling Vertx.setTimer again inside the handler.
On high level the schedule task will query records from one table and update other table.
Anyone has any other better solution, please guide me.
Vertx version - 3.9.4
To my best knowledge ScheduledExecutorService.scheduleWithFixedDelay shouldn't cause any problems with Vert.x, and you can continue using it.
Your suggestion of setting timer in a callback is the better way of solving that, though.
I am building an application that will need to run hundreds of short running tasks every minute. These functions are not doing anything special other than making calls to an HTTP endpoint. I need a reliable mechanism for scheduling these invocations every minute indefinitely. Failures to run at the scheduled time cannot be tolerated. I have considered the following options for the scheduler:
AWS Lambda
Mesosphere Chronos
Cron
Python Celery
Obviously there is a trade off between cost, maintainability (I will need to update the logic of these functions every once in a while), and reliability.
My question is, which of these options would be the most appropriate if I am most concerned about consistency/reliability? Are there options I'm missing that I should consider?
As you already mentioned, there are multiple technologies that could help you do this, I would say that the trick is more to find the logic flow/model to use.
For example, If the number of tasks are not fixed, a publish/subscribe pattern could apply, for this something like rabbitMQ or AWS SQS could be used.
There are multiple ways about how to submit a task to the queue and also how to de-queue, you could have multiple workers reading/waiting for events in where they could read one by one or by chunks (based on the num of cores per server) all this bound to the speed and precision you may want.
Scaling I would say is easier since if need more speed (precision to do all tasks every minute) just need to add more workers.
For more ideas check this article Using AWS Lambda with Amazon DynamoDB it covers a stream-based model / event-sourcing.
Many popular task queues (such as Google GAE TaskQueue, Celery) have the ETA/Countdown feature, which allows a task to be put into the queue after xxx seconds.
I am working on a project that needs a task queue with the ETA feature. However, there are some limitations that I have to use the Google Pubsub messaging system. Pubsub does not have the ETA feature. I am wondering how to implement a reliable and scalable ETA mechanism for a task queue. Both general architecture ideas and actual code samples are welcome.
Our system enqueues 600-2000 tasks/second, and about 10% of them need to have ETA. It is a distributed system and performance-critical.
I tried to trace the source code of celery, but couldn't find the actual logic of handling the ETA. It would also be good if someone can point me to the file/code of Celery that handle ETA.
I think I might have found how Celery did it. In eventlet.py, it uses eventlet's spawn_after feature to delay the worker creation "ETA" seconds.
secs = max(eta - monotonic(), 0)
g = self._spawn_after(secs, entry)
MarkLogic Scheduled Tasks cannot be configured to run at an interval less than a minute.
Is there any way I can execute an XQuery module at an interval of 1 second?
NOTE:
Considering the situation where the Task Server is fully loaded and I need to make sure that the secondly scheduled task gets the Task Server thread whenever it needs.
Please let me know if there is anything in MarkLogic that can be used to achieve this.
Wanting rapid-fire scheduled tasks may be a hint that the design needs rethinking.
Even running a task once a minute can be risky, and needs careful thought to manage the possibilities of overlapping tasks and runaway tasks. If the application design calls for a scheduled task to run once a second, I would raise that as a potentially serious problem. Back up a few steps, and if necessary ask a new question about the higher-level problem that led to looking at scheduled tasks.
There was a sub-question about managing queue priority for tasks. Task priorities can handle some of that. There are two priorities: normal and higher. The Task Server empties the higher-priority queue first, then the normal queue. But each queue is still a simple queue, and there's no way to change priorities after a task has been spawned. So if you always queue tasks with priority=higher, then they'll all be in the higher priority queue and they'll all run in order. You can play some games with techniques like using server fields as signals to already-running tasks. But wanting to reorder tasks within a queue could be another hint that the design needs rethinking.
If, after careful thought about all the pitfalls and dangers, I decided I needed a rapid-fire task of some kind.... I would probably do it using external requests. Pick any scripting language and write a simple while loop with an HTTP request to the MarkLogic cluster. Even so, spend some time thinking about overlapping requests and locking. What happens if the request times out on the client side? Will it keep running on the server? Will that lead to overlapping requests and require deadlock resolution? Could it lead to runaway resource consumption?
Avoid any ideas that use xdmp:sleep. That will tie up a Task Server thread during the sleep period, and then you'll have two problems.
Can WWF handle high throughput scenarios where several dozen records are 'actively' being processed in parallel at any one time?
We want to build a workflow process which handles a few thousand records per hour. Each record takes up to a minute to process, because it makes external web service calls.
We are testing Windows Workflow Foundation to do this. But our demo programs show processing of each record appear to be running in sequence not in parallel, when we use parallel activities to process several records at once within one workflow instance.
Should we use multiple workflow instances or parallel activities?
Are there any known patterns for high performance WWF processing?
You should definitely use a new workflow per record. Each workflow only gets one thread to run in, so even with a ParallelActivity they'll still be handled sequentially.
I'm not sure about the performance of Windows Workflow, but from what I have heard about .NET 4 at Tech-Ed was that its Workflow components will be dramatically faster then the ones from .NET 3.0 and 3.5. So if you really need a lot of performance, maybe you should consider waiting for .NET 4.0.
Another option could be to consider BizTalk. But it's pretty expensive.
I think the common pattern is to use one workflow instance per record. The workflow runtime runs multiple instances in parallel.
One workflow instance runs one thread at a time. The parallel activity calls Execute method of each activity sequentially on this single thread. You may still get performance improvement from parallel activity however, if the activities are asynchronous and spend most of the time waiting for external process to finish its work. E.g. if activity calls an external web method, and then waits for a reply - it returns from Execute method and does not occupy this thread while waiting for the reply, so another activity in the Parallel group can start its job (e.g. also call to a web service) at the same time.