The documentation says very little about the priority attribute. One of the rare documentation references says the following:
priority (int): The task priority, a number between 0 and 9.
Defaults to the :attr:`priority` attribute.
(Ref: http://docs.celeryproject.org/en/latest/_modules/celery/app/task.html#Task.apply_async )
However, if you go to the bottom of the following page - http://docs.celeryproject.org/en/latest/userguide/calling.html - you will see:
priority
A number between 0 and 255, where 255 is the highest priority.
Supported by: RabbitMQ, Redis (priority reversed, 0 is highest).
These two pieces of information contradict. Also, I would assume that the docstring is more up-to-date...
Question is which one is relevant, and more importantly: How are priorities ordered - does priority=0 mean highest priority, or not? Or it depends on the backend, as the docs page says...
I've also made a GitHub issue for this - https://github.com/celery/celery/issues/4599
Update 1: I have found the following in the kombu changelog for v4.0 (http://docs.celeryproject.org/projects/kombu/en/latest/changelog.html):
Redis: Priority 0 is now lowest, 9 is highest. (backward incompatible)
This to match how priorities in AMQP works.
Fix contributed by Alex Koshelev.
Related
Followup to Configuration of MessageChannelPartitionHandler for assortment of remote steps
Even though the first question was answered (I think well), I think I'm confused enough that I'm not able to ask the right questions. Please direct me if you can.
Here is a sketch of the architecture I am attempting to build. Today, we have a job that runs a step across the cluster that works. We want to extend the architecture to run n (unbounded and different) jobs with n (unbounded and different) remote steps across the cluster.
I am not confusing job executions and job instances with jobs. We already run multiple job instances across the cluster. We need to be able to run other processes that are scalable in hte same way as the one we have defined already.
The source data is all coming from database which are known to the steps. The partitioner is defining the range of data for the "where" clause in the source database and putting that in the stepContext. All of the actual work happens in the stepContext. The jobContext simply serves to spawn steps, wait for completion, and provide the control API.
There will be 0 to n jobs running concurrently, with 0 to n steps from however many jobs running on the slave VM's concurrently.
Does each master job (or step?) require its own request and reply channel, and by extension its own OutboundChannelAdapter? Or are the request and reply channels shared?
Does each master job (or step?) require its own aggregator? By implication this means each job (or step) will have its own partition handler (which may be supported by the existing codebase)
The StepLocator on the slave appears to require a single shared replyChannel across all steps, but it appears to me that the messageChannelpartitionHandler requires a separate reply channel per step.
What I think is unclear (but I can't tell since it's unclear) is how the single reply channel is picked up by the aggregatedReplyChannel and then returned to the correct step. Of course I could be so lost I'm asking the wrong questions.
Thank you in advance
Does each master job (or step?) require its own request and reply channel, and by extension its own OutboundChannelAdapter? Or are the request and reply channels shared?
No, there is no need for that. StepExecutionRequests are identified with a correlation Id that makes it possible to distinguish them.
Does each master job (or step?) require its own aggregator? By implication this means each job (or step) will have its own partition handler (which may be supported by the existing codebase)
That should not be the case, as requests are uniquely identified with a correlation ID (similar to the previous point).
The StepLocator on the slave appears to require a single shared replyChannel across all steps, but it appears to me that the messageChannelpartitionHandler requires a separate reply channel per step.
The messageChannelpartitionHandler should be step or job scoped, as mentioned in the Javadoc (see recommendation in the last note). As a side note, there was an issue with message crossing in a previous version due to the reply channel being instance based, but it was fixed here.
I am developing a model comprised of m consecutive machines in which n agents must be processed in random sequences of machines. I want to have an intelligent agent (Reinforcement Learning) to, in each action, set the priority rule to rank queued agents in each machine.
The problem I have is that I am not sure if I am correctly changing the queueing order of agents in each queue, whenever the ranking rule is changed.
After some googling, I found this post, which seems to be what I want.:
Change priority rule of a Queue block at runtime in Anylogic
In this post, user Stuart Rossiter posted an interesting solution, (case 2 - using service block), which consists of sorting the agents queued on the embedded service's queue, using self.queue.sortAgents().
However, AnyLogic does not recognize this expression, as when I try to use it, I get the error "queue cannot be resolved or is not a field". After some more googling, I was able to find that the embedded queue of services can be accessed through service.seize.queue; however, even through this way, the method sortAgents() cannot be used, as I get an error saying that the method is undefined.
So, I am asking how can I reorder the agents in the embedded queue of a service after changing the ranking rule in runtime?
Obviously, I am assuming that playing with the task priority of the service would not be enough, as that would only be used to rank the order of agents that arrive to the queue after the ranking rule is set, i.e., it does not update the order of jobs queued before the ranking rule is changed (this is also clearly explained by the same user Stuart Rossiter).
Thank you.
I know the basics of AnyLogic/Process Modeling Library and am about to teach simulation of basic queues with AnyLogic, transitioning from Simul8 that I 've used for many years.
I have agents of two types, 1 and 2, sent to respective queues 1 and 2, which then feed a single "service" point, so that type 1 takes higher priority (that is, whenever service is ready to pull work, it pulls from queue 1 if it is non-empty, regardless of the size of queue 2). How to capture this as simply as possible?
Having seen the reference pages for a Queue object, my preliminary (unworked) idea is to use a single queue, and control agent priority by the Queue.QUEUING_PRIORITY- Priority-based" option.
For comparison, a solution in the Simul8 software is: set "service" routing-in discipline to "priority"; and assign different priorities to the two queues.
Yes you are right you cant use two queues as the pull from the queues will be done in a round robin fashion. See the screenshot below from the AnyLogic training textbook
You should use queueing in a single queue and you can have either a single parameterised source or two.
See example below
I have 2 sources and at each of them, I set the priority to a local variable inside my agent. Agents from source 1's variable is set to 1 and the from source set to 0.
Then inside the queue, I set the priority so that the agents from source 1 is always in front.
we're using the latest master build (at the time of this writing: https://github.com/linkedin/Burrow/commit/12e681a3a8a61f84f17677996dc3e6a2b79fac41)
Our Kafka-Brokers are running 1.1.0
We switched recently from https://github.com/Morningstar/kafka-offset-monitor to Burrow, because we're adding authorization to our Clusters.
Now, most of our consumer-lags are 0 most of the time (according to Burrow, whereas on kafka-offset-monitor they were around 1K - 100K most of the time - both are OK from our point of view).
For reasons unknown to us, the consumer lag "jumps" e.g. from 0 to 1.4 Billion(!) from one minute to the next, and back again after another minute. We have about 20 consumers on our main topic, and all of their lags jump - but by different amounts. Some "only" jump from 1k to 1M, others from 0 to the billions described above.
Is anybody else seeing this?
Is there a known reason or do we have to adjust our config? - We didn't change anything about the default config for the evaluation or notifications...
We use https://github.com/rgannu/burrow-graphite to report to graphite, and our alarming system is based on those metrics...
Any help is appreciated
Can there be more than two scheduling policies working at the same time in Linux Kernel ?
Can FIFO and Round Robin be working on the same machine ?
Yes, Linux supports no less then 4 different scheduling methods for tasks: SCHED_BATCH, SCHED_FAIR, SCHED_FIFO and SCHED_RR.
Regardless of scheduling method, all tasks also have a fixed hard priority (which is 0 for batch and fair and from 1- 99 for the RT schedulign methods of FIFO and RR). Tasks are first and foremost picked by priority - the highest priority wins.
However, with several tasks available for running with the same priority, that is where the scheduling method kicks in: A fair task will only run for its allotted weighted (with the weight coming from a soft priority called the task nice level) share of the CPU time with regard to other fair tasks, a FIFO task will run for a fixed time slice before yielding to another task (of the same priority - higher priority tasks always wins) and RR tasks will run till it blocks disregarding other tasks with the same priority.
Please note what I wrote above is accurate but not complete, because it does not take into account advance CPU reservation features, but it give the details about different scheduling method interact with each other.
yes !! now a days we have different scheduling policies at different stages in OS .. Round robin is done generally before getting the core execution ... fifo is done, at start stage of new coming process ... !!!