I am struggling to resolve a simulation problem in Anylogic.
Problem:
1. I want to make 2 operators which work in the same machine to have different operation time (for example, when a part arriving at the machine, worker1 spend 3 minutes but worker2 spend 5 minutes.) I have tried the resource pool but the 2 workers that are seized always spend the same time at the machine.
How can I modelize this problem by using flowchart blocks?
Just use a Seize ---Service---Delay---Release structure (assuming worker1 spends the 3 mins at the start of the processing).
The Seize/Release blocks seize and release worker2. The Service block represents worker1 working for 3 mins. The Delay block covers the remaining 2 mins for already-seized worker2.
You do it like this (rather than with 2 Service blocks, the first using worker1 and worker2 for 3 mins, the second using worker2 for another 2 mins) because you want to ensure worker2 doesn't get seized by another request when the first Service block ends. (In some models this eventuality won't be a problem so you can use two Service blocks instead but I wouldn't because the way here makes explicit that the same worker2 worker is held for the whole 5 mins.
You can also put all this in a custom block (check the AnyLogic help) so it's a reusable single block.
Related
I 1 celery broker and several celery workers, all communicating with rabbitMQ. In my setup, I send several tasks to my celery workers, they'll process all the tasks (it takes ~1 hour), and then I'll manually terminate my celery workers.
I want to move towards a system where if a celery worker id 'idle' (which I define as: has 0 active tasks for a time period of timeout_seconds, which I will define beforehand), the worker will be terminated programatically. All workers will have approx the same # of tasks to run, and will all go 'idle' around the same time.
I have code set up that lets me terminate workers, but I am not sure how to detect that a worker is 'idle' and ready for termination. I think I want to use a signal but it doesn't look like there is one that fits my requirement
Here where I work we have a task that is doing basically what you want - automatically scales up/down the cluster depending on the "situation". The key in this process is the Celery inspect/control API, so I suggest you get familiar with it. This is an area that is not well-documented so start with the following:
insp = celery_app.control.inspect()
active_queues = insp.active_queues()
# Note: between these two calls some nodes may shut down and disappear
# from the dictionary so may need to deal with this...
active_stats = insp.active()
You can do this in a separate IPython session while your Celery cluster runs tasks, and look at what is there...
Is there a way to have parallel services and/or delays to occur per agent and move on with the activity that takes the longest. For example if I have an agent that can be painted and serviced at the same time, each requiring a different resource pool with different processing times but the agent will move forward when the process that took the longest is over.
Use a Split block to split your agent into two for the two parallel tasks (Service blocks) and then use a Combine to combine them back again afterwards (with the Combine block outputting the original agent 1).
You can also use RestrictedAreaStart and RestrictedAreaEnd blocks (capacity 1) around the split/combine area to ensure that other agents can't 'jump in' whilst the longest parallel process is still running (but the shorter one has already finished).
Something like the below (with resource pools).
Probably easiest to dynamically setup delay duration and resources needed from within 1 Service element:
calculate the duration your agent will use in each case (painting = 5 mins, servicing = 10 mins) --> use the longer value as the service delay
Also, make the agent require 1 painter and 1 service-engineer as resources.
Only drawback: Your painter will stay for 10 mins as well.
Alternative approach would likely involve creating your own, purely agent-based setup with seizing& releasing
I'd like to keep track of only the recent data and also employ the help of Vector clocks in resolving issues so I can easily discard data via L-W-W rule.(last write wins)
Say we have 3 nodes:
- Node1
- Node2
- Node3
Then we would use Vector clocks to keep track of causality and concurrency on each events/changes. We represent Vector clocks initially with
{Node1:0, Node2:0, Node3:0}.
For instance Node1 gets 5 local changes it would mean we increment its clock by 5 increments that would result into
{Node1: 5, Node2:0, Node3:0}.
This would be normally okay right?
Then what if at the same time Node2 updates its local and also incremented its clock resulting into
{Node1:0, Node2:1, Node3:0}.
At some point Node1 sends an event to Node3 passing the updates and piggybacking its vectorclock. So Node3 which has a VC of {Node1:0, Node2:0, Node3:0} would easily just merge the data and clock as there are no changes on it yet.
The problem I'm thinking about how to deal with is what would happen if Node2 sends an event to update into Node3 passing it's own VC and updates.
What would happen to the data and the clocks. How do I apply Last Write wins here when the first one that gets written to Node3 which was from Node1 would basically appear as the later write as it have a greater VC value on its own clock.
Node3's clock before merging: {Node1: 5, Node2: 0 , Node3: 1}
Node2's messagevc that Node3 received: {Node1:0, Node2:1, Node3:0}
How do I handle resolving data on concurrent VCs?
This is a good question. You're running into this issue because you are using counters in your vector clocks, and you are not synchronizing the counters across nodes. You have a couple of options:
Submit all writes through one primary server. The primary server can apply a total order to all of the writes and then send them to the individual nodes to be stored. It would be helpful to have some background on your system. For example, why are there three independent nodes? Do they exist to provide replication and availability? If so, this primary server approach would work well.
Keep your server's times in sync, as described in Google's Spanner paper. Then, instead of using a monotonically increasing counter for each node in the vector clock, you can use a timestamp based off of the server's time. Again, having background on your system would be helpful. If your system just consists of human users submitting writes, then you might be able to get away with keeping your server's times loosely in sync using NTP without violating the LWW invariant.
In a distributed system, 5 processes, using zookeeper to coordinate.
I need these processes to run one by one in every round.
run order is dynamic, but is known for every round.
Any zookeeper recipe can do this? Thanks
for example:
round 1: 1 2 3 4 5
round 2: 3 2 4 1 5
Your problem is not fully defined:
In a situation when a given process fails to start do we have any kind of a timeout or does the system just halt?
Do the processes terminate between rounds, so that they are started by some external mechanism for every round?
Do we care about process termination status? If a process started but then failed in the middle of the execution, should the next process start?
In the simplest case, when we
accept halting
don't differentiate between a successful termination and a failure
assume that every process is somehow started for every round and knows its position in the current round
we could use the following approach (probably can be simplified):
For a given round either pre-setup or make the first process in the round create a path /round.
The first process
checks that getChildren('/round') returns an empty list
creates an ephemeral znode /round/exec_1
creates a permanent znode /round/start_1
Every other process number N
gets the list of /round's children
checks if /round/start_{N-1} exists
if yes, then checks /round/exec_{N-1} and waits for it to be deleted (using watcher)
if not, wait for it to appear (using watcher), then act as if the node was already there.
If the answers to the questions above are different, this protocol can be adjusted.
But the general idea here is to define how we indicate that a process is good to go: we register the fact that a previous process has started and then wait for it to terminate.
I'm trying to clean up an enterprise BI system that currently is using a prioritized FIFO scheduling algorithm (so a priority 4 report from Tuesday will be executed before priority 4 reports from Thursday and priority 3 reports from Monday.) Additional details:
The queue is never empty, jobs are always being added
Jobs range in execution time from under a minute to upwards of 24 hours
There are 40 some odd identical app servers used to execute jobs
I think I could get optaPlanner up and running for this scenario, with hard rules around priority and some soft rules around average time in the queue. I'm new to scheduling optimization so I guess my question is what should I be looking for in this situation to decide if optaPlanner is going to help me or not?
The problem looks like a form of bin packing (and possibly job shop scheduling), which are NP-complete, so OptaPlanner will do better than a FIFO algorithm.
But is it really NP-complete? If all of these conditions are met, it might not be:
All 40 servers are identical. So running a priority report on server A instead of server B won't deliver a report faster.
All 40 servers are identical. So total duration (for a specific input set) is a constant.
Total makespan doesn't matter. So given 20 small jobs of 1 hour and 1 big job of 20 hours and 2 machines, it's fine that it takes all small jobs are done after 10 hours before the big job starts, given a total makespan of 30 hours. There's no desire to reduce the makespan to 20 hours.
"the average time in the queue" is debatable: do you care about how long the jobs are in the queue until they are started or until they are finished? If the total duration is a constant, this can be done by merely FIFO'ing the small jobs first or last (while still respecting priority of course).
There are no dependencies between jobs.
If all these conditions are met, OptaPlanner won't be able to do better than a correctly written greedy algorithm (which schedules the highest priority job that is the smallest/largest first). If any of these conditions aren't met (for example you buy 10 new servers which are faster), then OptaPlanner can do better. You just have to evaluate if it's worth spending 1 thread to figure that out.
If you use OptaPlanner, definitely take a look at real-time scheduling and daemon mode, to replan as new reports enter the system.