Queue manager with nested closures, commands, and dependencies - command

Question

I am pretty new to Laravel and am trying to create queues to handles my processing scrips.
The processing scripts will be triggers by a nightly cron job.
I have beanstalkd set up and working with queues already.
I also have very simply commands set up which will allow me to start a script using artisan.I am pretty new to Laravel and am trying to create queues to handles my processing scrips.
The processing scripts will be triggers by a nightly cron job.
I have beanstalkd set up and working with queues already.
I also have very simply commands set up which will allow me to start a script using artisan.
The main issue I am having is that I have dependencies in my scripts.
For example:
A -> B -> C
-> D
I need script A to run successfully first, then B need to run successfully, then C and D need to run but it doesn't really matter the order for C and D
Each of these are very different tasks and I want to keep them separate but need to manage the dependencies in my queue. I already have models and libraries created to handle my processing, that all works great.
Since I already have my logic in the models, I only need a small call for my queue.
Initially I wanted to just use closures and write it all in one file, a simple example is below.
Queue::push(function($jobA) use ($id)
{
ProcessA::doStuff($id);
$jobA->delete();
});
That works, but this is just for A, I still need to add B, C, and D.
So then I decided to add a closure inside of that.
Queue::push(function($jobA) use ($id)
{
$statusA = ProcessA::doStuff($id);
if($statusA)
{
Queue::push(function($jobB) use ($id)
{
ProcessB::doStuff($id);
$jobB->delete();
});
}
$jobA->delete();
});
This does not work.
There seems to be an issue with running a closure inside of a closure.
There may also be an issue with jobA possibly being deleted before jobB even starts.
That example was a little more simplified than what I actually need to do.
Here is a better example... after completion of A, I now need to get an array of things and loop through each of those and run B. Then from earlier I said I then have C and D which are spawned off of B... but I have yet to even get to the C and D part since this is getting a but complicated already.
Queue::push(function($jobA) use ($id)
{
$statusA = ProcessA::doStuff($id);
if($statusA)
{
$things = getThings($id);
foreach($things as $thing)
{
Queue::push(function($jobB) use ($thing)
{
ProcessB::doStuff($thing);
$jobB->delete();
});
}
}
$jobA->delete();
});
So nested closures still do not work.
My next step was to not nest the closures but instead write classes.
To start of the A process I would have to do
Queue::push('ProcessA', array('id' => $id));
Then
class ProcessA {
public function fire($job, $data)
{
$things = getThings($data['id']);
foreach($things as $thing)
{
Queue::push('ProcessB', array('thing' => $thing));
}
$job->delete();
}
}
Then
class ProcessB {
public function fire($job, $thing)
{
Queue::push('ProcessC', array('id' => $thing['id']));
Queue::push('ProcessD', array('id' => $thing['id']));
}
$job->delete();
}
ProcessC and ProcessD are irrelevant at this point, as long as they run.
This method seems to work but I think I was having issues sometimes when processA would completely finish before ProcessB was spawned. It was a long day figuring these things out and maybe this does work fine... need to test some more.
Also this method requires me to create a separate file for each ProcessA/B/C/D I am running due to the autoloading magic in Laravel or PHP 5.4. And as you can see each of these are pretty short. This just seems like a fairly complex way of doing things.
Does this method seemed flawed in any ways?
Should I be doing this in some other way?
Should I even bother with initializing this as artisan commands and switch to routes instead and call them with curl?
Is there any way to make nested closures work?

Answer 1

The simplest way I can come up with: Just fire an event when your first closure is done with its work, and then listen to that event somewhere else. When you catch it, push the next job onto the queue etc.