Lumen Database Queue first job always failing Allowed memory exhausted - queue

Question

I have a very odd situation where I set up a job to run in my Lumen database queue and all but the first job is processed. I do keep getting this particular error:
[2017-12-12 22:07:10] lumen.ERROR: Symfony\Component\Debug\Exception\FatalErrorException: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 702558208 bytes) in /var/www/vhosts/XXXXXXXXX$
Stack trace:
#0 /var/www/vhosts/XXXXXXXX/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(54): Laravel\Lumen\Application->handleShutdown()
#1 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns\{closure}()
#2 {main}
I have tried allowing the memory limit to go up but I keep getting the same error with differing values for the exhausted memory.
I find it very odd that it is always the first job and all of the rest of the jobs run perfectly fine. Should I be looking for bad data in the first job?
My code basically looks like this:
This is my Command file
namespace App\Console\Commands;
use App\Jobs\UpdateNNNAppListJob;
use Illuminate\Console\Command;
use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;
use Illuminate\Support\Facades\Queue;
class AddEmailsToAppList extends Command
{
/**
* The name and signature of the console command.
*
* #var string
*/
protected $signature = 'addemails:nnnmobileapp';
/**
* The console command description.
*
* #var string
*/
protected $description = 'This will add all mobile app users in the database to the nnn mobile app list.';
/**
* Create a new command instance.
*
* #return void
*/
public function __construct()
{
parent::__construct();
}
public function handle()
{
$chunkSize = 500; //this is the most middleware can handle with its bulk signup call
$emailChunks = $this->getEmailsToAdd($chunkSize);
$jobDelay = 120; //time between queued jobs
$jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method
foreach ($emailChunks as $emailChunk) {
Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
$jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
}
}
public function getEmailsToAdd($chunkSize)
{
$emails = DB::table('app_users')
->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
->select('app_users.email')
->get()
->chunk($chunkSize);
return $emails;
}
}
Here is my Job File
<?php
namespace App\Jobs;
use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;
class UpdateMmpAppListJob extends Job
{
/**
* Array of emails to update list with
* #var array
*/
protected $emailArray;
/**
* The number of times the job may be attempted.
*
* #var int
*/
public $tries = 2;
public function __construct($emailArray)
{
$this->emailArray = $emailArray;
}
public function handle()
{
$listCodeToAddTo = 'NNNAPP';
$sourceId = 'NNNNNNN';
$middlewareApi = new MiddlewareApi();
try {
$middlewareApi->post_add_customer_signup_bulk($listCodeToAddTo, $this->emailArray, $sourceId);
} catch (\Exception $e) {
Log::error('An error occurred with theUpdateMmpAppListJob: ' . $e);
mail('djarrin#NNN.com', 'UpdateNnnAppListJob Failure', 'A failure in the UpdateNnnAppListJob, here is the exception: ' . $e);
}
}
public function failed(\Exception $exception)
{
mail('djarrin#moneymappress.com', 'Push Processor Que Failure', 'A failure in the UpdateMmpAppListJob, here is the exception: ' . $exception);
}
}
Any help/suggestions on this issue would be appreciate.

Answer 1

Your code calls ->get() which will load the entire result into memory. This causes the huge memory allocation that you're seeing. Remove it and let ->chunk(...) work with the database builder instead of the in-memory Collection that get() has returned. You would also have to provide a callback to chunk that processes every chunk.
public function handle() {
$chunkSize = 500; //this is the most middleware can handle with its bulk signup call
$jobDelay = 120; //time between queued jobs
$jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method
DB::table('app_users')
->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
->select('app_users.email')
->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
$jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
});
}

Answer 2

The above concept is correct but this syntax was required to get past the
[2017-12-14 22:08:26] lumen.ERROR: RuntimeException: You must specify an orderBy clause when using this function. in /home/vagrant/sites/nnn/vendor/illuminate/database/Query/Builder.php:1877
This is for Lumen 5.5:
public function handle()
{
$chunkSize = 500; //this is the most middleware can handle with its bulk signup call
$jobDelay = 120; //time between queued jobs
$jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method
$emails = DB::table('app_users')
->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
->select('app_users.email')->orderBy('app_users.id', 'desc');
$emails->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
$jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
});
}