Why the Storage isn't working on inside a job? - queue

I have some users who need to extract a large amount of data from from the database. I am creating a job that receive the parameters, query the database and write the data on a txt file. And as far as the file is ready I will send a notification e-mail.
To store the file locally, I am using the Storage facade inside the job.
If I run my code on a regular controller, the file is generated, but when I run the same command inside a job, the file is not generated (even having the job processed).
I am running laravel 5.4 on php 7.2. and have linked the storage (php artisan storage:link)
Since my code have a lot of complexity (consulting models, mail and so on) I created a simple test controller and a simple test job to isolate the file creation from the rest of the code. The same problem is happening in the simple version.
Here is the controller:
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use Storage;
use App\Jobs\testeStorageJob;
class testController extends Controller
{
public function test(){
Storage::put('testFile.txt', 'Some Content on a regular controller');
testeStorageJob::dispatch();
}
}
and here is the job:
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Storage;
class testeStorageJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $tries = 1;
/**
* Create a new job instance.
*
* #return void
*/
public function __construct()
{
//
}
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
Storage::put('testFileQueue.txt', 'Some content on a queue');
}
}
I've checked the filesystem config and it is the default values.
I was expecting to have 2 files on the "Storage/App" folder testFile.txt and testFileQueue.txt, but I only have testFile.txt.

Related

Laravel Nova: Queuable actions create single job for multiple resources

I have a simple Action in Laravel Nova which updates stock inventory via an API call. I want to be able to queue this as individual jobs for each product that requires an update to stagger the API calls.
When I add my action to a resource and run it via the index page by selecting multiple products and running the action, only a single job is created in my jobs table.
So when the queue is processed, rather than queuing each product for an individual update, a single job is run which loops all selected products and makes multiple API requests in quick succession, which is not my desired result.
Is there a way for the action to create a job for each resource that the action is run against?
Action Class
class UpdateInventory extends Action implements ShouldQueue
{
use InteractsWithQueue, Queueable, SerializesModels;
public function __construct()
{
$this->connection = config('queue.default');
$this->queue = 'inventory_update';
}
/**
* Perform the action on the given models.
*/
public function handle(ActionFields $fields, Collection $products)
{
foreach ($products as $product) {
try {
$inventoryService = resolve(InventoryService::class);
$inventoryService->updateProductInventory($product); // <- API calls within
$this->markAsFinished($product);
} catch (\Exception $e) {
$this->markAsFailed($product, $e);
}
}
return Action::message("Inventory update started");
}
}
class UpdateInventory extends Action implements ShouldQueue
{
use InteractsWithQueue, Queueable, SerializesModels;
/**
* The number of models that should be included in each chunk.
*
* #var int
*/
public static $chunkCount = 1;
}
You may instruct Nova to your queue actions as a batch. That requires migration so you create batches table with following command php artisan queue:batches-table and migrate your database. This table contains total_jobs, pending_jobs, failed_jobs, failed_job_ids fields so you can keep track each job. If each job has one resource, it should resolve your issue.
More information:
Laravel Nova Actions - Job Batching
Laravel Queues - Job Batching

Symfony 4.2 Creating Repository as service in vendor/acme/demo-bundle

I am working on a third party bundle which is in the vendor/ directory.
I have an Entity class which looks like this:
/**
* #ORM\Entity(repositoryClass="Acme\DemoBundle\Repository\ArticleRepository")
* #ORM\Table(name="acme_demo_article")
*/
class Article
And a Repository class like this:
class ArticleRepository extends ServiceEntityRepository
{
public function __construct(RegistryInterface $registry)
{
parent::__construct($registry, Article::class);
}
}
This generates the following error:
The "Acme\DemoBundle\Repository\ArticleRepository" entity repository
implements
"Doctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepositoryInterface",
but its service could not be found. Make sure the service exists and
is tagged with "doctrine.repository_service".
If i remove the repositoryClass from the entity definition, I dont have the error anymore and i can use doctrine as such from my controller:
this->getDoctrine()->getRepository(Article::class)->findBy([], null, $limit, ($page - 1) * $limit);
I tried adding the repository as a service in the bundle service definition but it does not change anything:
vendor/Acme/demo-bundle/Resources/config/services.yaml
services:
Acme\DemoBundle\Repository\:
resource: '../../Repository/ArticleRepository.php'
autoconfigure: true
tags: ['doctrine.repository_service']
bin/console debug:autowire or debug:container wont show the service.
I also tried adding the extension:
namespace Acme\BlogBundle\DependencyInjection;
use Symfony\Component\Config\FileLocator;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Extension\Extension;
use Symfony\Component\DependencyInjection\Loader\YamlFileLoader;
class AcmeBlogExtension extends Extension
{
public function load(array $configs, ContainerBuilder $container)
{
$loader = new YamlFileLoader(
$container,
new FileLocator(__DIR__.'/../Resources/config')
);
$loader->load('services.xml');
}
}
Did not work either. I dont have the impression that the extension is being called. I tried adding a constructor to it and dump, die in the constructor, but there are no results of the dump.
So my question is how do i define my repositories as a service from the vendor directory ?
The source code is overhere: https://github.com/khalid-s/sf4-bundle-test
After much struggling, i succedded in my task. I dont think that's it should be done like this, but if this can help someone...
I added in my DependencyInjection folder of the bundle:
class AcmeBlogExtension extends Extension
{
public function load(array $configs, ContainerBuilder $container)
{
$loader = new YamlFileLoader(
$container,
new FileLocator(__DIR__.'/../Resources/config')
);
$loader->load('services.yaml');
}
}
I created a compiler (this is the part which i struggled to figure out) to register my service
class RepositoryCompiler implements CompilerPassInterface
{
/**
* #inheritdoc
*/
public function process(ContainerBuilder $container)
{
$container->register('acme_blog.repository', ArticleRepository::class);
}
}
I added in my Bundle class:
class AcmeBlogBundle extends Bundle
{
/** #info this function normally is useless */
public function getContainerExtension()
{
// This is only useful if the naming convention is not used
return new AcmeBlogExtension();
}
/**
* #inheritDoc
*/
public function build(ContainerBuilder $container)
{
$container->addCompilerPass(new RepositoryCompiler());
parent::build($container);
}
}
And finally the service itself:
services:
Acme\BlogBundle\Repository\:
resource: '../../Repository/*Repository.php'
autoconfigure: true
autowire: true
tags: ['doctrine.repository_service']
The autoconfigure and autowire are useless since they are not taken into consideration when i debug:container which looks like this:
php bin/console debug:container acme
Information for Service "acme_blog.article.repository"
=======================================================
---------------- -----------------------------------------------
Option Value
---------------- -----------------------------------------------
Service ID acme_blog.article.repository
Class Acme\BlogBundle\Repository\ArticleRepository
Tags doctrine.repository_service
Public yes
Synthetic no
Lazy no
Shared yes
Abstract no
Autowired no
Autoconfigured no
---------------- -----------------------------------------------
One very important note which made me loose a lot of time:
Do clear your cache after every change to your services. Even in dev
mode they are not reloaded after every refresh

Symfony serializer - set circular reference global

Is there any way to set the circular reference limit in the serializer component of Symfony (not JMSSerializer) with any config or something like that?
I have a REST Application with FOSRestBundle and some Entities that contain other entities which should be serialized too. But I'm running into circular reference errors.
I know how to set it like this:
$encoder = new JsonEncoder();
$normalizer = new ObjectNormalizer();
$normalizer->setCircularReferenceHandler(function ($object) {
return $object->getName();
});
But this has to be done in more than one controller (overhead for me).
I want to set it globally in the config (.yml) e.g. like this:
framework:
serializer:
enabled: true
circular_limit: 5
Found no serializer API reference for this so I wonder is it possible or not?
For a week have I been reading Symfony source and trying some tricks to get it work (on my project and without installing a third party bundle: not for that functionality) and I finally got one. I used CompilerPass (https://symfony.com/doc/current/service_container/compiler_passes.html)... Which works in three steps:
1. Define build method in bundle
I choosed AppBundle because it is my first bundle to load in app/AppKernel.php.
src/AppBundle/AppBundle.php
<?php
namespace AppBundle;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\HttpKernel\Bundle\Bundle;
class AppBundle extends Bundle
{
public function build(ContainerBuilder $container)
{
parent::build($container);
$container->addCompilerPass(new AppCompilerPass());
}
}
2. Write your custom CompilerPass
Symfony serializers are all under the serializer service. So I just fetched it and added to it a configurator option, in order to catch its instanciation.
src/AppBundle/AppCompilerPass.php
<?php
namespace AppBundle;
use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Reference;
class AppCompilerPass implements CompilerPassInterface
{
public function process(ContainerBuilder $container)
{
$container
->getDefinition('serializer')
->setConfigurator([
new Reference(AppConfigurer::class), 'configureNormalizer'
]);
}
}
3. Write your configurer...
Here, you create a class following what you wrote in the custom CompilerPass (I choosed AppConfigurer)... A class with an instance method named after what you choosed in the custom compiler pass (I choosed configureNormalizer).
This method will be called when the symfony internal serializer will be created.
The symfony serializer contains normalizers and decoders and such things as private/protected properties. That is why I used PHP's \Closure::bind method to scope the symfony serializer as $this into my lambda-like function (PHP Closure).
Then a loop through the nomalizers ($this->normalizers) help customize their behaviours. Actually, not all of those nomalizers need circular reference handlers (like DateTimeNormalizer): the reason of the condition there.
src/AppBundle/AppConfigurer.php
<?php
namespace AppBundle;
class AppConfigurer
{
public function configureNormalizer($normalizer)
{
\Closure::bind(function () use (&$normalizer)
{
foreach ($this->normalizers as $normalizer)
if (method_exists($normalizer, 'setCircularReferenceHandler'))
$normalizer->setCircularReferenceHandler(function ($object)
{
return $object->getId();
});
}, $normalizer, $normalizer)();
}
}
Conclusion
As said earlier, I did it for my project since I dind't wanted FOSRestBundle nor any third party bundle as I've seen over Internet as a solution: not for that part (may be for security). My controllers now stand as...
<?php
namespace StoreBundle\Controller;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Method;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
class ProductController extends Controller
{
/**
*
* #Route("/products")
*
*/
public function indexAction()
{
$em = $this->getDoctrine()->getManager();
$data = $em->getRepository('StoreBundle:Product')->findAll();
return $this->json(['data' => $data]);
}
/**
*
* #Route("/product")
* #Method("POST")
*
*/
public function newAction()
{
throw new \Exception('Method not yet implemented');
}
/**
*
* #Route("/product/{id}")
*
*/
public function showAction($id)
{
$em = $this->getDoctrine()->getManager();
$data = $em->getRepository('StoreBundle:Product')->findById($id);
return $this->json(['data' => $data]);
}
/**
*
* #Route("/product/{id}/update")
* #Method("PUT")
*
*/
public function updateAction($id)
{
throw new \Exception('Method not yet implemented');
}
/**
*
* #Route("/product/{id}/delete")
* #Method("DELETE")
*
*/
public function deleteAction($id)
{
throw new \Exception('Method not yet implemented');
}
}
The only way I've found is to create your own object normalizer to add the circular reference handler.
A minimal working one can be:
<?php
namespace AppBundle\Serializer\Normalizer;
use Symfony\Component\Serializer\Normalizer\ObjectNormalizer;
use Symfony\Component\PropertyAccess\PropertyAccessorInterface;
use Symfony\Component\PropertyInfo\PropertyTypeExtractorInterface;
use Symfony\Component\Serializer\Mapping\Factory\ClassMetadataFactoryInterface;
use Symfony\Component\Serializer\NameConverter\NameConverterInterface;
class AppObjectNormalizer extends ObjectNormalizer
{
public function __construct(ClassMetadataFactoryInterface $classMetadataFactory = null, NameConverterInterface $nameConverter = null, PropertyAccessorInterface $propertyAccessor = null, PropertyTypeExtractorInterface $propertyTypeExtractor = null)
{
parent::__construct($classMetadataFactory, $nameConverter, $propertyAccessor, $propertyTypeExtractor);
$this->setCircularReferenceHandler(function ($object) {
return $object->getName();
});
}
}
Then declare as a service with a slithly higher priority than the default one (which is -1000):
<service
id="app.serializer.normalizer.object"
class="AppBundle\Serializer\Normalizer\AppObjectNormalizer"
public="false"
parent="serializer.normalizer.object">
<tag name="serializer.normalizer" priority="-500" />
</service>
This normalizer will be used by default everywhere in your project.

ManyToOne with FOSUSerBundle ignoring exclusion policy

Building a JSON response for an API type thing, to retrieve a specific set of data that includes a ManyToOne relationship in the entity for my entity that extends FOSUSerBundle's User entity (called Account in my case).
The problem is, the Account entity thats included as a field in the response, is wanted, but I dont want to include all of the password and role type stuff.
I've been browing the internet for a couple hours now, and I've followed many guides on this, and I've cleared my cache every single time, and to no avail; So here's where I ended up:
// app/config/config.yml
jms_serializer:
metadata:
auto_detection: true
directories:
FOSUserBundle:
namespace_prefix: "FOS\\UserBundle"
path: "%kernel.root_dir%/Resources/serializer/FOS"
I've for below I've tried User.Model.yml and Model.User.yml and User.Entity.yml as well in a vain thought that the file name actually matters
// app/Resources/serializer/FOS/Entity.User.yml
FOS\UserBundle\Model\User:
exclusion_policy: ALL
properties:
id:
expose: true
and what I get still looks like this:
{
"status":"ok",
"api_version":"1.0",
"code":200,
"data":{
"video":{
"id":1,
"published":true,
"visibility":true,
"title":"Megaman 2",
"slug":"megaman-2",
"summary":"A rap song about Megaman",
"description":"A rap song\r\nAbout megaman",
"youtube_id":"R6L9bUouDr8",
"date_published":"2014-07-02T14:09:26-0700",
"date_created":"2014-07-02T14:09:26-0700",
"date_updated":"2014-07-02T14:09:26-0700",
"author_id":3,
"author":{
"id":3,
"username":"kharrison",
"username_canonical":"kharrison",
"email":"(sorry private)",
"email_canonical":"(sorry, private)",
"enabled":true,
"salt":"(sorry, private)",
"password":"(sorry, private)",
"last_login":"2014-07-04T15:17:34-0700",
"locked":false,
"expired":false,
"roles":[
"ROLE_SUPER_ADMIN"
],
"credentials_expired":false,
"display_name":"Kyle Harrison",
"slug":"kyle-harrison",
"bio":"Test"
}
}
}
}
The "author" field, is my Account entity thats being run through the JMSSerializer
I want to exclude ALL of that, except the user ID, Display name, and slug.
And finally this is how the API works:
// My/Bundle/Controller/BaseAPIController.php
//......... other code
/**
* #param string $status
* #param integer $code
* #return Response
*/
public function render_api($status, $code)
{
$this->apiResponse->setStatus($status);
$this->apiResponse->setCode($code);
return new Response($this->apiResponse->serialize($this->get('jms_serializer')), $this->apiResponse->getCode(), ["Content-type"=>"application/json"]);
}
//............. other code
and finally, that calls this:
// My/Bundle/Models
class APIResponse {
protected $status;
protected $apiVersion;
protected $code;
protected $data;
public function __construct($apiVersion, $status = "OK", $code = 500)
{
$this->status = $status;
$this->code = $code;
$this->apiVersion = $apiVersion;
$this->data = [];
}
// ... getters and setters
/**
* #return mixed
*/
public function serialize($serializer) {
return $serializer->serialize($this, "json");
}
}
I've for below I've tried User.Model.yml and Model.User.yml and
User.Entity.yml as well in a vain thought that the file name actually
matters.
It does matter, actually. It's a concatenation of the namespace and class name. In this case, you're trying to configure the FOS\UserBundle\Model\User class, so the file name should be Model.User.yml. (FOS\UserBundle\ should be excluded from the file name, since you configured it as namespace_prefix in your config.yml)
Also make sure that your Account class doesn't re-declare (overwrite) the properties, as the serializer config only works if you configure it for the class that actually declares the properties.
Ok So, the actual answer, couldn't have been arrived to via the information I provided. But Nic's Answer did lead me towards the solution. The description of how the the serializer looks at and deciphers the config file lead me to the real problem at hand.
This is what I failed to show:
<?php
namespace [PRIVATE]\[PRIVATE]Bundle\Entity;
use Doctrine\ORM\Mapping as ORM;
use FOS\UserBundle\Model\User as BaseUser;
use JMS\Serializer\Annotation\ExclusionPolicy;
use JMS\Serializer\Annotation\Expose;
use JMS\Serializer\Annotation\Groups;
use JMS\Serializer\Annotation\VirtualProperty;
/**
* Account
*
* #ORM\Table()
* #ORM\Entity(repositoryClass="[PRIVATE]\[PRIVATE]Bundle\Entity\AccountRepository")
*/
class Account extends BaseUser
{
The problem lays with the Alias I provided the FOS\UserBundle\Model\User namespace. I no longer remember why I wrote that that way. However, the moment I remove the Alias and rewrote the extends to resemble this instead:
<?php
namespace [PRIVATE]\[PRIVATE]Bundle\Entity;
use Doctrine\ORM\Mapping as ORM;
use FOS\UserBundle\Model\User;
use JMS\Serializer\Annotation\ExclusionPolicy;
use JMS\Serializer\Annotation\Expose;
use JMS\Serializer\Annotation\Groups;
use JMS\Serializer\Annotation\VirtualProperty;
/**
* Account
*
* #ORM\Table()
* #ORM\Entity(repositoryClass="[PRIVATE]\[PRIVATE]Bundle\Entity\AccountRepository")
*/
class Account extends User
{
combined with the new correct filename from Nic's answer, the config based Exclusion policy for JMSSerializerBundle totally kicks in, and every instance of FOSUserBundle's items are now completely hidden, except for the fields I've now explicitly told it to expose.
This is exactly what I wanted :)
Thanks everyone for your help! Cheers
~k
I'm not sure it's the exact way you want it, more a way around:
way around 1: Select only the properties you want (via the entity manager) and then serialize the array obtained.
It's what I do with what I call my API (which is not a class as you but controllers)

Symfony 2.4.2 - Pass the entity manager between controllers in different bundles

I'm having a problem with passing the entity manager between two layers of controllers.
The system I'm building has the following structure:
2 Bundles:
Core Bundle (let's call it Backend Controller)
This is the bundle that contains all the Models (entities) and business rules/logic.
API Bundle (call it Frontend controller)
Is responsible for checking the permissions of passed in api keys and communicating with the Core bundle to get the info.
Here's an example with the User controllers and entities:
UserController.php in APIBundle:
<?php
namespace Acme\Bundle\APIBundle\Controller;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\Request;
use Acme\Bundle\CoreBundle\Controller\UserController as User;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
class UserController extends BaseController implements AuthenticatedController
{
public function readAction(Request $request) {
$user = new User($this->getDoctrine()->getManager());
$user->load(2);
return $this->response($user);
}
}
UserController.php in CoreBundle:
<?php
namespace Acme\Bundle\CoreBundle\Controller;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;
use Symfony\Component\HttpFoundation\Response;
use Acme\Bundle\CoreBundle\Entity\User;
class UserController extends BaseController
{
function __construct($em) {
parent::__construct($em);
$this->entity = new User();
}
/**
* Get userId
*
* #return integer
*/
public function getUserId()
{
return $this->entity->userId;
}
/**
* Set firstName
*
* #param string $firstName
* #return User
*/
public function setFirstName($firstName)
{
$this->entity->firstName = $firstName;
return $this;
}
// ...
public function load($id) {
if (!$this->entity instanceof \Acme\Bundle\CoreBundle\Entity\BaseEntity) {
throw new \Exception('invalid entity argument');
}
$this->entity = $this->em->getRepository('AcmeCoreBundle:User')->find($id);
}
}
Please, tell me if I'm doing this right. It seems strange to pass the entity manager between the controllers every time.
Maybe there's a better way of doing that?
Does the idea between the separation of the bundles make sense?
Thank you, every idea is greatly appreciated.
If CoreBundle UserController is never accessed through HTTP nor do its methods return instances of Symfony\Component\HttpFoundation\Response then it's not really a controller.
You should better define it as a service, as in CoreBundle\Service\User, and inject the EntityManager through the DI container.
sevices.yml
corebundle.userservice:
class: Acme\CoreBundle\Service\User
arguments: [#doctrine.orm.entity_manager]
It will then be available from Acme\Bundle\APIBundle\Controller\UserController with the following:
$user = $this->get('corebundle.userservice');
Of course, you can also define Acme\Bundle\APIBundle\Controller\UserController as a service on its own, then inject 'corebundle.userservice', for convenience.
I suggest you read the Symfony docs on Dependency Injection.
Search to get entity manager in Entity class is a wrong way !
In CoreBundle, you use the UserController.php same as an Entity class.
Read docs to understand how to properly use repository in symfony.
In UserController of APIBundle you must call a custom repository function. This repository is declare in your entity.