I'm facing an issue with Laravel queued jobs.
I'm using Laravel v8.40.0 with Redis v6.2.5 and Horizon v5.7.14 for managing jobs.
I have a job class called MyJob which should write a message in log file.
If I use Queue::push(new MyJob()) everything seems to work fine: I see the job in Horizon and the new row in log file.
But if I use dispatch(new MyJob()) or MyJob::dispatch() it doesn't seem to push my job into queue: I can't see the job in Horizon and I see no results in log file.
I was following the docs (https://laravel.com/docs/8.x/queues#dispatching-jobs) to use queues correctly and I don't understand where I'm doing wrong.
Thank you
Related
I must be missing something obvious. I've got a couple of jobrunr jobs where i'm using the lambda enqueue format version 5.1.6. Like this:
JobId jobId = BackgroundJob.<MyService>enqueue(x -> x.doWork());
I would like to validate the plumbing and work in the jobs is executing via some integration tests with Spring, but don't see the options to run now, eager mode, etc? Thanks
You can't, I'm afraid.
You can mock the JobScheduler and capture the args. JobRunr itself is also tested very well so if you pass a job, you can rest assured it will be enqueued.
You could also put the pollIntervalInSeconds to 5 and use awaitility then to verify your job executed. There are many examples of this in the JobRunr repo.
We are using the JMSOutboundGateway to send message and receive message using the reply channel within the JMSOutboundGateway. When we run multiple iterations of the same job using the same JMSOutboundGateway, it fails with this error "Message contained wrong job instance id [85] should have been [86]" ( org.springframework.batch.integration.chunk.ChunkMessageChannelItemWriter.getNextResult() ) .
This is due to same JMSOutBoundGateway instance being using when I run the second when the first job is still in progress.
Is there a way I can run parallel execution of the same job type ?
This is a known issue, see https://github.com/spring-projects/spring-batch/issues/1372 and https://github.com/spring-projects/spring-batch/issues/1096.
The workaround is to use a separate instance of the writer for each job to prevent sharing the same reply channel.
I have multiple tasks as different django apps using a RabbitMQ broker. This was setup with standard django configuration and was working perfectly. I was using groups, chains and calling them from different modules.
As a standard practice, I had:
celery.py:
app = Celery('<proj>')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
And in project/init.py:
from __future__ import absolute_import
from .celery import app as celery_app
All tasks were inherited from celery.Task with run() overwritten.
Now I got a requirement to call a different task on a different RabbitMQ broker.
So here's what I did where I had to call the different task:
diff_app = Celery('diff')
diff_app.config_from_object({'BROKER_URL':'<DIFF_BROKER_URL>'})
Now to call:
diff_app.send_task('<task_name>', (args1,arg2,))
After I do this, when I call my previous tasks, they get routed to this new broker. The moment I comment out this code, everything is fine back again.
When I check celery_app (described above) conf, the broker url is correct. But when I check any previous task->app->conf->broker url, it is updated with new broker. How to fix this?
I removed 'autodiscover_tasks' and associated '_app' with each 'Task' class. This got me through with the issue.
I have run the complete source for Getting Started - Creating a Batch Service
Knowing that the sample uses the memory-based database provided by the #EnableBatchProcessing, is the db query result expected or it will only be available if data will be persisted permanently?
After adding some debug lines, it seems that the DB query is executed first before the job gets executed. Was this the expected behavior?
Is there anything I'm missing here.
Thanks
Alex
You aren't missing anything. This is related to issue number 8 for that guide (https://github.com/spring-guides/gs-batch-processing/issues/8). I just created a pull request to address this issue. You can view the PR here (https://github.com/spring-guides/gs-batch-processing/pull/9) until it's merged.
UPDATE
The PR has been merged and the guid has been updated. The new version should no longer have this issue.
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement.
I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job.
I have now few design options.
Similar to the below written job, I need to maintain several Jobs called by a Abstract Class may be JobScheduler .
/*Can this Code be abstracted from the application and written as
as a seperate job. Because my understanding is that the
Application code itself has to have the addJars embedded
which internally sparkContext takes care.*/
SparkConf sparkConf = new SparkConf().setAppName("MyApp").setJars(
new String[] { "/path/to/jar/submit/cluster" })
.setMaster("/url/of/master/node");
sparkConf.setSparkHome("/path/to/spark/");
sparkConf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.setLocalProperty("spark.scheduler.pool", "test");
// Application with Algorithm , transformations
extending above point have multiple versions of jobs handled by service.
Or else use an Spark Job Server to do this.
Firstly, I would like to know what is the best solution in this case, execution wise and also scaling wise.
Note : I am using a standalone cluster from spark.
kindly help.
It turns out Spark has a hidden REST API to submit a job, check status and kill.
Check out full example here: http://arturmkrtchyan.com/apache-spark-hidden-rest-api
Just use the Spark JobServer
https://github.com/spark-jobserver/spark-jobserver
There are a lot of things to consider with making a service, and the Spark JobServer has most of them covered already. If you find things that aren't good enough, it should be easy to make a request and add code to their system rather than reinventing it from scratch
Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Here is a good client that you might find helpful: https://github.com/ywilkof/spark-jobs-rest-client
Edit: this answer was given in 2015. There are options like Livy available now.
Even I had this requirement I could do it using Livy Server, as one of the contributor Josemy mentioned. Following are the steps I took, hope it helps somebody:
Download livy zip from https://livy.apache.org/download/
Follow instructions: https://livy.apache.org/get-started/
Upload the zip to a client.
Unzip the file
Check for the following two parameters if doesn't exists, create with right path
export SPARK_HOME=/opt/spark
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
Enable 8998 port on the client
Update $LIVY_HOME/conf/livy.conf with master details any other stuff needed
Note: Template are there in $LIVY_HOME/conf
Eg. livy.file.local-dir-whitelist = /home/folder-where-the-jar-will-be-kept/
Run the server
$LIVY_HOME/bin/livy-server start
Stop the server
$LIVY_HOME/bin/livy-server stop
UI: <client-ip>:8998/ui/
Submitting job:POST : http://<your client ip goes here>:8998/batches
{
"className" : "<ur class name will come here with package name>",
"file" : "your jar location",
"args" : ["arg1", "arg2", "arg3" ]
}