If I have a batch job that needs to be executed every day, how should I choose to use Spring Batch or ElasticJob. What is the difference between them before and can they work together?
Related
Is it possible to use spring batch as a regular job framework?
I want to create a device service (microservice) that has the responsibility
to get events and trigger jobs on devices. The devices are remote so it will take time for the job to be complete, but it is not a batch job (not periodically running or partitioning large data set).
I am wondering whether spring batch can still be used a job framework, or if it is only for batch processing. If the answer is no, what jobs framework (besides writing your own) are famous?
Job Description:
I need to execute against a specific device a job that will contain several steps. Each step will communicate with a device and wait for a device to confirm it executed the former command given to it.
I need retry, recovery and scheduling features (thought of combining spring batch with quartz)
Regarding read-process-write, I am basically getting a command request regarding a device, I do a little DB reads and then start long waiting periods that all need to pass in order for the job/task to be successful.
Also, I can choose (justify) relevant IMDG/DB. Concurrency is outside the scope (will be outside the job mechanism). An alternative that came to mind was akka actors. (job for a device will create children actors as steps)
As far as I know - not periodically running or partitioning large data set are not primary requirements for usage of Spring Batch.
Spring Batch is basically a read - process - write framework where reading & processing happens item by item and writing happens in chunks ( for chunk oriented processing ) .
So you can use Spring Batch if your job logic fits into - read - process - write paradigm and rest of the things seem secondary to me.
Also, with Spring Batch , you should also evaluate the part about Job Repository . Spring Batch needs a database ( either in memory or on disk ) to store job meta data and its not optional.
I think, you should put more explanation as why you need a Job Framework and what kind of logic you are running that you are calling it a Job so I will revise my answer accordingly.
Am invoking spring jobs based on event, however i hv couple jobs to execute on specific event which could execute in parallel, Is there any utility class which can execute multiple jobs in parallel? Thanks
We don't offer anything specific for launching multiple jobs based on a single message out of the box with Spring Batch. However, writing a message handler that can handle that scenario should be pretty trivial.
We have job which polls for file and db every M-F between 1PM-5PM using cron expression. During this time if file arrives it downloads the file and invoke a job. This is working fine and we have used spring integration and batch.
Now we need some customization where we have multiple job where job1 one should poll like above once file is processed successfully, it should stop polling.
Second requirement is, in case if file does not come during polling period we want to send some notification to ops team so that they can take some actions.
Would that help ? Exit Spring Integration when no more messages
You would be able to implement custom behavior in that advice, based on polling result and the time of the day.
Garry is also mentionning that conditional pollers are coming in next versions :
http://docs.spring.io/spring-integration/docs/4.2.0.BUILD-SNAPSHOT/reference/html/messaging-channels-section.html#conditional-pollers
I am trying to design a spring batch job which I want to process a dynamic set of files parallely. Meaning when the batch job itself is started, the number of files to process is not known - the files are available dynamically. The job should run and continue to process the files parallely as and when a new file arrives,till it has finished processing all files.
I have gone through the spring batch project page, and from my understanding it looks like Multi-threaded Step is suitable for my case. But the thing that I am not sure of is whether it can support dynamic availability of files to be processed?
Any inputs will be highly appreciated.
Thanks and regards,
Priya
You have a couple options here:
MultiResourceItemReader - This ItemReader wraps an ItemWriter like the FlatFileItemReader and loops through the resources provided via an expression.
Partitioning - This option is better for parallel processing of files. Using the MultiResourcePartitioner, you can execute files in parallel with all the restartability, etc features you'd normally get with Spring Batch.
You can read more about partitioning in the documentation here: http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html
Spring XD has a very interesting API for providing batch jobs creation/deployment/undeployment.
But, I was wondering why Spring XD is not proposing in its REST Api a GET method to access the result dataset of a job execution (and step execution) ?
Do I need to plug another Spring MVC application to display my batch jobs results ?
Thanks a lot.