I went through the Introducing Spring Cloud Task, but things are not clear for the following questions.
I'm using Spring Batch
What's the use of Spring Cloud Task when we already have the metadata provided by Spring Batch ?
We're planning to use Spring Cloud Data Flow to monitor the Spring Batch. All the batch jobs can be imported into the SCDF as task and can be scheduled there, but don't see support for MongoDB. Hope MySQL works well.
What is the difference between Spring Cloud Task and Spring Batch?
Spring Cloud Task has a broader scope than Spring Batch. It is designed for any short lived task, including but not limited to (Spring) Batch jobs. A short lived task could be a Java process, a shell script, a Docker container, etc. Spring Cloud Task has its own meta-data tables to track the progress/status/stats of tasks.
In the context of Spring Batch, Spring Cloud Task provides a number of additional features:
Batch informational messages: ability to emit messages based on Spring Batch listeners events. Those messages can be consumed by streaming apps and make it possible to bridge tasks and streaming apps.
DeployerPartitionHandler: an additional partition handler that is suitable to cloud environments to dynamically deploy workers in a remote partitioning setup.
Related
I am creating a spring boot application using Temporal as a workflow engine and kafka topics .
So in order to validate whole setup is working fine, I am writing integration test cases using spring boot test, embedded kafka, and temporal. But I am not getting any option or way to run in memory workflow engine.
maybe you can try this distribution of Temporal that runs as a single process
https://github.com/DataDog/temporalite
I have a spring batch application with the master-slave pattern.
Now, this batch job runs once at a time for one client/customer.
I want to re-design the spring batch job with the master-slave pattern in such a way that it supports parallel jobs for the respective clients/customers.
Please provide me a path forward.
Using Spring for Apache Kafka, or Spring AMQP, I can achieve message pub/sub. Spring Cloud Bus uses kafka/rabbitmq to do the approximately same things, what's the differencce between them?
Spring Cloud Bus is an abstraction built on top of Spring Cloud Stream (and hence kafka and rabbitmq). It is not general purpose, but is built for sending administrative commands to multiple nodes of a service at once. For example, sending a refresh (from spring cloud commons) to all nodes of the user service. There is only one channel, where in spring cloud stream there are many. Think of it as distributed spring boot actuator.
I am currently working on Spring Boot and Spring Batch application to read 200,000 records from Database, process it and generate XML output.
I wrote single threaded Spring Batch program which uses JDBCPagingItemReader to read batch of 10K records from Database and StaxEventItemReader to generate this output. Total process is taking 30 minutes. I am wanting to enhance this program by using Spring Batch local Partitioning. Could anyone share Java configuration code to do this task of Spring Batch partitioning which will split processing into multi thread + multi files.. I tried to multi thread java configuration but StaxEventItemReader is single thread so it didn't work. Only way I see is Partition.
Appreciate help.
You are correct that partitioning is the way to approach this problem. I don't have an example of JDBC to XML of how to configure a partitioned batch job, but I do have one that is CSV to JDBC in which you should be able to just replace the ItemReader and ItemWriter with the ones you need (JdbcPagingItemReader and StaxEventItemWriter respectively). This example actually uses Spring Cloud Task to launch the workers as remote processes, but if you replace the partitionHandler with the TaskExecutorPartitionHandler (instead of the DeployerPartitionHandler as configured), that would execute the partitions internally as threads.
https://github.com/mminella/S3JDBC
I would like to use Hypersonic in memory DB to persist jobs as I need to run the same job multiple times on separate threads. The SimpleJobRepository can not be used as I run into Optimistic locking issue.
Does any one has a sample java configuration file and how to wire the hypersonic for spring batch job in java with annotation?