We have implemented the Remote Chunking and it is working perfectly. But we want to process message in multi thread at worker, please help is there any way to configure multi thread in worker.
Related
I am currently working on Spring Boot and Spring Batch application to read 200,000 records from Database, process it and generate XML output.
I wrote single threaded Spring Batch program which uses JDBCPagingItemReader to read batch of 10K records from Database and StaxEventItemReader to generate this output. Total process is taking 30 minutes. I am wanting to enhance this program by using Spring Batch local Partitioning. Could anyone share Java configuration code to do this task of Spring Batch partitioning which will split processing into multi thread + multi files.. I tried to multi thread java configuration but StaxEventItemReader is single thread so it didn't work. Only way I see is Partition.
Appreciate help.
You are correct that partitioning is the way to approach this problem. I don't have an example of JDBC to XML of how to configure a partitioned batch job, but I do have one that is CSV to JDBC in which you should be able to just replace the ItemReader and ItemWriter with the ones you need (JdbcPagingItemReader and StaxEventItemWriter respectively). This example actually uses Spring Cloud Task to launch the workers as remote processes, but if you replace the partitionHandler with the TaskExecutorPartitionHandler (instead of the DeployerPartitionHandler as configured), that would execute the partitions internally as threads.
https://github.com/mminella/S3JDBC
Need to design multi threading with Spring batch. Spring batch partition or using java multi threading, Which one is a better choice? We have many processes, each process holds jobs and sub jobs. these sub jobs needs to be executed in parallel.How can I do the retry mechanism in partition??
Go for the partition with master-slave concept. I have tried this and it boots the performance in good amount.
Restart Scenario :
Once your partitioner starts and your items are divided to the slaves.
Lets say you have 3 slaves and each slave holds 1 file to process.
Manually delete some items in the file which is assigned to the Slave2 so that it should get failed(Either in reader or writer of your slave step).
Then restart the job. Now it should start reading from the file which was assigned to the Slave2.
We have partitioned a large number of our jobs to improve the overall performance of our application. We are now investigating running several of these partitioned jobs in parallel (kicked off by an external scheduler). The jobs are all configured to use the same fixes reply queue. As a test, I generated a batch job that has a parallel flow where partitioned jobs are executed in parallel. On a single server (local testing) it works fine. When I try on a multiple server cluster I see the remote steps complete but the parent step does not ever finish. I see the messages in the reply queue but they are never read.
Is this an expected problem with out approach or can you suggest how we can try to resolve the problem?
With regards to this post, I am using HornetQ 2.4.0 embedded, and using the hornetq core api. I use it to queue messages received via a web service call. I have another thread that dequeues messages synchronously, and processes them. When enqueuing, I sometimes get the "WARN: HQ212051: Invalid concurrent session usage" warning. Does this apply to this embedded usage that I am doing, where I am not using the JMS api? I'm running in embedded jetty, and don't have a container. If I do need to guarantee single thread access, I would have to do my own pool or do a thread local session, correct? (would rather not synch access to the shared objects for performance reasons)
I am using celery 3.0.13 and using mongodb as the broker.
I am trying to purge the tasks in a custom queue. I stopped all the workers and I tried to purge waiting tasks in the custom queue using the "celery purge" command but the command reports that no tasks were purged. I rechecked that the tasks are still in the queue even after running the commands (using flower).
Am I missing anything?
Thanks.
It may be related to this issue:
https://github.com/celery/celery/issues/1047