Im my chunk processing ,I have read one value from file and in my processor im passing this value to the DB , that will return 4 records for that single value. And im returning 4 records to the writer which is going to write in the DB . I'm failing the job in the 3rd record which is returned for the value read from the file. But after failing the job, 3 records from the DB is not rollbacked?
How the chunk is maintaining the transaction whether it is based on read count and write count of the record or not?
Related
I have a folder with .CSV files for each User.
I have read the records from these files and make HTTP request for each record.
Because of traffic issue at downstream application - i cannot make random calls.
We have limitation like down stream application can process max 5 records of type File-1, 3 records of type File 2 and 1 record of type File 3.
So, I have select 5 records from File 1, 3 records from File 2, 1 record from File 3 and process them (send http request in async way in item processor).
How can i do this via Spring batch? For reading multiple files can use MultiSourceReader but am more worried about the logic of selecting N number of records.
Thanks in advance.
I have a list of records to process via a spring batch job. Each record has millions of data points, I want to process each record one after another otherwise database will not handle the load.
Data will be like this:
artworkList will contain 10 records and each artwork record will containt 30 million of data.
I am using spring batch with quartz schedular.
I have a use case where for every input record ( returned by Itemreader ) the processor will produce millions of rows( huge dataset ) which needs to be persisted in a database table
I tried to handle all the inserts in ItemWriter but before even it reaches ItemWriter for the step, i am getting out of memory. I have only one step in my Job
How to handle the persistence of large dataset as output in the spring batch step ?
Note: local chunking is something i could not use here as the input with just one record for which it is failing
I have a flow of Reader -> Processor -> writer
Every 50 Million records the writer is writing the data into a file and zip it.
the problem is that once the Reader has finished the Writer still "holds" many records which are not written since it didn't reach the 50 M records threshold.
Any advise on how to implement it in a way that the data will be written to many files with 50 M records each and a single file with the renaming records ?
If you use a MulitResoureceItemWriter, you can use chunk size to dictate how this should work. It can be configured to write at your specific threshold and if there is a remainder in the final chunk, that will also be written out. You can read more about this useful delegate in the documentation here: https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/item/file/MultiResourceItemWriter.html
Say I have 1000 data which is being read by reader and my chunk size is 50.In my processor after doing some business logic , deleting data from some table by calling java stored procedure and in case of deletion fail , incrementing failed-deletion count. At last when control comes to writer if failed-deletion count> 0 then I want to rollback the entire deletion for this particular chunk and this should not affect the other chunk process. Can someone please help on this? for more information please let me know.