I want to get the last processed item in ItemWriter and write one of the string value of the item in execution context , I am using chunks writes. Since it is chunk write , so write method would be called multiple times , how do I go about this?
Here is my writer:
#Component
public class MyItemWriter implements ItemWriter<MyDbEntity>{
private JobExecution jobExecution;
#BeforeStep
public void initWriter(StepExecution stepExecution) {
jobExecution = stepExecution.getJobExecution();
}
#Override
public void write(List<? extends MyDbEntity> items) throws Exception {
}
I want to get the last processed item in ItemWriter and write one of the string value of the item in execution context
Let's say your MyDbEntity provides a getter getData for the data you want to write in the execution context. A simple way to do it is to write the data with same key for each item. This will erase the data of the previous item and when the step is finished, the data of the last item will be the one present in the execution context. Something like:
#Override
public void write(List<? extends MyDbEntity> items) throws Exception {
for(MyDbEntity item: items) {
// write item where needed
jobExecution.getExecutionContext().put("data", item.getData());
}
}
The job execution will be persisted after the step is completed (successfully or not) and you can get access to the data of the last item from the execution context.
Hope this helps.
Related
I’m writing an application in Spring Batch to do this:
Read the content of a folder, file by file.
Rename the files and move them to several folders.
Send two emails: one with successed name files processed and one with name files that throwed errors.
I’ve already get 1. and 2. but I need to make the 3 point. ¿How can I store the file names that have send to the writer method in an elegant way with Spring Batch?
You can use a Execution Context to store the values of file names which gets processed and also which fails with errors.
We shall have a List/ similar datastructure which has the file names after the business logic. Below is a small snippet for reference which implements StepExecutionListener,
public class FileProcessor implements ItemWriter<TestData>, StepExecutionListener {
private List<String> success = new ArrayList<>();
private List<String> failed = new ArrayList<>();
#Override
public void beforeStep(StepExecution stepExecution) {
}
#Override
public void write(List<? extends BatchTenantBackupData> items) throws Exception {
// Business logic which adds the success and failure file names to the list
after processing
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
stepExecution.getJobExecution().getExecutionContext()
.put("fileProcessedSuccessfully", success);
stepExecution.getJobExecution().getExecutionContext()
.put("fileProcessedFailure", failed);
return ExitStatus.COMPLETED;
}
}
Now we have stored the file names in the execution context which we will be able to use it in send email step.
public class sendReport implements Tasklet, StepExecutionListener {
private List<String> success = new ArrayList<>();
private List<String> failed = new ArrayList<>();
#Override
public void beforeStep(StepExecution stepExecution) {
try {
// Fetch the list of file names which we have stored in the context from previous step
success = (List<String>) stepExecution.getJobExecution().getExecutionContext()
.get("fileProcessedSuccessfully");
failed = (List<BatchJobReportContent>) stepExecution.getJobExecution()
.getExecutionContext().get("fileProcessedFailure");
} catch (Exception e) {
}
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// Business logic to send email with the file names
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
logger.debug("Email Trigger step completed successfully!");
return ExitStatus.COMPLETED;
}
}
I have spring webflux stream consumer which calls a REST endpoint and consumes the messages received and save to an RDBMS. i am trying to find a way to batch it. I see the subscribe() has an overloaded method which gets called on Completion. I am trying to find how to get hold of the data when this completion consumer gets called since i am calling a CompletionConsumer which is of type Runnable and all i am having is the run() method which dont take any parameters.
**CLIENT**
WebClient.create("http://localhost:8080")
.get()
.uri("/objects")
.accept(MediaType.TEXT_EVENT_STREAM)
.exchange()
.flatMapMany(clientResponse ->clientResponse.bodyToFlux(MyObject.class))
.subscribe(null,null,completionProcessorSubscriber);
**COMPLETION SUBSCRIBER**
#Service
public class CompletionProcessorSubscriber implements Runnable{
#Autowired
LegacyDAOImpl dao;
Logger logger = LoggerFactory.getLogger(CompletionProcessorSubscriber.class);
public void run() {
logger.info("\ninside RUNNNNNNNNN\n\n");
// here how to get hold of the data stream ?
}
Below is the Documentation from the Flux API
*/
public final Disposable subscribe(
#Nullable Consumer<? super T> consumer,
#Nullable Consumer<? super Throwable> errorConsumer,
#Nullable Runnable completeConsumer) {
return subscribe(consumer, errorConsumer, completeConsumer, null);
}
You should avoid adding to much logic to subscriber methods. Instead, you should utilize the rich set of operators provided by Flux API.
In this case the operators you need are buffer to collect batches and concatMap to execute batches sequentially.
In the following example I assume the LegacyDAOImpl is a blocking service whose work should be assigned to an appropriate thread pool.
public static void main(String[] args) throws InterruptedException
{
webClient.get()
.uri("/objects")
.accept(MediaType.TEXT_EVENT_STREAM)
.exchange()
.flatMapMany(clientResponse -> clientResponse.bodyToFlux(MyObject.class))
.buffer(100) // batch size
.concatMap(batchOfMyObjects -> Mono.fromRunnable(() -> legacyDAOImpl.saveAll(batchOfMyObjects))
.subscribeOn(Schedulers.elastic())) // blocking IO goes to elastic thread pool
.subscribe();
}
private static class LegacyDAOImpl
{
public void saveAll(List<MyObject> myObjects)
{
// save here
}
}
private static class MyObject
{
}
I have requirement in my project what ever exception occur in ItemProccesor need to store Exception in JobExecution context and at the end of JobExecution send mail for Exceptional records but how to get JobExecution Object in processListner?
I tried using #beforestep in processListner but JobExecution object was null is there any way to get JobExecution context in process Listner
I got solution in spring batch for above issue, need to specify jobscope in process listener and access job execution context in listner class code is mention below.
#Bean
#JobScope
public CaliberatedProcessorListener calibratedProcessorListener() {
return new CaliberatedProcessorListener();
}
public class CaliberatedProcessorListener <T, S> implements ItemProcessListener<T, S> {
#Value("#{jobExecution}")
public JobExecution jobExecution;
#Override
public void beforeProcess(T calibratedProessorInPut) {
// // do nothing
}
#Override
public void afterProcess(T calibratedProessorInput, S calibratedProessorOutPut) {
// do nothing
}
#Override
public void onProcessError(T item, Exception calibratedProcessorEx) {
FtpEmailData ftpEmailData = (FtpEmailData) jobExecution.getExecutionContext().get("calDeviceBatchInfo");
ftpEmailData.getExceptionList().add(new CalibratedDeviceException(calibratedProcessorEx.getMessage()));
}
}
I need to prepare two set of List and write them into FlatFile. The first set will be only simple retrieving from SQL and before write into FlatFile will do some string formatting. Another set of data slightly complex, first I need to get data from some table and insert into a temp table. The data will grab from this temp table and similarly need to perform some string formatting and also updating the temp file. Finally, both set data write into FlatFile.
Come into Spring Batch, I will have 3 steps.
First Step
First Reader read from DB
First Processor string formatting
First Writer write into file
Second Step
BeforeRead Retrieve and Insert to Temp table
Second Reader read from temp table
Second Processor string formatting and update temp table status
Second Writer write into file
Third Step
MUltiResourceItemReader read two files
Write into Final File
Tasklet
Delete both file and purge the temp table.
My question now is for first and second step if I don't write into file, possible to pass the data into third step?
Taking in account what Hansjoerg Wingeier said, below are custom implementations of ListItemWriter and ListItemReader which lets you define a name property. This property is used as a key to store the list in the JobExecutionContext.
The reader :
public class CustomListItemReader<T> implements ItemReader<T>, StepExecutionListener {
private String name;
private List<T> list;
#Override
public T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
if (list != null && !list.isEmpty()) {
return list.remove(0);
}
return null;
}
#Override
public void beforeStep(StepExecution stepExecution) {
list = (List<T>) stepExecution.getJobExecution().getExecutionContext().get(name);
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
public void setName(String name) {
this.name = name;
}
}
The writer :
public class CustomListItemWriter<T> implements ItemWriter<T>, StepExecutionListener {
private String name;
private List<T> list = new ArrayList<T>();
#Override
public void write(List<? extends T> items) throws Exception {
for (T item : items) {
list.add(item);
}
}
#Override
public void beforeStep(StepExecution stepExecution) {}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
stepExecution.getJobExecution().getExecutionContext().put(name, list);
return null;
}
public void setName(String name) {
this.name = name;
}
}
Normally, you don't want to do that.
If you just have a couple of hundred entries, it would work. You could, for instance, write a special class, that implements the reader and writer interface. When writing, just store the data in a list, when reading, read the entries from the list. Just instantiate it as a bean and use it in both steps (1 and 2) as your writer. by simply make the write method synchronized, it would even work when step 1 and 2 are executed in parallel.
But the problem is, that this solution doesn't scale with the amount of your input data. the more data you read, the more memory you need.
This is one of the key concepts of batch-processing: having a constant memory usage regardless of the amount of data that has to be processed.
Question is : How to make an Item reader in spring batch to deliver a list instead of a single object.
I have searched across, some answers are to modify the item reader to return list of objects and changing item processor to accept a list as input.
How to do/code the item reader ?
take a look at the official spring batch documentation for itemReader
public interface ItemReader<T> {
T read() throws Exception, UnexpectedInputException, ParseException;
}
// so it is as easy as
public class ReturnsListReader implements ItemReader<List<?>> {
public List<?> read() throws Exception {
// ... reader logic
}
}
the processor works the same
public class FooProcessor implements ItemProcessor<List<?>, List<?>> {
#Override
public List<?> process(List<?> item) throws Exception {
// ... logic
}
}
instead of returning a list, the processor can return anything e.g. a String
public class FooProcessor implements ItemProcessor<List<?>, String> {
#Override
public String process(List<?> item) throws Exception {
// ... logic
}
}