I have a job that is built of the following components
Processing Step - activates an external processing that logs it's result in the DB and returns an internal id so I can take this id and process it further
Logging Step - built of a tasklet, that contains a chunk
The chunk is built of an item reader - that I plan will use #{stepExecutionContext['job.id']} as part of the sql written in the xml file so it will get the relevant logging info
I'm trying to work with the solution suggested here - 11.8 Passing Data to Future Steps but i get this error when i try to add a property on the step execution context or the job execution context
chunkContext.getStepContext().getStepExecutionContext().put("job.id", jobId);
And I get this error:
java.lang.UnsupportedOperationException: null
at java.util.Collections$UnmodifiableMap.put(Collections.java:1342)
at ...
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:395)
Did i miss something?
StepContext available from ChunkContext is a read-only object; use a StepExecutionListener and save the step execution context passed as parameter in method StepExecutionListener.afterStep(StepExecution stepExecution)
I always follow this path:
chunkContext
.getStepContext()
.getStepExecution()
.getJobExecution()
.getExecutionContext()
.put("", "");
I also had problems doing the put in StepExecution.
Maybe it loses data when it goes to the next step.
However, I leave the last sentence to the most expert.
chunkContext.getStepContext().getStepExecutionContext() retrun copy use chunkContext.getStepContext().getStepExecution().getExecutionContext()
Related
I am new to Spring Batch. I found out that by use of ExecutionContextPromotionListener, I can set up key value pairs and get them in future steps.
<step id="step1">....</step>
<decision id="decision1">.... />
When I used Tasklet instead of reader, I did following:
Created bean of ExecutionContextPromotionListener with expected keys in my batch config file.
Registered the Listener to my step.
Put the key-value pairs in executionContext retrieved from chunkContext inside my Tasklet like below:
Now the Decider can read from the step execution context as follows and decide.
But, I want to take decision based on Reader from previous step. So, in my decision, how to get the value from Reader? Or is my approach wrong. Please suggest.
simple way is you can make use of Execution context from your step and can pass on the values into the next step .
So, in you first step do something like this
// ...
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
stepContext.put("DATA_KEY", dataToShare );
then in your next step you can read this using the execution context.
ExecutionContext jobContext = jobExecution.getExecutionContext();
dataToShare = jobContext.get("DATA_KEY");
you just need to manage the keys - the key that you use to put in first step and read it in next step
I'm using Spring Batch with partitioning.
Initially, I was getting DeadlockLoserDataAccessException and then I have configured our steps as faultTolerant, please see the following code -
Step masterCalculationStep = stepBuilderFactory.get("STEP_1")
.<Map<Long, List<CostCalculation>>, List<TempCostCalc>>chunk(1).reader(reader)
.processor(processor)
.writer(writer)
.faultTolerant()
.retryLimit(5)
.retry(DeadlockLoserDataAccessException.class)
.build();
but now we are getting another exception -
org.springframework.batch.core.step.skip.NonSkippableReadException:
Non-skippable exception during read
Don't know why this new exception and how to resolve it?
The RetryPolicy in a chunk oriented step is not applied to the reader. So if your reader might throw a transient exception, you need to add the retry logic around the reader yourself. This can be done for example with:
AOP by applying org.springframework.retry.interceptor.RetryOperationsInterceptor to your reader
or by using org.springframework.retry.support.RetryTemplate in a decorator of your reader that reties the read method when it throws a transient exception
Similar questions can be found here:
Spring batch retry mechanism for reader failure
Retry in case of a failed reading
Hope this helps.
I have a spring batch job wherein I need to set job parameter in beforeStep
I am using below code for the same:
#Override
public void beforeJob(JobExecution jobExecution) {
String pid = fetchPid();
jobExecution
.getJobParameters()
.getParameters()
.put("pid", new JobParameter(pid));
}
When I run above code and debug , I see that pid is not present in the jobparameters . What could be wrong here?
What could be wrong here?
The JobParameters#getParameters returns a unmodifiable map of parameters (See Javadoc). So adding the pid key as you did it wont work.
I need to set job parameter in beforeStep
I guess you mean in beforeJob and not beforeStep since your code shows the beforeJob method. Using the JobExecutionListener to add parameters is too late because the parameters are used to identify the instance, and at the time of invoking beforeJob, the execution has been already launched with the given parameters. You need to prepare the parameters upfront then use them to launch the job using jobLauncher.run(job, jobParameters).
Commit interval will commit the data at specified intervals. I want to commit the entire file at a single shot since my requirement is to validate the file (line by line) and if it fails at any point . roll back. no commit. is there any way to achieve this in spring batch?
You can either set your commit-interval to Integer.MAX_VALUE (231-1) or create your own CompletionPolicy.
Here's how you configure a step to use a custom CompletionPolicy :
<chunk reader="reader" writer="writer" chunk-completion-policy="completionPolicy"/>
<bean id="completionPolicy" class="xx.xx.xx.CompletionPolicy"/>
Then you have to either choose an out-of-the-box CompletionPolicy provided by Spring Batch (a list of implementations is available on previous link) or create your own.
What do you mean by "commit"?
You are talking about validating and not about writing the read data to another file or into database.
As mentioned in the comment by Michael Prarlow, memory problems could arise, if the size of the file changes.
In order to prevent this, I would suggest to start your job with a validation step. Simply read the data chunkwise, check the data line by line in your processor and throw a none-skippable exception, if the line is not valid. Use a passthroughwriter, so nothing is persisted. If there is a problem, the whole job will fail.
If you really have to write the data into a db or another file, you could do this in a second step. Since you have validated your data, you shouldn't observe any problems.
Simple PassThroughItemWriter
public class PassThroughItemWriter<T> implements ItemWriter<T> {
public void write(List<? extends T> items) {
// do nothing
}
}
or, if you use the Java-Api to build your job and steps, you could simply use a lambda:
stepBuilders.get("step")
.<..., ...>chunk(..)
.reader(...)
.processor(...) // your processor with the validation logic
.writer(items -> {}) // empty lambda expression
.build();
How do I "tell" EF to log queries globally? I was reading this blog post: EF logging which tells in general how to log sql queries. But I still have a few questions regarding this logger.
Where would I need to place this line context.Database.Log = s =>
logger.Log("EFApp", s);?
Can it be globally set? Or do I have to place it everywhere I do DB
operations?
In the "Failed execution" section, the blogger wrote that, and I
quote:
For commands that fail by throwing an exception, the output contains the message from the exception.
Will this be logged too if I don't use the context.Database.Log?
Whenever you want the context to start logging.
It appears to be done on the context object so it should be done every time you create a new context. You could add this line of code in your constructor though to ensure that it is always enabled.
It will not log if you do not enable the logging.
I don't recommend to use that's functionality, because, it hasn't reason to exists in the real case.
Thats it use a lot of to debug code only. But, wether you wanna know more than details ... access link... https://cmatskas.com/logging-and-tracing-with-entity-framework-6/
In this case you can put code like this
public void Mylog()
{
//Thats a delegate where you can set this property to log using
//delegate type Action, see the code below
context.Database.Log = k=>Console.Write("Any query SQL")
//Or
context.Database.Log = k=>Test("Any query SQL")
}
public void Test(string x){
Console.Write(x)
}
I hope thats useufull