Spring Batch repeat step until a criteria is met - spring-batch

I have this Spring Batch flow:
return jobBuilderFactory.get(JOB).preventRestart().incrementer(new RunIdIncrementer()).listener(jobCompletionListener)
.start(clean).next(generateFiles1).next(callApi1)
.next(clean).next(generateFiles2).next(callApi2)
.next(clean).next(generateFiles3).next(callApi3)
.next(clean).next(generateFiles4).next(callApi4)
.build();
I must repeat the first three steps (clean, generateFiles1 and callApi1) until a certain criteria is met (I have to count some data in the database to check if I need to call the API again). And so on for the next three steps.
I have seen the on and to functions explained there, but it seems to me that it does not allow to write such loops.
I could define such flows:
final FlowBuilder<Flow> flowBuilderStep1 = new FlowBuilder<>("Step1");
flowBuilderStep1.start(clean).next(generateFiles1).next(callApi1).end();
final Flow step1 = flowBuilderStep1.build();
final FlowBuilder<Flow> flowBuilderStep2 = new FlowBuilder<>("Step2");
flowBuilderStep2.start(clean).next(generateFiles2).next(callApi2).end();
final Flow step2 = flowBuilderStep2.build();
And then build the conditional structure (maybe after adding Decider or afterStep() some place):
return jobBuilderFactory.get(JOB).preventRestart().incrementer(new RunIdIncrementer()).listener(jobCompletionListener)
.start(step1).on("RETRY").to(step1).on("CONTINUE")
.to(step2).on("RETRY").to(step2).on("CONTINUE")
.to(step3).on("RETRY").to(step3).on("CONTINUE")
.to(step4)
.end().build();
But I don't think it would loop properly. Am I right? Can a loop be accomplished (without a xml config)?

I had to do something like that to make it work.
return jobBuilderFactory.get(JOB).preventRestart().incrementer(new RunIdIncrementer()).listener(jobCompletionListener)
.start(step1).next(step1Decider).on(RETRY).to(step1).from(step1Decider).on(CONTINUE)
.to(step2).next(step2Decider).on(RETRY).to(step2).from(step2Decider).on(CONTINUE)
...

Related

Spring Webflux - how to retrieve value from Mono/Flux multiple times without making multiple calls to get those Mono/Flux

I'm using Spring Webflux & reactor, Java 11, Spring boot 2.4.5, Spring 5.3.6 versions for this reactive application.
Use case:
I need to make a call to API and get data from it. From this data I take uniqueId and then call bunch of API's to get other data, and then finally combine all this data to new Object and return.
Example code:
Mono<Response> 1stAPIResponse = 1stAPI.callMethod(eventId); // response object has ProductId, and other details.
Mono<List<String>> productIds = 1stAPIResponse.map(Response::ProductId).collect(Collectors.toList());
Mono<2ndAPIResponse> 2ndAPIResponse = productIds.flatMap(ids -> 2ndAPI.callMethod(ids));
Mono<3rdAPIResponse> 3rdAPIResponse = productIds.flatMap(ids -> 3rdAPI.callMethod(ids));
...
1stAPIResponse.foreach(response -> {
FinalResponse.builder()
.productId(response.productId)
.val1(2ndAPIResponse.get(response.productId))
.val3(3ndAPIResponse.get(response.productId))
. ...
.build()});
Here the problem is, when ids are passed to 2ndAPI, 3rdAPI,... method, it makes call to 1stAPI and get the data each time. And finally when creating object it makes another call to 1st API. In this example it makes total of 3 calls.
How can I avoid similar multiple calls from occurring?
One way to avoid this is, I can make 1stAPI call blocking but is it correct? doesn't it defeat non-blocking style of coding?
Ex: Response 1stAPIResponse = 1stAPI.callMethod(eventId).toFuture().get();
How can I write a correct reactive program (without blocking) but still make only one call to 1stAPI?
Let me know for any questions.
So, you need to refactor your code in more reactive style and use zip operator for parallel calls:
1stAPI.callMethod(eventId)
.flatmap(response -> // collect your id to list (ids);
return 2ndAPI.callMethod(ids).zipWith(3ndAPI.callMethod(ids))
.flatmap(tuple2 -> FinalResponse.builder() // tuple contains result of 2ndAPI and 3ndAPI
.productId(response.productId)
.val1(2ndAPIResponse.get(response.productId))
.val3(3ndAPIResponse.get(response.productId)))
...
)

Spring Batch: How do I use decider based on my Reader result?

I am new to Spring Batch. I found out that by use of ExecutionContextPromotionListener, I can set up key value pairs and get them in future steps.
<step id="step1">....</step>
<decision id="decision1">.... />
When I used Tasklet instead of reader, I did following:
Created bean of ExecutionContextPromotionListener with expected keys in my batch config file.
Registered the Listener to my step.
Put the key-value pairs in executionContext retrieved from chunkContext inside my Tasklet like below:
Now the Decider can read from the step execution context as follows and decide.
But, I want to take decision based on Reader from previous step. So, in my decision, how to get the value from Reader? Or is my approach wrong. Please suggest.
simple way is you can make use of Execution context from your step and can pass on the values into the next step .
So, in you first step do something like this
// ...
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
stepContext.put("DATA_KEY", dataToShare );
then in your next step you can read this using the execution context.
ExecutionContext jobContext = jobExecution.getExecutionContext();
dataToShare = jobContext.get("DATA_KEY");
you just need to manage the keys - the key that you use to put in first step and read it in next step

How to group steps in one FlowStep and re-run all of the steps after failed?

I am trying to use Spring Batch to handle a bunch of precheck, validation and actions together.
If the precheck or validation fails, then I want to re-run all the pre-checks and validations regardless they previously succeeded or failed; but if an action fails, I want to start from the failed action only.
I want to group the validation or precheck in a step such as FlowStep, but it seems that the Spring retry does not support this.
Does anybody know a solution? thanks in advance!
You can group two steps in a FlowStep as follows:
#Bean
public Flow preCheckAndValidationFlow() {
return new FlowBuilder<SimpleFlow>("flow")
.start(preCheckStep())
.next(validationStep())
.build();
}
#Bean
public Step preCheckAndValidationStep(JobRepository jobRepository) {
FlowStep flowStep = new FlowStep();
flowStep.setFlow(preCheckAndValidationFlow());
flowStep.setJobRepository(jobRepository);
return flowStep;
}
If you want the preCheckStep to re-run (even if it was successful the first time) when the validation step fails, then you need to set the allowStartIfComplete flag on the preCheckStep.
IMO, it would be much more simpler for you to create a single step that contains both the prechecks and validation logic.

Any way to ensure frisby.js test API calls go in sequential order?

I'm trying a simple sequence of tests on an API:
Create a user resource with a POST
Request the user resource with a GET
Delete the user resource with a DELETE
I've a single frisby test spec file mytest_spec.js. I've broken the test into 3 discrete steps, each with their own toss() like:
f1 = frisby.create("Create");
f1.post(post_url, {user_id: 1});
f1.expectStatus(201);
f1.toss();
// stuff...
f2 = frisby.create("Get");
f2.get(get_url);
f2.expectStatus(200);
f2.toss();
//Stuff...
f3 = frisby.create("delete");
f3.get(delete_url);
f3.expectStatus(200);
f3.toss();
Pretty basic stuff, right. However, there is no guarantee they'll execute in order as far as I can tell as they're asynchronous, so I might get a 404 on test 2 or 3 if the user doesn't exist by the time they run.
Does anyone know the correct way to create sequential tests in Frisby?
As you correctly pointed out, Frisby.js is asynchronous. There are several approaches to force it to run more synchronously. The easiest but not the cleanest one is to use .after(() -> ... you can find more about after() in Fisby.js docs.

How do I put a specific order to Jbehave story execution?

When you submit sorties to Jbehave with
#Override
public InjectableStepsFactory stepsFactory()
{
return new InstanceStepsFactory(configuration(),
new LoginSteps(), new PreferencesSteps(), new BetterSteps());
}
They are executed in BetterSteps, LoginSteps, PreferencesStpes fasion.
How do I make these classes having scenarios execute in a custom order which is not alphabetical?
Say LoginSteps followed by PreferenceSteps followed by BetterSteps etc?
There is a work around for your problem..
You can make your story name starting from Sn where n is 1,2,3..... like S1_LoginSteps, S2_PreferenceSteps
Jbehave will execute stories alphabetical starting from S1 then S2....
Do I understand you correctly that you have steps with identical or similar patterns in all of these steps classes and want to control which of them are used over others?
If so, have a look at step prioritization here: http://jbehave.org/reference/stable/prioritising-steps.html and apply priorities to your preferred steps.