How do I support passing a unique job name via environment variable to a Spring Batch Job? - spring-batch

I'm just getting started with using Spring Batch. I'm working on a command line launcher using AWS Batch with a Docker image. Trying to just sort the job instance naming.
Would it be acceptable to use a #Value for the below literal string in the jobBuilder? Essentially I'm passing in S3 File keys which will be unique values already as I have a task that's grabbing the file before my FlatFileReader runs. The goal being to facilitate retries against the job when required due to a failure.
#Bean
public Job jobParametersJob() {
return jobBuilderFactory.get("PassedInValue")
.start(step1())
.next(step2())
.next(step3())
.build();
}

I ended up solving by using a job incrementer that implements JobParametersIncrementer. Note in my case, I'm not currently passing in any job parameters so this is setting them here as my parameters are currently just passed in via environment variables to the docker container.
public class JobIncrementer implements JobParametersIncrementer {
#Value("${s3filekey}")
String s3filekey;
#Override
public JobParameters getNext(JobParameters jobParameters) {
return new JobParametersBuilder().addString("s3filekey",s3filekey).toJobParameters();
}...//then in job configuration...
#Bean
public Job jobParametersJob() {
return jobBuilderFactory.get("jobParametersJob")
.incrementer(jobIncrementer)
.start(step1())
.next(step2())
.next(step3())
.build();
}

Related

spring Batch process is running automatically even with the spring.batch.job.enabled=false flag

My Goal : Prevent batch Job from lunching while project startup.
I want to run a batch job from spring boot project once in every one hour using #scheduled annotation. However the job is starting as soon as I run the project....
spring.batch.job.enabled=false
#Scheduled( cron = "* * */1 * * *")
public void jobScheduled(){
logger.info("Job triggered");
getAllFileNames().stream().forEach( fileName-> {
jobRunner.runFlatFileBatchJob(fileName);
});
}
I have declared my class like below
#Configuration
#EnableScheduling
public class JobScheduler {
#Autowired
public JobScheduler(JobRunner jobRunner){
logger.info("job scheduler created... ");
this.jobRunner = jobRunner;
}
The Job is starting as soon as Run the application. I want it to wait till the project loads completely and other integration objects prepare themselves.
Thanks in advance.
Santrupta Dash

spring batch database connection failure is not returning proper exit status

I have gone through few posts related to the similar issue, i am calling spring batch application through shell script and getting the exit status.Everything works fine on successful execution. ExitStatus gets populated as 0. However if there is any database error(to create database error i gave the wrong port of database) then ExitStatus is being returned as empty. Code is below
I have referred below posts and implemented similarly
Make a spring-batch job exit with non-zero code if an exception is thrown
Spring batch return custom process exit code
Shell Script:
java -jar $JOBDIR/lib/feed*.jar
result=$?
echo $result
Java:
public static void main(String[] args) {
ConfigurableApplicationContext context
=SpringApplication.run(App.class, args);
int exitCode = SpringApplication.exit(context);
System.out.print("Exit code is" + exitCode);
System.exit(exitCode);
}
#Primary
#Bean(destroyMethod = "")
public DataSource dataSource() throws Exception {
return BatchDataSource.create(url, user, password);
}
in case of database error it is not even reaching end of the main method System.exit(exitCode); Can any one guide me what is wrong??
if there is any database error(to create database error i gave the wrong port of database) then ExitStatus is being returned as empty.
That's because in that case, your job is not executed at all. According to your configuration, the dataSource bean creation error prevent the Spring application context from being correctly started and your job is not even executed.

MultiResourceItemReader: Resources dont exist at bean initialization

Scenario:
Spring Batch job with 2 Steps:
Tasklet which downloads csv files (file names unknown before
runtime) to a directory.
Chunk based step with a Reader which needs to read all csv files
Challenge:
Since the file names are unknown, we use PathMatchingResourcePatternResolver.getResources() to get the resources.
The returned resources are always of length 0 since there are no files in the directory at bean initialization.
#Bean
Resource[] resources() throws IOException {
final PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
final Resource[] resources = resolver.getResources("file:" + destinationDir + "/*.csv");
return resources;
}
Any ideas? Thanks in advance!
You can save the names of the files inside the JobExecutionContext while in the tasklet step, and then use those names to initialize your resources in the chunk step.
More details at the Spring Batch Docs: Configuring a Step.
This late binding in Spring Batch is possible due to Step Scope Beans. You can read more about it here.

How to redirect spring cloud task logs into spring cloud task sink application?

I am having a Spring cloud Task sink application, that will trigger the Spring cloud Task.
#SpringBootApplication
#EnableBinding(Sink.class)
#RestController
#EnableScheduling
#EnableTaskLauncher
#Slf4j
public class FileTaskLauncherApp {
#Autowired
private Sink sink;
#Value("${spring.task.artifactory.url}")
private String uri;
#Value("${spring.task.name:file_task_launcher}")
private String taskName;
#GetMapping("/triggerTask")
public String publishTask(){
log.info("Publishing task with task launcher request...");
Map<String, String> prop = new HashMap<>();
prop.put("server.port", "0");
Map<String,String> deployProp=new HashMap<>();
deployProp.put("deployer.*.local.inheritLogging","true");
TaskLaunchRequest request = new TaskLaunchRequest(
uri, null,
prop,
deploymentProp, taskName);
GenericMessage<TaskLaunchRequest> message = new
GenericMessage<TaskLaunchRequest>(
request);
this.sink.input().send(message);
return "SUCCESS";
}
}
But Spring cloud task sink will be calling Spring Cloud Task and each task is a short lived micro service having its own functionality. I wanted to redirect application logs from Spring cloud task into Task sink application.
This is my application.properties:
server.port=8084
spring.cloud.stream.kafka.binder.brokers= localhost:2181
spring.cloud.stream.bindings.input.destination=fileTask
spring.task.artifactory.url=maven://com.tgt.fulfillment:file-generation-task:1.0.1-SNAPSHOT
spring.task.name=file_task_launcher
deployer.*.local.inheritLogging=true
This is my logs coming from task sink application
12:40:39.057 [http-nio-8084-exec-1] INFO o.s.c.task.launcher.TaskLauncherSink - Launching Task for the following uri maven://com.test:file-generation-task:1.0.1-SNAPSHOT
12:40:39.140 [http-nio-8084-exec-1] INFO o.s.c.d.spi.local.LocalTaskLauncher - Command to be executed: /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/bin/java -jar /Users/z003c1v/.m2/repository/com/test/file-generation-task/1.0.1-SNAPSHOT/file-generation-task-1.0.1-SNAPSHOT.jar
12:40:39.153 [http-nio-8084-exec-1] INFO o.s.c.d.spi.local.LocalTaskLauncher - launching task file_task_launcher-2c630ad9-acbb-43e0-8140-3ce49506f8e2
Logs will be in /var/folders/y5/hr2vrk411wdg_3xl3_10r295rp30bg/T/file_task_launcher7177051446839079310/1539587439103/file_task_launcher-2c630ad9-acbb-43e0-8140-3ce49506f8e2
As per the below spring documentation by enabling deployer.*.local.inheritLogging=true in deployment properties, application logs should be redirected to server logs, but this is not happening.
Reference: http://docs.spring.io/spring-cloud-dataflow/docs/1.4.0.RELEASE/reference/htmlsingle/#_logging
Could somebody please help me in resolving this issue at the earliest.
Can you share your stream definition that consists of the task launcher sink?
The inheritLogging property is a local deployer property and hence it should be specified when deploying the stream not at the app level property you mentioned above.
Something like:
stream deploy --name mystream --properties "deployer.*.local.inheritLogging=true”

Extbase: scheduler cannot load a class

I have a scheduler (extension "scheduler" 6.2.0) task class:
namespace MyVendor\MyExt\Scheduler;
class MultiStepProcessTask extends \TYPO3\CMS\Scheduler\Task\AbstractTask {
public function execute() {
$objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Extbase\\Object\\ObjectManager');
$businessLogic = $objectManager->get(\MyVendor\MyExt\Scheduler\MultiStepProcessTaskBusinessLogic::class);
return $businessLogic->run();
}
}
And a class implementing business logic for the scheduler task:
namespace MyVendor\MyExt\Scheduler;
class MultiStepProcessTaskBusinessLogic {
public function run() {
...
}
}
I created a scheduler task in BE. If I run it manually - it's OK. If it is started automatically - following error message comes:
[scheduler]: Task failed to execute successfully. Class:
MyVendor\MyExt\Scheduler\MultiStepProcessTask, UID: 8. Could not
analyse class:
MyVendor\MyExt\Scheduler\MultiStepProcessTaskBusinessLogic maybe not
loaded or no autoloader? (msg#4.0.255)
The best is - after being once run manually, the task runs automatically without errors until the cache is cleared. After that it needs one manual run again.
One more strange thing: in the main scheduler task class (MultiStepProcessTask) no injection is possible. That's why I had to create business logic object via objectManager
The best solution would be to write a command controller based on extbase. There you can use the ObjectManager and you can run it directly from CLI and of course also call it via scheduler!