I am using Spring Batch and my org not willing to use Spring Cloud Data Flow, is there any way we can create UI and show details of batch job and somehow also restart the batch Job?
Related
We have a use case where data can be sourced from different sources (DB, FILE etc) and transformed and stored to various sinks (Cassandra, DB or File).We would want the ability to split the jobs and do parallel loads - looks like Spring Batch RemoteChunking provides that ability.
I am new to SCDF and Spring batch and wondering what is the best way to use it.
Is there a way to provide configuration for these jobs (source connection details, table and query) and can this be done through an UI (SCDF Server UI ?). Is it possible to compose the flow?
This will run on Kubernetes and our applications are deployed through Jenkins pipeline.
We would want the ability to split the jobs and do parallel loads - looks like Spring Batch RemoteChunking provides that ability.
I don't think you need remote chunking, you can rather run parallel jobs, where each job handles an ETL process (for a particular file, db table).
Is there a way to provide configuration for these jobs (source connection details, table and query)
Yes, those can be configured like any regular Spring Batch job is configured.
and can this be done through an UI (SCDF Server UI ?
If you make them configurable through properties of your job, you can specify them through the UI when you run the task.
Is it possible to compose the flow?
Yes, this is possible with Composed Task.
I have had experience working with Spring Batch a few months but I have got a doubt a few days ago. I have to process a file and then update a database from it but this is not a scheduled batch process because it has to be executed just once.
Is Spring batch recommended to execute not scheduled processes like this one? Or the fact that is not scheduled has nothing to do with using Spring batch or not
Thanks
Is Spring batch recommended to execute not scheduled processes like this one? Or the fact that is not scheduled has nothing to do with using Spring batch or not
Yes, the fact that your job has to be executed only once has nothing to do with using Spring Batch or not. There is a difference between developing the job (using Spring Batch or not) and scheduling the job (using cron, quartz, etc).
For your use case (process a file and then update a database), I would recommend using Spring Batch to develop your job. Then, you can choose to run it:
only once or on demand (Spring Batch provides APIs to run the job)
or schedule it to run repeatedly using your favourite scheduler
I have been using Spring Batch and my metadata is in DB2. I have been using Spring Batch admin API (jars) to look at the current status of various jobs and getting details about job, like number of items read, commit count, etc. Now, since Spring Batch Admin is moved to spring-data-cloud, how do look at these informations? Is there a good API set I could use?
Basically, in Spring Cloud Data flow, you first need to create Spring Cloud Task that will have your Batch application: See example [here][1]
With the help of Spring Cloud #EnableTaskLauncher you can get the current status of job, run the job, stop the job, etc.
You need to send TasKLauncherRequest for it.
See APIs of TaskLauncher
Edition:
To get spring batch status, u need to have first Task execution id of spring cloud task. Set<Long> getJobExecutionIdsByTaskExecutionId(long taskExecutionId); method of [TaskExplorer][3]
See Task Explorer for all the apis. With it, use JobExplorer to get status of jobs
I have a Spring Batch job developed with Spring Boot (1.4.1.RELEASE).
It successfully runs from command line and writes job execution data to MySQL. It shows up as non-launchable job in Spring Batch Admin (2.0.0.M1, pointing to MySQL) and I can see job execution metrics.
Now I'd like to turn it into a launchable job so I can run it within Spring Batch Admin.
I wonder if anyone has done that before. The documentation has a section Add your Own Jobs For Launching. But it does not specify where to add the implementation jar(s) for the job?
Is it spring-batch-admin/WEB-INF/lib?
With Spring Boot, the non-launchable job is one big, all-in-one executable jar. Its dependencies overlap with Spring Batch Admin. For example, they both have spring-batch*.jar, spring*.jar but different versions.
Is there a way, like the job definition xml file, to keep them in separate contexts? Thank you.
Spring Batch Admin looks for your job definitions in src/main/resources/META-INF/spring/batch/jobs folder. You could add your job-definition.xml file in that folder and define your batch jobs in that xml.
We need to change an already running job. We should be able to push the job change without restarting the server.
Is it possible to reload a Spring batch job after the jobs / application context has been loaded.
The DefaultJobLoader allows you to reload the application context for your jobs.
Dynamic job deployment and editing of deployed job configurations (without requiring a server restart) is a feature we implemented in Trooper Batch profile (built on Spring Batch and Spring Batch admin). Screen shots are here : https://github.com/regunathb/Trooper/wiki/Writing-Batch-jobs-in-Trooper