In a job I am using MultiResourceItemReader to read files from folder, in the processor of this job I am need to invoke (initiate) a Job(to process file) for each file available it the folder.
I am not sure whether I should be having it as a Job or a Step. So I need to programmatically start a Job or Step.(which in turn triggers multiple steps)
in my MultiResourceItemReaderJob's processor
Please let me know how I can do it. any possible links/sample code will help.
Thank you very much
Related
We have some spring-batch jobs are triggered by autosys with shell scripts as short lived processes.
Right now there's no way to view what is going on in the spring-batch process so I was exploring ways to view the status & manage(stop) the jobs.
Spring Cloud Data Flow is one of the options that I was exploring - but it seems that may not work when jobs are scheduled with Autosys.
What are the other options that I can explore in this regard and what is the recommended approach to manage spring-batch jobs now?
To stop a job, you first need to get the ID of the job execution to stop. This can be done using the JobExplorer API that allows you to explore meta-data that Spring Batch is aware of in the job repository. Once you get the job execution ID, you can stop it by calling the JobOperator#stop method, please refer to the Stopping a job section of the reference documentation.
This is independent of any method you used to launch the job (either manually, or via a scheduler or a graphical tool) and allows you to gracefully stop a job and leave the repository in a consistent state (ready for a restart if needed).
I had developed a Job in Talend and built the job and automated to run the Windows Batch file from the below build
On the Execution of the Job Start Windows Batch file it will invoke the dimtableinsert job and then after it finishes it will invoke fact_dim_combine it is taking just minutes to run in the Talend Open Studio but when I invoke the batch file via the Task Scheduler it is taking hours for the process to finish
Time Taken
Manual -- 5 Minutes
Automation -- 4 hours (on invoking Windows batch file)
Can someone please tell me what is wrong with this Automation Process
The reason of the delay in the execution would be a latency issue. Talend might be installed in the same server where database instance is installed. And so whenever you execute the job in Talend, it will complete as expected. But the scheduler might be installed in the other server, when you call the job through scheduler, it would take some time to insert the data.
Make sure you scheduler and database instance is on the same server
Execute the job directly in the windows terminal and check if you have same issue
The easiest way to know what is taking so much time is to add some logs to your job.
First, add some tWarn at the start and finish of each of the subjobs (dimtableinsert and fact_dim_combine) to know which one is the longest.
Then add more logs before/after the components inside the jobs.
This way you should have a better idea of what is responsible for the slowdown (DB access, writing of some files, etc ...)
I am writing Spring batch for chunk processing.
I have Single Job and Single Step. Within that Step I have Chunks which are dynamically sized.
SingleJob -> SingleStep -> chunk_1..chunk_2..chunk_3...so on
Following is a case I am trying to implement,
If Today I ran a Job and only chunk_2 failed and rest of chunks ran successfully. Now Tomorrow I want to run/restart ONLY failed chunks i.e. in this case chunk_2. (I don't want to run whole Job/Step/Other successfully completed Chunks)
I see Spring batch allow to store metadata and using that it helps to restart Jobs. but I did not get if it is possible to restart specific chunk as discuss above.
Am I missing any concept or if it is possible then any pseudo code/theoretical explanation or reference will help.
I appreciate your response
That's how Spring Batch works in a restart scenario, it will continue where it left off in the previous failed run.
So in your example, if the in the first run chunk1 has been correctly processed and chunk2 failed, the next job execution will restart at chunk2.
I have a job in Rundeck with many tasks within, but when some task fails I have to duplicate de Job, remove all the other tasks, save it and then run this new reduced copy of my original job.
Is there a way to run only specific tasks without having to do all this workaround?
Thanks in advance.
AFAIK there is no way to do that.
As a workaround, you can simply add options for every step in your Rundeck job, so for instances, if you have 3 script steps in your job, you can add 3 options named: skip_step_1, skip_step_2 and skip_test_3 and then assign true to the ones that have finished successfully and false to the one that has failed in the first execution. And for every script step, you can add a condition whether to run it or not.
A smiliar feature request is already proposed to the rundeck team :
Optionally execute workflow step based on job options
i'm trying to set up a dependency between two jobs in the Talend Administration Center. When the first job is finished the second job should start. I don't want to implement it by tRunJob component in one of the jobs. That's why i have set up a file trigger. At the end of the first job, the trigger creates a file in a specified directory. Im not sure at which root directory the file would be searched. I've set up the path but it doesn't start the trigger. Does someone have any experience with that? Or is there an other possibility to set up such an dependency?
Cheers.
Your solution is correct but valid only outside TAC, and we need TAC mainly for job scheduling and for managing dependecy, so you have to develop your jobs separately and independently, then create your tasks in TAC and a execution plan to schedule your tasks.