i'm trying to set up a dependency between two jobs in the Talend Administration Center. When the first job is finished the second job should start. I don't want to implement it by tRunJob component in one of the jobs. That's why i have set up a file trigger. At the end of the first job, the trigger creates a file in a specified directory. Im not sure at which root directory the file would be searched. I've set up the path but it doesn't start the trigger. Does someone have any experience with that? Or is there an other possibility to set up such an dependency?
Cheers.
Your solution is correct but valid only outside TAC, and we need TAC mainly for job scheduling and for managing dependecy, so you have to develop your jobs separately and independently, then create your tasks in TAC and a execution plan to schedule your tasks.
Related
I want to create a chain process. When a parent's job ends, a child (or multiple children) job starts. The main problem is that I want to have only the current job pods deployed.
Something like: job1 starts --> job 1 finish --> job 2 starts --> job 2 finish --> job 3 starts --> job 3 finish.
How can I do this? I thought that I could create in disabled mode job 1, job 2, and job 3 at the same time and enable them when the respective parent job ends (maybe with a service?).
I recently read about a tool called argo-workflow but I am not sure if it will do the chain effect that I am searching for.
Yes. CI/CD solution like Argo workflow is the way to go. You can check this example on how to execute different task one by one using steps here.
We have some spring-batch jobs are triggered by autosys with shell scripts as short lived processes.
Right now there's no way to view what is going on in the spring-batch process so I was exploring ways to view the status & manage(stop) the jobs.
Spring Cloud Data Flow is one of the options that I was exploring - but it seems that may not work when jobs are scheduled with Autosys.
What are the other options that I can explore in this regard and what is the recommended approach to manage spring-batch jobs now?
To stop a job, you first need to get the ID of the job execution to stop. This can be done using the JobExplorer API that allows you to explore meta-data that Spring Batch is aware of in the job repository. Once you get the job execution ID, you can stop it by calling the JobOperator#stop method, please refer to the Stopping a job section of the reference documentation.
This is independent of any method you used to launch the job (either manually, or via a scheduler or a graphical tool) and allows you to gracefully stop a job and leave the repository in a consistent state (ready for a restart if needed).
I have a job in Rundeck with many tasks within, but when some task fails I have to duplicate de Job, remove all the other tasks, save it and then run this new reduced copy of my original job.
Is there a way to run only specific tasks without having to do all this workaround?
Thanks in advance.
AFAIK there is no way to do that.
As a workaround, you can simply add options for every step in your Rundeck job, so for instances, if you have 3 script steps in your job, you can add 3 options named: skip_step_1, skip_step_2 and skip_test_3 and then assign true to the ones that have finished successfully and false to the one that has failed in the first execution. And for every script step, you can add a condition whether to run it or not.
A smiliar feature request is already proposed to the rundeck team :
Optionally execute workflow step based on job options
In a job I am using MultiResourceItemReader to read files from folder, in the processor of this job I am need to invoke (initiate) a Job(to process file) for each file available it the folder.
I am not sure whether I should be having it as a Job or a Step. So I need to programmatically start a Job or Step.(which in turn triggers multiple steps)
in my MultiResourceItemReaderJob's processor
Please let me know how I can do it. any possible links/sample code will help.
Thank you very much
My requirement is:
Workflow should run daily at 2pm. Workflow has been scheduled to run at 2pm
We have lookup on master tables. Records with IDs that are not present in the master tables will get rejected.
These new IDs have to be loaded into the master tables manually and then the workflow has to be re-run.
Daily the same thing happens.
My question is -
Is it possible to schedule a workflow to run twice every day(one for the first run, the other to run after the master table is updated)?
If No, can I manually start a scheduled workflow? Will it make the workflow unscheduled?
Please, Can any one help me with this?
Informatica's scheduler is a weak spot. I guess using two copies of the same workflow with different schedules would be the easiest solution.
Got a solution for my problem.
Once a workflow is scheduled, even if a particular session has to be re-run manually, whole workflow has be run from the workflow manager.
If that particular session is run manually, scheduling will be gone.
So always run the workflow instead of a session, so that scheduling will remain.