I want to create a chain process. When a parent's job ends, a child (or multiple children) job starts. The main problem is that I want to have only the current job pods deployed.
Something like: job1 starts --> job 1 finish --> job 2 starts --> job 2 finish --> job 3 starts --> job 3 finish.
How can I do this? I thought that I could create in disabled mode job 1, job 2, and job 3 at the same time and enable them when the respective parent job ends (maybe with a service?).
I recently read about a tool called argo-workflow but I am not sure if it will do the chain effect that I am searching for.
Yes. CI/CD solution like Argo workflow is the way to go. You can check this example on how to execute different task one by one using steps here.
Related
Using Azure DevOps, I am creating a release pipeline in which I would like to have one stage having 1 job with 5 steps. First three steps are same task types but with different variables, and I would like to make them parallel so some flow would go like:
Job
parallel: step1, step2, step3
then: step4 (after all 3 parallel steps succeeded/failed)
then: step5 (after step4 is done)
This is the current job setup
I am not sure how to set up Control Options - run this task for all of these steps. I would need somehow to set the first three to run immediately (maybe Custom condition "always()") and step 4 and step 5 to run sequentially after previous steps are done.
Step 5 can be: Even if the previous task has failed unless the deployment was cancelled, but I am not sure if I set the same setting for step 4 will it consider only step 3 as previous or all three previous (step1 - step3) tasks.
Also for Execution plan Parallelism I guess it's ok to set multi-agent to three since I would have max 3 steps executing in parallel overall.
parallel: step1, step2, step3
If you have 5 task in one agent job, and just want to run previous three tasks parallel first, I'm afraid to say that this does not supported in Azure Devops.
While you put several tasks in one agent job, that's means the task will and must running in order. And also, as you mentioned in the second pic, the specified in Multi-agent is used for run agent job parallel, rather than run task parallel.
Fortunately, until now, there has been such suggestion raised by other user in our official Developer Community. And there are many users has the same demand with you. You can vote and comment there. Our Product Group team will take your suggestions kindly. If it has enough votes which means its high priority, and the team will consider it seriously.
Why rundeck not launching scheduled spark jobs even if the previous job is still executing?
Rundeck is skipping the jobs set to launch during the execution of the previous job, then after the completion of its execution launch new job based on the schedule.
But I want to launch a scheduled job even if the previous job is executing.
Check your workflow strategy, here you have an explanation about that:
https://www.rundeck.com/blog/howto-controlling-how-and-where-rundeck-jobs-execute
You can design a workflow strategy based on "Parallel" to launch the jobs simultaneously on your node.
Example using the parallel strategy with a parent job.
Example jobs:
Job one, Job two and Parent Job (using parallel strategy).
I have a job in Rundeck with many tasks within, but when some task fails I have to duplicate de Job, remove all the other tasks, save it and then run this new reduced copy of my original job.
Is there a way to run only specific tasks without having to do all this workaround?
Thanks in advance.
AFAIK there is no way to do that.
As a workaround, you can simply add options for every step in your Rundeck job, so for instances, if you have 3 script steps in your job, you can add 3 options named: skip_step_1, skip_step_2 and skip_test_3 and then assign true to the ones that have finished successfully and false to the one that has failed in the first execution. And for every script step, you can add a condition whether to run it or not.
A smiliar feature request is already proposed to the rundeck team :
Optionally execute workflow step based on job options
i'm trying to set up a dependency between two jobs in the Talend Administration Center. When the first job is finished the second job should start. I don't want to implement it by tRunJob component in one of the jobs. That's why i have set up a file trigger. At the end of the first job, the trigger creates a file in a specified directory. Im not sure at which root directory the file would be searched. I've set up the path but it doesn't start the trigger. Does someone have any experience with that? Or is there an other possibility to set up such an dependency?
Cheers.
Your solution is correct but valid only outside TAC, and we need TAC mainly for job scheduling and for managing dependecy, so you have to develop your jobs separately and independently, then create your tasks in TAC and a execution plan to schedule your tasks.
Is there any way to do the following:
I have 2 jobs. One job on offline node has to trigger the second one. Are there any plugins in Jenkins that can do this. I know that TeamCity has a way of achieving this, but I think that Jenkins is more constrictive
When you configure your node, you can set Availability to Take this slave on-line when in demand and off-line when idle.
Set Usage as Leave this machine for tied jobs only
Finally, configure the job to be executed only on that node.
This way, when the job goes to queue and cannot execute (because the node is offline), Jenkins will try to bring this node online. After the job is finished, the node will go back to offline.
This of course relies on the fact that Jenkins is configured to be able to start this node.
One instance will always be turn on, on which the main job can be run. And have created the job which will look in DB and if in the DB no running instances, it will prepare one node. And the third job after running tests will clean up my environment.