Is there a way to run set of rundeck steps in a workflow in parallel?
Example:
==> means sequential
|| means parallel
Ex: Step 1 ==> Step 2.A || Step 2.B ==> Step 3
The feature of Ruleset Workflow Strategy is a plugin that is only available in the Rundeck Enterprise Version. Here you can find the official documentation
In the Rundeck community version, you can set the parallel strategy to run all steps at the same time.
Related
I am new to kubernetes. I want to run a cronjob using kubernetes scheduler that does the following tasks in a sequence:
Ingest data from datawarehouse into a mysql RDMS
Process the data
Indexing of the data using elastic search
Step 2 should occur after the completion of step 1 and step 3 should occur after the completion of step 2. May I know how can I achieve this without using argo or Brigade and simply with kubernetes cronjobs (The idea is to make the ingestion-processing-indexing workflow simple)? And if there's a sample code available for it?
I have a Azure Devops pipeline (yml) with a stage that deploys an application to an environment and then runs a bunch of tests against it. This pipeline is triggered when a PR is created. We sometimes have multiple runs of the same pipelines happening at once resulting in two deployments to the same environment happening at the same time.
Is it possible to configure the pipeline in such a way that the deploy stage can only be executed one at a time?
Simple example of what I'm trying to do:
Pipeline (yml) with stages: 1) Build -> 2) Deploy/Test -> 3) Release
Run 1: Build: complete -> Deploy/Test In progress -> Release waiting for stage 2
Run 2: Build: complete -> Deploy/Test waiting for Run 1 stage 2
If you use YAML then make you release in deployment job where you use environment with enabled exclusive lock. However, this has some drawback:
On developer community you can find a feature request for better handling this. And there is one workaround which involved calling REST API. But, as someone already mentioned:
Workarounds involving polling end up with race conditions where two queued builds can both end up starting.
So there is no ideal solution, but if you can please upvote above mentioned request.
I've a realtime spark job which runs in EMR cluster and I've another batch job which runs in another EMR cluster and this job is triggered at specific time.
How to run both these jobs in one EMR cluster ?
Any suggestions.
If the steps in both the EMR are not dependent on each other, then you can use the feature called Concurrency in the EMR to solve your use case. This feature simply means that you can run more than 1 step in parallel at a time.
This feature is there from the EMR version 5.28.0. If you are using the older version than this then you can not use this feature.
While launching the EMR from the AWS console, this feature is termed as 'Concurrency' in the UI. you can choose any number between 1 to 256.
If you are launching the EMR from the AWS CLI, then this feature is termed as 'StepConcurrencyLevel'.
You can read more about this at multiple steps now in EMR and AWS CLI details
Using Azure DevOps, I am creating a release pipeline in which I would like to have one stage having 1 job with 5 steps. First three steps are same task types but with different variables, and I would like to make them parallel so some flow would go like:
Job
parallel: step1, step2, step3
then: step4 (after all 3 parallel steps succeeded/failed)
then: step5 (after step4 is done)
This is the current job setup
I am not sure how to set up Control Options - run this task for all of these steps. I would need somehow to set the first three to run immediately (maybe Custom condition "always()") and step 4 and step 5 to run sequentially after previous steps are done.
Step 5 can be: Even if the previous task has failed unless the deployment was cancelled, but I am not sure if I set the same setting for step 4 will it consider only step 3 as previous or all three previous (step1 - step3) tasks.
Also for Execution plan Parallelism I guess it's ok to set multi-agent to three since I would have max 3 steps executing in parallel overall.
parallel: step1, step2, step3
If you have 5 task in one agent job, and just want to run previous three tasks parallel first, I'm afraid to say that this does not supported in Azure Devops.
While you put several tasks in one agent job, that's means the task will and must running in order. And also, as you mentioned in the second pic, the specified in Multi-agent is used for run agent job parallel, rather than run task parallel.
Fortunately, until now, there has been such suggestion raised by other user in our official Developer Community. And there are many users has the same demand with you. You can vote and comment there. Our Product Group team will take your suggestions kindly. If it has enough votes which means its high priority, and the team will consider it seriously.
Start using cypress in docker compose on gitlab,
how can I run tests parallel in docker compose?
I use this command in docker:
npx cypress run -b chrome
What do I need to use for starting tests parallel and combine test results?
You will still need some type of "dashboard" to coordinate, record and parallelize your tests.
You can either use Cypress.io dashboard, or run your own service that coordinates parallelization.
For example (disclaiming that I am the author): https://github.com/agoldis/sorry-cypress
Cypress.io used to provide free parallelization but recently I have got this message:
This Thursday, July 11, 2019, we will begin enforcing limits on test recordings and based on your organization's recent usage of the Cypress Dashboard, your account will be impacted.
After exceeding 100% of your plan's test recording limit, parallelization will be disabled and new test recordings will be hidden from the dashboard.
Magnolia from Customer Support
Cypress.io provides free access to their dashboard for Open Source projects, and there's also a "free" plan with limited recording and parallelization capacity (as of Sep 24, 2019).
See https://www.cypress.io/pricing/
The Cypress Dashboard is required to run tests in parallel across machines.
Here's how it works:
The Cypress Dashboard keeps track of how long each of your tests takes every time you run them.
When your CI initially spin up a bunch of machines, they will reach out to the Cypress Dashboard and ask, "hey, what tests should I run?"
The Cypress dashboard knows how many machines you've spun up, and it will delegate tests to them such that the testing completes at roughly the same time on each one.
The Cypress Dashboard parallelization is free for all plans, including the free tier.
Diagram of the above from the parallelization docs