In this release pipeline, I have two tasks: one is running the kubectl command, and I need it to keep running while I run the second task. After researching for a while, I know that parallel tasks are not available in Azure DevOps, so I tried with multiple agents. However I could not make it work.
May I know which part am I missing?
My current config looks like this:
And in each of the agents, I selected "Multi-Agent" on parallelism with number of 2.
But it seems not the one I want.
What I want is, run the first job with kubectl port-forward command. And keep it running while second job start running. After second job Run script is finished, then the first job can end.
May I know in Azure DevOps is there a way to achieve this?
Thank you so much.
The easiest would be actually to use seprate stages. But if you want to use single stage you can do it as follows:
Define variable like this:
Configure parallelism on the job:
And then define custom condition on the tasks:
One task should have eq(variables['Script'], 'one') and the other eq(variables['Script'], 'two')
You will get two agents runs your jobs but in each job will actually do only one task:
Related
We are running into the following issue:
We have a job in our pipeline that runs tests. The number of tests need to be distributed over 4 agents to run optimal. It can happen that only one agent is available and the job will start to run all the load on that specific agent, which can then time-out because it takes too long for other agents to become available in time to share in the load.
In essence, if we run with 4 agents, the job will run with optimal efficiency.
My question: is it possible to let a job wait for a specific number of agents to become available before starting the tasks in the job?
That`s not possible through out-of-box features.... But you may create a simple PowerShell script that will query your agents statuses: https://learn.microsoft.com/en-us/rest/api/azure/devops/distributedtask/agents/list?view=azure-devops-rest-7.1
and use includeAssignedRequest
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/agents?includeAssignedRequest={includeAssignedRequest}&api-version=7.1-preview.1
if you see assignedRequest, your build agent is busy...
We're using SonarQube for tests, and there's one token it uses, as long as one pipeline is running, it goes fine, but if I run different pipelines (all of them have E2E tests as final jobs), they all fail, because they keep calling a token that expires as soon as its used by one pipeline (job). Would it be possible to have -all- pipelines pause at job "x" if they detect some pipeline running job "x" already? The jobs have same names across all pipelines. Yes, I know this is solved by just running one pipeline at a time, but that's not what my devs wanna do.
The best way to make jobs run one by one is set demands for your agent job to run on a specific self-hosted agent. Just as below, set a user-defined capabilities for the self-hosted agent and then require run on the agent by setting demands in agent job.
In this way, agent jobs will only run on this agent. Build will run one by one until the previous one complete.
Besides, you could control if a job should run by defining approvals and checks. By using Invoke REST API check to make a call to a REST API such as Gets a list of builds, and define the success criteria as build count is zero, then, next build starts.
We have a cloud full of self-hosted Azure Agents running on custom AMIs. In some cases, I have some cleanup operations which I'd really like to do either before or after a job runs on the machine, but I don't want the developer waiting for the job to wait either at the beginning or the end of the job (which holds up other stages).
What I'd really like is to have the Azure Agent itself say "after this job finishes, I will run a set of custom scripts that will prepare for the next job, and I won't accept more work until that set of scripts is done".
In a pinch, maybe just a "cooldown" setting would work -- wait 30 seconds before accepting another job. (Then at least a job could trigger some background work before finishing.)
Has anyone had experience with this, or knows of a workable solution?
I suggest three solutions
Create another pipeline to run the clean up tasks on agents - you can also add demand for specific agents (See here https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) by Agent.Name -equals [Your Agent Name]. You can set frequency to minutes, hours, as you like using cron pattern. As while this pipeline will be running and taking up the agent, the agent being cleaned will not be available for other jobs. Do note that you can trigger this pipeline from another pipeline, but if both are using the same agents - they can just get deadlocked.
Create a template containing scripts tasks having all clean up logic and use it at the end of every job (which you have discounted).
Rather than using static VM's for agent hosting, use Azure scaleset for Self hosted agents - everytime agents are scaled down they are gone and when scaled up they start fresh. This saves a lot of money in terms of sitting agents not doing anything when no one is working. We use this option and went away from static agents. We have also used packer to create the VM image/vhd overnight to update it with patches, softwares required, and docker images cached.
ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops
For those discovering this question, there's a much better way: run your self-hosted agent with the --once flag, documented here.
You'll need to wrap it in your own bash script, but something like this works:
while :
do
echo "Performing pre-job setup..."
echo "Waiting for job..."
./run.sh --once
echo "Cleaning up..."
sleep 2
done
Another option would be to use a ScaleSet VM setup which preps a new agent for each run and discards the VM when the run is done. It can prepare new VMs in the background while the job is running.
And I suspect you could implement your own IMaintenanceProvirer..
https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/Maintenance/MaintenanceJobExtension.cs#L53
I am using Azure DevOps Server 2020 and I have a release pipeline which has around 21 copy file tasks in it to copy the output of multiple microservices to different target paths and this takes almost around 23 mins to complete the release pipeline.
I want to optimize the release pipeline and save some time and thus I am thinking of running all the copy task simultaneously.
Under the copy tasks in Control Options section, I see Run this task option is available where we do have the option to define custom conditions but I am not sure which custom conditions do I need to define exactly so that all my copy tasks gets executed parallelly.
Could anyone please let me know what custom conditions will allow all the copy task to get executed in one go?
Currently it is not possible to have tasks run in parallel. It has been raised as a suggestion here but the feature hasn't been implemented
How to run multiple Copy Files task in a Azure DevOps Release pipeline simultaneously with Custom Conditions?
Just as TheWinterCoder pointed, Currently it is not possible to have tasks run in parallel.
But, as a workaround, you could divide the replication task into several different jobs and make the jobs run in parallel:
This requires you to have multiple agents available in the local agent pool:
I have a pipeline with 6 tasks in one job, this job runs on agent01(self-hosted), there are two users, user1 and user2(more privileges) on agent01.
My requirement is, my pipeline job running on agent01 must use user1 for task 1,2,3 and use2 for task 4,5,6. How to achieve this ? please suggest.
How to switch to a different user on same agent to run a particular
task in AzureDevOps pipeline
I'm afraid this is impossible to achieve. When pipeline is running, in the Initialize job step, the user has been determined, and we cannot assign another user to the tasks.
As Shayki suggested in the comment, you can split the job into two jobs and use different agents to run.