I am using the utility function add_run_files and want to remove them after the run completes. How do I do this
I have gone through this documentation
You can set retention policy in your yml like:
pipelines:
- name: sample_pipeline_retention_policy
configuration:
retentionPolicy:
maxAgeDays: 10
minRuns: 5
All the data associated with runs exceeding the maxAgeDays will be deleted, including any storage in artifactory. Refer to this doc for more info
Related
I would like to have a master pipeline capable of running the pipelines of our system's individual components. I'd also like to be able to run any of those components' pipelines individually. Additionally, some of the component pipelines are configured using yaml, while others are using the classic approach. (I'm not sure if that figures into any possible solutions to this problem.) Those that are configured using yaml typically contain multiple jobs, and I'd need all of the jobs to run in those cases.
Using approach #2 recommended here, I tried the following:
jobs:
- job: build_and_deploy
displayName: Build and Deploy
cancelTimeoutInMinutes: 1
pool:
name: some-pool
steps:
- checkout: self
- template: component_one_pipeline.yml
- template: component_two_pipeline.yml
I receive an error for the following "unexpected values": trigger, resources, name, variables, and jobs. I'm guessing these aren't allowed in any yaml file referenced in the template step of another pipeline yaml file. As I mentioned above, though, I need these values in their files because we need to run the pipelines individually.
If possible, could someone point me in the direction of how to get this done?
EDIT: I have also tried the approach given here. I was thinking I'd have a master pipeline that essentially did nothing except serve as a trigger for all of the child pipelines that are supposed to run sequentially. Essentially, the child pipelines should subscribe to the master pipeline and run when it's done. I ended up with the following 2 files:
# master-pipeline.yml
trigger: none
pool:
name: some agent pool
steps:
- script: echo Running MASTER PIPELINE
displayName: 'Run master pipeline'
#child-pipeline.yml
trigger: none
#- testing-branch (tried these combinations trying to pick up master run)
#- main
pool:
name: some agent pool
resources:
pipelines:
- pipeline: testing_master_pipeline
source: TestingMasterPipeline
trigger: true
steps:
- script: echo Running CHILD PIPELINE 1
displayName: 'Run Child Pipeline 1'
Unfortunately, it's not working. I don't get any exceptions, but the child pipeline isn't running when I manually run the master pipeline. Any thoughts?
Thanks in advance.
The way that those approaches you linked work, and Azure DevOps build triggering works in general, is that a build completion can trigger another build, and you have to have the trigger in build to be triggered. So:
Yaml templates can't have things like triggers, so they won't really help you here (though you can of course split any of the individual pipelines to templates). Triggers are in the main yaml pipeline fail, which references the template-files. So you can't have a individual component pipelines as templates.
Yaml pipelines can be chained with the resources-declaration mentioned in the first link. The way this works is that the resource declaration is in the pipeline to be triggered, and you configure the conditions (like branch filters: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops#branch-filters) to the pipeline to be triggered. For example, in your component pipeline you declare the master pipeline as resource, and set the conditions when the component pipeline will be triggered, like when the master pipeline is run against /release/* -branch. Or just set the trigger to true in order to trigger the component pipeline from any succesful run of the master pipeline. The component pipeline can still have its own pipeline triggers at the start of the pipeline declaration.
The classic build definitions can also be chained via edit build definition -> triggers -> build completion (see, for example, here: https://jpearson.blog/2019/03/27/chaining-builds-in-azure-devops/). This works the same way as with yaml pipelines; you configure the conditions for this the classic pipeline to trigger, so add the master pipeline as trigger to the component pipelines. Again, you can also set pipeline triggers for the component pipeline.
The limitation here is, that a classic pipeline can be triggered by an yaml pipeline, but not vice versa. A similar limitation in the yaml resources-declaration; they can't be triggered by a classic pipeline. If you need such triggering, or otherwise find the "native" triggers not to be enough, you can of course shoot an Azure DevOps API call in either type of pipeline to trigger any pipeline. See: https://blog.geralexgr.com/cloud/trigger-azure-devops-build-pipelines-using-rest-api, or just search for the azure devops rest api and associated blog posts that trigger the api with powershell, the rest api -task or by some other means.
As it turns out, I needed to set the pipelines' default branch to the one I was testing on for things to work correctly. The code in my original post works fine.
If you run into this problem and you're using a testing branch to test your pipelines, check out the information here on how to configure your pipeline to listen for triggers on your branch. Apparently a pipeline only listens for them on its default branch. Note: The example in the link uses the "classic" approach to pipeline configuration as an example, but you can reach the same page from a yaml configuration's edit screen by clicking the 3 dots on the right and selecting "Triggers."
Hope this helps someone.
Been working on migrating the .gitlab-ci.yml to azure-pipelines.yml. I am not able to find equivalents of some specific gitlab keywords in azure.
For eg:
(1)
rules:
- if: $CI_MERGE_REQUEST_ID
when: manual
timeout: 5 minutes
interruptible: false
allow_failure: true
(2)
paths:
- $ARTIFACTS_DIR/
expire_in: 1 week
timeout: 15 minutes
How to have the particular job working only on a specific rule? The equivalent of predefined variable
$CI_MERGE_REQUEST_ID, the keys like rules, if,when,timeout, interruptible,allow_failure, artifacts, paths, expire_in, timeout on azurepipelines.yml file?
Some insights would be great?
GitLab CI and Azure DevOps are two different systems, so keep in mind not every feature of GitLab CI has a one-to-one match in ADO and there are likely to be significant differences in how they are used.
For the features you mentioned, here are the analogs in Azure DevOps:
GitLab keyword
ADO Equivalent
rules
jobs.job.condition or steps.step.condition
allow_failure
jobs.job.continueOnError (also available in steps/tasks)
timeout
jobs.job.timeoutInMinutes
when:manual
See Manual Intervention task (set and first task and use condition: on this task for equivalent of rules:if:when:manual)
artifacts
see steps.publish, steps.download, pipeline artifacts, and build artifacts
expire_in
see retention policies.
interruptible
no analog: all jobs can be cancelled in ADO and this cannot be prevented. Closest solution would be to set a high cancelTimeoutInMinutes value
Predefined variables like CI_MERGE_REQUEST_ID only exist for GitLab CI, not Azure DevOps. Azure DevOps pipelines do have their own predefined variables -- System.PullRequest.PullRequestId would be the equivalent of CI_MERGE_REQUEST_ID, for example... but this may depend on exactly how you are using ADO with your repository.
I have two repos on my Azure DevOps project. One for the Cloud Infrastructure deployment and another that contains my application code.
I have a YAML pipeline that is triggered after any of those repos build pipeline finishes. The pipeline looks a bit like this like this:
resources:
pipelines:
- pipeline: MyProject-Code
- pipeline: MyProject-Infrastructure
jobs:
- job: DeployInfrastructure
steps:
# Here are the tasks the deploy the project infrastructure
- job: DeployCode
steps:
# Here are the tasks that deploy the code
I would like to put a condition on the DeployInfrastructure job so it is only executed if the triggering pipeline is the infrastructure one as I do not need to redeploy it if the change only affects the application code.
However, when reading the documentation from Microsoft there does not seem to be a very straightforward way of doing this.
Have a look at Pipeline resource variables
In each run, the metadata for a pipeline resource is available to all
jobs in the form of predefined variables. The is the
identifier that you gave for your pipeline resource. Pipeline
resources variables are only available at runtime.
There are also a number of predefined variables called Build.TriggeredBy.*, amongst them Build.TriggeredBy.DefinitionName, however documentation suggests that for yaml pipeline with pipeline triggers the resource variables should be used instead
If the build was triggered by another build, then this variable is set
to the name of the triggering build pipeline. In Classic pipelines,
this variable is triggered by a build completion trigger.
This variable is agent-scoped, and can be used as an environment
variable in a script and as a parameter in a build task, but not as
part of the build number or as a version control tag.
If you are triggering a YAML pipeline using resources, you should use
the resources variables instead.
I'm having a hard time getting a grasp on this to be honest.
Right now my lab project is as follows:
PR to master -> Triggers Pre-Build Pipeline as condition to merge the code ->
On merge Infrastructure pipe runs only if any changes happen in my Infrastructure folder ->
On merge I want to run my deploy pipeline to deploy my web app to Azure.
The pipes in question do the things they ought to, i.e.
Pre build builds, publishes artifact, runs Unit tests, validates ARM templates.
Infra pipe deploys the necessary infra for my web app such as ResourceGroup, App plan, app service, key vault.
Deploy Pipe downloads the artifact produced in pre deploy and deploys to a stage slot and swaps it to production slot.
What I can't seem to get to work is the pipeline chaining through dependencies, if changes happen to both infra and web app code in master I want the infra pipe to run first and the deploy pipe only if it succeeds.
If I merge only app code I want only the deploy pipe to run regardless if the infra pipe ran or not.
If I merge only infra code I want only the infra pipe to run.
If I merge both app and infra code I want both infra and deploy pipe to run in specific order.
I feel this shouldn't be all that hard to accomplish, but I've spent way too much time trying to solve this to no avail, anyone able to help? :)
Edit:
Hey Sorry #HughLin-MSFT Been Trying to work around this a bit since we're trying to avoid running scripts left and right. :)
I saw you have Build Queuing planned in an upcoming release so for now I think we might have to wait for that.
If I were to merge my deploy and infra pipe, can I use:
trigger:
branches:
include:
- master
paths:
include:
- Infrastructure/*
At stage level and somehow skip a stage instead?
Seen multiple articles mention "Continue if skipped" but can't find any information on how to actually skip a stage.
For the first and second cases, you just need to set Path filters in Triggers, the pipeline only triggers when the file at the specified path is changed. Please refer to this.
For the third case, you can try to add two agent jobs in the infra pipe, add Trigger Azure DevOps Pipeline task to the second agent job to trigger the deploy pipe, and then set Only when all previous jobs have succeeded in Run this job drop-down box for job2. In addition, you need to add a powershell task before the Trigger Azure DevOps Pipeline task, and use a script to detect whether there is app code, run job2 if there is, and cancel job2 if not.
Update:
First you can create a new pipeline and create a variable:changedcode
Use Builds - Get rest api to get the commit , then get the changed code folder with Commits - Get Changes rest api.
Assign changed code folder name as value to changedcode variable.
Set custom conditions for the agent job. In the Infra job, if the changedcode variable value is Infra, run the Infra job. In the Infra job, use the Builds-Queue rest api or Trigger Azure DevOps Pipeline task to trigger the Infra pipeline. The same is true for Deploy job, the only difference is the custom condition expression.
Here is a sample structure in yaml:
jobs:
variables:
changedcode: ""
- job:
steps:
- powershell： |
#Get the changed code folder with rest api
- job: Infra
condition: containsValue($(changedcode), "Infra"))
- powershell： |
#queue Infra pipeline with rest api or Trigger Azure DevOps Pipeline task
- job: Deploy
condition: (containsValue($(changedcode), "deploy")),and ....
- powershell： |
#queue Deploy pipeline with rest api or Trigger Azure DevOps Pipeline task
I want to build a docker image in my pipeline and then run a job inside it, without pushing or pulling the image.
Is this possible?
It's by design that you can't pass artifacts between jobs in a pipeline without using some kind of external resource to store it. However, you can pass between tasks in a single job. Also, you specify images on a per-task level rather than a per-job level. Ergo, the simplest way to do what you want may be to have a single job that has a first task to generate the docker-image, and a second task which consumes it as the container image.
In your case, you would build the docker image in the build task and use docker export to export the image's filesystem to a rootfs which you can put into the output (my-task-image). Keep in mind the particular schema to the rootfs output that it needs to match. You will need rootfs/... (the extracted 'docker export') and metadata.json which can just contain an empty json object. You can look at the in script within the docker-image-resource for more information on how to make it match the schema : https://github.com/concourse/docker-image-resource/blob/master/assets/in. Then in the subsequent task, you can add the image parameter in your pipeline yml as such:
- task: use-task-image
image: my-task-image
file: my-project/ci/tasks/my-task.yml
in order to use the built image in the task.
UDPATE: the PR was rejected
This answer doesn't currently work, as the "dry_run" PR was rejected. See https://github.com/concourse/docker-image-resource/pull/185
I will update here if I find an approach which does work.
The "dry_run" parameter which was added to the docker resource in Oct 2017 now allows this (github pr)
You need to add a dummy docker resource like:
resources:
- name: dummy-docker-image
type: docker-image
icon: docker
source:
repository: example.com
tag: latest
- name: my-source
type: git
source:
uri: git#github.com:me/my-source.git
Then add a build step which pushes to that docker resource but with "dry_run" set so that nothing actually gets pushed:
jobs:
- name: My Job
plan:
- get: my-source
trigger: true
- put: dummy-docker-image
params:
dry_run: true
build: path/to/build/scope
dockerfile: path/to/build/scope/path/to/Dockerfile