My pipeline publishes two different build artifacts when all its tests have passed - stage: publish_pipeline_as_build.
One of my tests needs to use the build that was made in the current run, of the current version.
But additionally, I need to get the build artifact of the previous version, in order to run some compatibility tests.
How do I download the build artifact from that other pipeline run?
I know the build artifact name (from runtime script), but how would I find that?
I tried playing around with azure-cli az pipelines runs artifact list. It requies a --run-id and actually my script won’t have that.
So far I kind of managed, assuming the response of az pipelines runs list retuns the latest match to the query first:
az pipelines runs list --project PROJNAME --query "[?sourceBranch=='refs/heads/releases/R21.3.0.2']" | jq '.[0]'
I currently seem to run out of Ideas.
Perhaps just some confused/frustrated questions that pop up:
How can I find that specific build artifact name's latest version and download it?
How are pipeline tasks fed with runtime generated values?
Is this so ridiculously difficult when doing it in Azure DevOps, or am I just going the wrong way?
The job I'm trying to get there with:
jobs:
- job: test_session_integration
dependsOn: easysales_Build
steps:
- template: ./utils/cache_yarn_and_install.yml
- template: ./utils/update_webdriver.yml
- template: ./utils/download_artifact.yml
parameters:
artifact: easysales_$(Build.BuildId)_build
path: $(System.DefaultWorkingDirectory)/dist
# current release name as output
- template: ./utils/get_release_name.yml
# previous release name, branch and build name output
- template: ./utils/get_prev_release.yml
# clone prev version manually - can't use output variables as task input
# (BTW: why? that is super inconvenient, is there really no way?)
- bash: |
git clone --depth 1 -b $(get_prev_release.BRANCH_NAME) \
"https://${REPO_USERNAME}:${REPO_TOKEN}#dev.azure.com/organisation/PROJECTNAME/_git/frontend-app" \
./reference
workingDirectory: $(System.DefaultWorkingDirectory)
env:
REPO_TOKEN: $(GIT_AUTH_TOKEN)
REPO_USERNAME: $(GIT_AUTH_USERNAME)
name: clone_reference_branch
Any clues?
I'd be glad for any rubber ducking hints or clues on how I would be able to achieve what I need.
I'm new to Azure DevOps and currently struggle to find orientation in the vast but also quite in many places bits and pieces documentation Microsoft offers to me. It's fine, but frankly I struggle quite a bit with it. Is it just me having this problem?
All stages and full YAML on pastebin
The main template with the stages (Expanded templates made with "download full YAML"):
stages:
- stage: install_prepare
displayName: install & prepare
jobs:
- template: az_templates/install_hls_lib_build_job.yml
- stage: test_and_build
displayName: test and build projects
dependsOn: install_prepare
jobs:
- template: az_templates/build_projects_jobs.yml
- template: az_templates/test_session_integration_job.yml
- stage: publish_pipeline_as_build
displayName: Publish finished project artifacts as builds
dependsOn: test_and_build
jobs:
- template: az_templates/build_artifact_publish_jobs.yml
I by now found a solution. Perhaps not a definitive one, but should sort of work:
In my library variable group I added a azure devops personal access token with read access to the necessary groups as ADO_PAT_TOKEN
I Sign in with azure-cli with that token
I get the latest run id with azure cli
- bash: |
az devops configure --defaults organization=$(System.CollectionUri)
az devops configure --defaults project=$(System.TeamProject)
echo "$AZ_DO_TOKEN" | az devops login
AZ_QUERY="[?sourceBranch=='refs/heads/$(prev_release.BRANCH_NAME)'] | [0].id"
ID=$(az pipelines runs list --query-order FinishTimeDesc --query "$AZ_QUERY")
echo "##vso[task.setvariable variable=ID;isOutput=true]$ID"
env:
AZ_DO_TOKEN: $(ADO_PAT_TOKEN)
name: prev_build_run
I then download the artifact with azure-cli and the queried run id
- bash: |
az pipelines runs artifact download \
--artifact-name 'easysales_$(prev_release.PREV_RELEASE_VERSION)' \
--run-id $(prev_build_run.ID) \
--path '$(System.DefaultWorkingDirectory)/reference/dist/easySales'
workingDirectory: $(System.DefaultWorkingDirectory)
name: download_prev_release_build_artifact
This roughly seems to work for me now... finally 😉
Missing
The personal access token I added to the secrets may work, but AFAIS these tokens can not be created with no expiry date further than one year in the future.
That is not ideal, since I don't want my pipeline to stop working and perhaps no one around knows how to fix this.
Perhaps someone knows how I can use azure CLI within the current pipeline without authentication?
Given it accesses only the current organization and project?
Or does anyone see a more elegant solution to my admittedly clumsy solution.
Related
We've been migrating some of our manual deployment processes from Octopus to Azure DevOps Yaml pipelines. One of the QoL changes we're sorely missing is to be able to select the environment from a drop-down list/ auto-complete field as we could in Octopus.
Is there a way to achieve this? Currently, the only way I can think of doing it is to have a repo with a .yaml template file updated with a list of new environments as part of our provisioning process... Which seems less than ideal.
If you are going to trigger the pipeline manually then you can make use of Runtime parameters in the Azure DevOps pipeline.
For Example:
In order to make OS image name selectable from a list of choices, you can use the following snippet.
parameters:
- name: EnvName
displayName: EnvName
type: string
default: A
values:
- A
- B
- C
- D
- E
- F
trigger: none # trigger is explicitly set to none
jobs:
- job: build
displayName: build
steps:
- script: echo building $(Build.BuildNumber) with ${{ parameters.EnvName }}
Documentation about runtime parameters are here.
The downside to this is that the trigger: None limits you that the pipeline can only be manually triggered. Not sure how this works with other trigger options.
We have a project with repo on Azure DevOps where we store ARM templates of our infrastructure. What we want to achieve is to deploy templates on every commit on master branch.
The question is: is it possible to define one pipeline which could trigger a deployment only of ARM templates changed with that commit? Let's go with example. We 3 templates in repo:
t1.json
t2.json
t3.json
The latest commit changed only t2.json. In this case we want pipeline to only deploy t2.json as t1.json and t3.json hasn't been changed in this commit.
Is it possible to create one universal pipeline or we should rather create separate pipeline for every template which is triggered by commit on specific file?
It is possible to define only one pipeline to deploy the changed template. You need to add a script task to get the changed template file name in your pipeline.
It is easy to get the changed files using git commands git diff-tree --no-commit-id --name-only -r commitId. When you get the changed file's name, you need to assign it to a variable using expression ##vso[task.setvariable variable=VariableName]value. Then you can set the csmFile parameter like this csmFile: '**\$(fileName)' in AzureResourceGroupDeployment task
You can check below yaml pipeline for example:
- powershell: |
#get the changed template
$a = git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)
#assign the filename to a variable
echo "##vso[task.setvariable variable=fileName]$a"
- task: AzureResourceGroupDeployment#2
inputs:
....
templateLocation: 'Linked artifact'
csmFile: '**\$(fileName)'
It is also easy to define multiple pipelines to achieve only deploying the changed template. You only need to add the paths trigger to the specific template file in the each pipeline. So that the changed template file only triggers its corresponding pipeline.
trigger:
paths:
include:
- pathTo/template1.json
...
- task: AzureResourceGroupDeployment#2
inputs:
....
templateLocation: 'Linked artifact'
csmFile: '**\template1.json'
Hope above helps!
What you asked is not supported out of the box. From what I understood you want to have triggers (based on file change) per a step or a job (depending hoq you organize your pipeline). However, I'm not sure if you need this. Deploying ARM template which was not changed will not affect you Azure resources if use use Create Or Update Resource Group doc here
You can also try to manually detect which file was changed (using powershell for instance and git commands), then set a flag and later use this falg to fire or not some steps. But it looks like overkill for what you want achieve.
I have repo A that holds pipeline templates. Repo A has the following azure-template.yml:
# Repo A / azure-template.yml
stages:
- stage: ${{ variables.stageName }}
jobs:
- job:
steps:
- task:
A lot of the code in repo A that has the templates refers to variables in the following format:
${{ variables.variableName }}. The variable file is in a different folder in repo A. (e.g. variables/variables.yaml)
Now let's move to repo B. Repo B has my azure-pipeline.yml that needs to build from repo A:
# Repo B / azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: repoA
ref: refs/heads/develop
variables:
- template: variables/variables.yml#templates
stages:
- template: azure-template.yml#templates # Template reference
When I run azure-pipeline.yml, I get the following error:
An error occurred while loading the YAML build pipeline. The string must have at least one character. Parameter name: environmentName
That parameter is not one of mine. I don't have it declared or set anywhere. This tells me it is Azure specific but I have no idea where/why it's there or where it is even set.
How can I run a pipeline with a yaml template that refers to a
variables file in a different repo with Azure DevOps?
You're in correct direction, at least a working direction. It's supported to do that like what you've done above. About the error you got, I assume there could be something wrong with your yaml syntax. You can try following steps to locate the issue:
Copy the content of azure-template.yml and variables.yaml directly into azure-pipeline.yml file, and run the pipeline again to check if the issue persists.
In your Azure-pipeline.yml, specify the trigger and pool.
In azure-template.yml, try replacing the ${{ variables.stageName }} with hard-code value.
This is my first time to see this error message, but according to Parameter name: environmentName.You can also check if Release.EnvironmentName has valid value in one PS task. Hope it helps :)
Problem
I have two repositories A and B in one project, each with their own pipeline A-CI and B-CI. The repos are Azure Repos Git (so not external ones). I got it working to trigger pipeline B-CI whenever A-CI has completed. If A-CI got triggered by a commit to branch develop, then B-CI is triggered to build master although B also has a develop branch.
I want to build a new release of B for the dev environment, when a new dev version of A has been built.
Is it possible to let a pipeline-resource trigger the B-CI pipeline to build the branch with the same name as the branch which was just built by the pipeline-resource? It would be fine for me if it would fallback to master in case a matching branch is not available in B.
This scenario is working however if A-C and B-CI would both refer to different pipeline yamls of the same repository.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
B-CI
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- master
- develop
- feature/*
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is always refs/heads/master
Write-Host $(Build.Reason) # is ResourceTrigger
Background Info
The main idea behind that is, that A contains the IaC project and whenever the infrastructure of the project changes, then all apps should be deployed, too.
I do not want to put the IaC into the app repo because we have multiple apps, so I would have to split the IaC code into several chunks.
And then I would probably still have the same problem because some resources, like Azure KeyVault, are shared among the apps so A would still include the common stuff used by all apps and changes to it would require re-deployments of all apps.
Please check pipeline triggers:
If the triggering pipeline and the triggered pipeline use the same repository, then both the pipelines will run using the same commit when one triggers the other. This is helpful if your first pipeline builds the code, and the second pipeline tests it.
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
In this case, since master if the default branch of your B-CI, $(Build.SourceBranch) is always refs/heads/master.
As a workaround:
You can create a new yaml pipeline for repository B. You can use similar content of the yaml file for B-CI. And you only need to change something in it to:
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- develop
When we create the new yaml file, it's always placed in master branch. For me, I created a file with same name in dev branch, and copy the same content in it. Then i delete the new yaml file in master branch, now when dev of A-CI pipeline is built, dev of B repos will be used.
I think I have found a nicely working workaround as the built-in pipeline-triggers are not addressing our specific problem (though I can't say if we have an odd approach and there are better ways).
What I am doing now it to use the Azure CLI DevOps extension based on this docs entry and trigger the pipelines manually.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
displayName: Build something and create a pipeline artifact
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
# steps to publish artifact omitted
- stage: TriggerAppPipelines
displayName: Trigger App Pipeline
jobs:
- job: TriggerAppPipelinesJob
displayName: Trigger App Pipeline
steps:
- bash: az extension list | grep azure-devops
displayName: 'Ensure Azure CLI DevOps extension is installed'
- bash: |
echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
az devops configure --defaults organization=https://dev.azure.com/MyOrg project="MyProject" --use-git-aliases true
displayName: 'Login Azure CLI'
env:
AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
# By passing the build Id of this A-CI run, I can use that in B-CI to download the matching artifact of A-CI.
# If there is no matching branch, then the command fails.
- bash: |
az pipelines run --branch $(Build.SourceBranch) --name "B-CI" --variables a_Version="$(Build.BuildId)" -o none
displayName: 'Trigger pipeline'
B-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is same as the the triggering A-CI branch
Write-Host $(Build.Reason) # B-CI is triggered manually but the user is Project Collection Build Service, so automated runs can be distinguished
As B-CI is triggered manually now, there is no need for a resource node anymore.
I neet to run jobB only when jobA passes. I have to create a release tag only when my test stage passes successfully (I have added a code for that to happen here). My repository has 'README.md' file already. I am just checking its existence in my test stage. So, my test stage will always pass. Please let me know how do I write a code to create a release tag. A tag can be for example v1.1
stages:
- build
- test
- release
jobA:
stage: test
script:
- test -e README.md && exit 0
jobB:
stage: release
when: on_success
script:
# code for creating a release tag
In addition of Greg Woz's answer, in which GITLAB_API_TOKEN (a PAT -- PErsonal Access Token) is used, you now can also use a dedicated private project token:
See GitLab 13.9 (February 2021)
Support PRIVATE-TOKEN to create releases using the Release-CLI
In this milestone, we added the ability to use the release-cli with a PRIVATE-TOKEN as defined in the Create a release API documentation.
This enables overriding the user that creates a release, and supports automation by allowing connection of a project-level PRIVATE-TOKEN or by using a bot user account’s PRIVATE-TOKEN.
See Documentation and Issue.
It may not answer fully for your question but hopefully it helps. Perhaps try the below. My solution assumes you have package.json with version as it's the most popular case - but there is nothing wrong if you use any other way to define version:
version:
stage: version
only:
- master
script:
- VERSION_NUMBER=$(node -p "require('./package.json').version")
- "curl --header \"PRIVATE-TOKEN: $GITLAB_API_TOKEN\" --request POST \"https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/repository/tags?tag_name=v${VERSION_NUMBER}&ref=$CI_COMMIT_SHA&message=Tagged%20with%20version%20v${VERSION_NUMBER}\""
Then you can use:
publish:
stage: publish
only:
- tags
script:
- npm version --no-git-tag-version ${CI_COMMIT_TAG:1}
- npm publish --access public
for example in a next step.
Gitlab 13.2 introduced release keyword in .gitlab-ci.yml. There's no need anymore to create a "jobB" job, you can add keyword to main job as long as it can access release-cli:
stages:
- build
- test
- release
jobA:
stage: test
image: registry.gitlab.com/gitlab-org/release-cli:latest
script:
- test -e README.md && exit 0
release:
tag_name: $TAG
description: './path/to/CHANGELOG.md'
GitLab Docs has some examples to dynamically provide $TAG and description.
Probably the dependencies mechanism is a right tool for doing that. So, you can use README.MD (or test results folder) as an artifact in jobA and add the dependency on jobA in the jobB
jobB:
....
dependencies:
- jobA
If the jobA fail it won't provide dependent artifact for jobB. Thus, it also will fail.
(Edit)
Have you checked Protected tags gitlab settings? (Settings -> Repository -> Protected tags)
Default settings are:
By default, protected tags are designed to:
Prevent tag creation by everybody except Maintainers
Prevent anyone from updating the tag
Prevent anyone from deleting the tag