Azure publish and download build artifact yaml task not working - azure-devops

So I am trying to copy file and publish it via Azure yaml pipeline task PublishBuildArtifacts#1 so that I can use that file in other pipeline by downloading build artifact.
- script: |
cat ./pipelines/config/abc.config
mkdir $(Build.ArtifactStagingDirectory)/abc-config
cp -R ./pipelines/config/abc.config $(Build.ArtifactStagingDirectory)/abc-config
displayName: "Archiving runtime config file"
# Publish Config Build Artifacts
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: "$(Build.ArtifactStagingDirectory)/abc-config/"
ArtifactName: "abc-config"
publishLocation: "Container"
when I use the task DownloadBuildArtifacts#0 and download that file to a specific path, it supposed to store that file to ./pipelines/config/abc.config
but when I type ls to see if I get this file, for some reason I couldn't able to get that file.
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
project: '$(System.TeamProjectId)'
pipeline: '$(System.DefinitionId)'
downloadType: 'single'
artifactName: 'abc-config'
downloadPath: '$(System.ArtifactsDirectory)/pipelines/config'
- bash: |
# input file and remove empty lines and/or lines with crlf
ls ./pipelines/config
pipeline Logs:
logs show that it's trying to download that file by adding artifact name before that file. I don't understand this behavior of adding artifact name before the file name.
Downloading abc-config/runtime.config to /home/vsts/work/1/a/pipelines/abc-config/runtime.config
Downloaded abc-config/runtime.config to /home/vsts/work/1/a/pipelines/abc-config/runtime.config
What should I need to do to store that file to pipelines/config/runtime.config?
Documentation of task: DownloadBuildArtifacts#0 doesn't mention anything about this behavior. so I am not sure what am I missing here. Any help would be highly appreciated.
Thank you.

When you publish build artifacts, the artifacts will be saved in the artifactName folder. So when you download them using DownloadBuildArtifacts task. The artifactName folder and all the artifacts contents will be downloaded. This is the default behavior.
As a workaround, you can change the artifactName to config for PublishBuildArtifacts task. See below:
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: "$(Build.ArtifactStagingDirectory)/abc-config/"
ArtifactName: "config"
publishLocation: "Container"
Then change the downloadPath to $(System.ArtifactsDirectory)/pipelines for DownloadBuildArtifacts task. See below:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
project: '$(System.TeamProjectId)'
pipeline: '$(System.DefinitionId)'
downloadType: 'single'
artifactName: 'config'
downloadPath: '$(System.ArtifactsDirectory)/pipelines'

Related

Get Build Artifacts from Azure DevOps Microsoft-hosted Build Pipeline

I have quite a bit of experience with Azure DevOps on-premise and know where the build artifacts are stored when I run a build pipeline. However, we are moving to Microsoft-hosted build agents, and won't have any on-prem build server to store the artifacts from the build.
My question is, how do I get the build artifacts created from a build pipeline processed by a Microsoft-hosted build agent? Ultimately, I would like to download those artifacts to a file share that we have on-prem. Is this something that can be done?
I created a build pipeline in Azure DevOps (not on-prem) and ran the build. I added the marketplace extension Publish Build Artifact to the build pipeline and I expected that an artifact would be published so that I could download it to our company server. No artifact was produced.
In Azure yaml based pipelines, you can either use the pipeline artifacts method in this the artifacts are stored and associated with the pipeline or the universal method required additional steps to setup that uses the internal feed to store the artifacts. For pipeline artifacts you necessarily don't need to version the artifact since it is contained inside the pipeline. For universal unless you are overwriting the artifact you need to uniquely version them by any version mechanism you want to use, either to leverage off from app source code versioning or your own custom versioning.. like timestamp etc.
Below is an example of an dot. net build with both publish methods done in yaml pipeline.
# To run and store your build
- task: DotNetCoreCLI#2
displayName: "Build Project"
enabled: true
inputs:
command: "build"
projects: $(SolutionPath)
arguments: "--configuration Release --output $(Build.ArtifactStagingDirectory)"
# Here we start the postbuild steps
# This is to archive the build artifacts, stored by file number.
- task: ArchiveFiles#2
displayName: "Archive Artifacts"
inputs:
rootFolderOrFile: "$(Build.ArtifactStagingDirectory)"
includeRootFolder: false
archiveType: "zip"
archiveFile: "$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip"
replaceExistingArchive: true
# This is to publish the artifact inside the pipeline, stored by file number.
- task: PublishBuildArtifacts#1
displayName: "Publish Artifacts"
inputs:
PathtoPublish: "$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip"
ArtifactName: "drop"
publishLocation: "Container"
# This is to publish the artifact inside the universal artifacts feeds, stored by date/timestamp variable passed below as filename from another step.
- task: UniversalPackages#0
inputs:
command: 'publish'
publishDirectory: '$(Build.ArtifactStagingDirectory)'
feedsToUsePublish: 'internal'
vstsFeedPublish: 'your feed'
vstsFeedPackagePublish: 'azure-pipeline-dotnetapi'
versionOption: 'custom'
versionPublish: '$(setBuildValues.ApplicationVersion)'
packagePublishDescription: '$(setBuildValues.appVersion)'
For download of artifacts to a path you can use the below task and pass your network location as the path for download, i am not sure if it can use UNC paths might have to map the network path.
steps:
- task: DownloadBuildArtifacts#1
displayName: 'Download Build Artifacts'
inputs:
buildType: "current"
downloadType: "single"
downloadPath: ${{ parameters.DownloadPath }} #INSERT YOUR DOWNLOAD PATH HERE.

Azure SQL - DacPac Deployment

I am currently trying to test my DacPac deplyoment pipelines. However, I get error message all the time:
##[error]No files were found to deploy with search pattern C:\agent\_work\5\a\sqlproj_artifacts_1\DWH\DWH\bin\Release\database.dacpacCheck out how to troubleshoot failures at
I can't find what the problem is. Here the code, idea why it does not work? Have done before and worked perfectly....
steps:
- task: VSBuild#1
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.SourcesDirectory)'
ArtifactName: 'sqlproj_artifacts_$(System.JobAttempt)'
publishLocation: 'Container'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'sqlproj_artifacts_$(System.JobAttempt)'
downloadPath: '$(System.ArtifactsDirectory)'
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: '$(System.ArtifactsDirectory)/sqlproj_artifacts_$(System.JobAttempt)/DWH/DWH/bin/Release/$(azureSqlDBName).dacpac'
I'm using AuthenticationType: 'server'
And this is the local path where project is:
source\repos\DataWarehouse\DWH\DWH
Tried to adjust the path, but it still doesn't work.
Hope you can help me, thanks!
You can check the path of the artifact you are looking for on the build screen when you click on artifacts:
Here the $(azureSqlDBName).dacpac file name looks suspicious to me, as in if the dacpac is produced from a sql project, then probably the file will have the project name, not the azure sql db name.
In addition to this, I also would try to publish separate build artifacts for separate deployment, so instead of creating an artifact for the whole source artifact directory, I would copy the dacpac from the source to staging directory and then publish the dacpac artifact:
- task: CopyFiles#2
displayName: 'Copy DacPac Files'
inputs:
SourceFolder: '$(Build.SourcesDirectory)'
Contents: '**/*.dacpac'
TargetFolder: '$(Build.ArtifactStagingDirectory)/db'
CleanTargetFolder: true
OverWrite: true
flattenFolders: true
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/db'
ArtifactName: 'sqlproj_artifacts_$(System.JobAttempt)'
publishLocation: 'Container'
Then it is easier to find what you are looking for when you need to use the artifact. Please note the flattenFolders flag which comes in handy when you are dealing with a single package within the artifact.
One more thing is the use of pipeline artifacts instead of build artifacts, since you are already working with yaml pipelines, and possibly multi-stage yamls, you can utilize pipeline artifacts which are automatically downloaded to the default workspace folder.
Hope it helps,

How to copy .jar of one repository to another repository in azure devops

I am using azure CI CD pipeline for build and deploy.
I am generating the abc.jar artifacts from A azure repository and I want to copy this artifact in another azure repository B (Both A and B repo are in same project)and then want to build repository B and then deploy.
I tried downloading artifacts manually and placing it in other repo. However I want this to be done automatically. Could you please help in how can I copy .jar of one repository to another?
You have three options:
pipeline resource - here you define your pipeline A as resource, you can also trigger pipeline B when pipeline A ends (or some stage in pipeline A even)
task DownloadBuildArtifacts
task DownloadPipelineArtifact
name: pipelineB
resources:
pipelines:
- pipeline: hadar
source: kmadof.hadar
branch: master
trigger:
branches:
- master
steps:
- script: env | sort
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '4fa6b279-3db9-4cb0-aab8-e06c2ad550b2'
pipeline: '72'
branchName: 'refs/heads/master'
buildVersionToDownload: 'specific'
downloadType: 'single'
downloadPath: '$(Pipeline.Workspace)/build'
artifactName: 'drop'
buildId: '$(resources.pipeline.hadar.runID)'
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: '4fa6b279-3db9-4cb0-aab8-e06c2ad550b2'
definition: '72'
specificBuildWithTriggering: true
buildVersionToDownload: 'specific'
pipelineId: '$(resources.pipeline.hadar.runID)'
artifactName: 'drop'
targetPath: '$(Pipeline.Workspace)/pipeline'
- download: hadar
So you don't have to add you artifact to any repo. I recommend you use resources.

Running a YAML build pipeline manually with artifacts selected from resources in the UI are ignored

We have a YAML file (pipelineB) which does what it must do, it downloads the latest build artifact from a specific other existing build pipeline (pipelineA). This is automatically triggered. Now we want to run this build pipelineB manually and be able to select another artifact version to be used. In the Azure DevOps build pipeline there is an option saying "Run pipeline" where you can select "Resources" to be used. If you select a resource you end up with a list of all builds from pipelineA which you can choose from.
If we then choose an older (e.g. 1.2.43-10019-master) build from the pipeline runs of pipelineA and run pipelineB and we look in the logging we see that it ignores what we manually selected and always downloads the latest version. I can understand that it does that because the DownloadBuildArtifact#0 step tells to use the latestFromBranch build version to download.
My question: how can we make use of manually selecting a resource build artifact and use the selected version further in the YAML pipeline? And ideally by default if you don't do a manual run/selection it should just use the latest version of an artifact.
Below an exerpt from our YAML pipeline:
name: pipelineB
resources:
pipelines:
- pipeline: pipelineA
source: pipelineA
branch: master
trigger:
branches:
- master
steps:
- task: DownloadBuildArtifacts#0
name:
displayName: 'Download pipelineA artifact'
inputs:
buildtype: specific
project: ourProjectName
pipeline: pipelineA
branchName: refs/heads/master
buildVersionToDownload: latestFromBranch
downloadType: specific
downloadPath: $(Pipeline.Workspace)
Working solution based on answer by #Krzysztof Madej. Only for the step
DownloadBuildArtifacts#0 the field buildVersionToDownload needs to be changed to specific and a new field needs to be introduced buildId referring to the pipelineA resource.
steps:
- task: DownloadBuildArtifacts#0
name:
displayName: 'Download pipelineA artifact'
inputs:
buildtype: specific
project: ourProjectName
pipeline: pipelineA
branchName: refs/heads/master
buildVersionToDownload: 'specific'
downloadType: specific
buildId: '$(resources.pipeline.pipelineA.runID)'
downloadPath: $(Pipeline.Workspace)
Please change buildVersionToDownload to specific and then use buildId: '$(resources.pipeline.hadar.runID)'
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '4fa6b279-3db9-4cb0-aab8-e06c2ad550b2'
pipeline: '72'
branchName: 'refs/heads/master'
buildVersionToDownload: 'specific'
downloadType: 'single'
downloadPath: '$(Pipeline.Workspace)'
artifactName: 'drop'
buildId: '$(resources.pipeline.hadar.runID)'
You can check available variables for pipeline resource here
resources.pipeline.<Alias>.projectName
resources.pipeline.<Alias>.projectID
resources.pipeline.<Alias>.pipelineName
resources.pipeline.<Alias>.pipelineID
resources.pipeline.<Alias>.runName
resources.pipeline.<Alias>.runID
resources.pipeline.<Alias>.runURI
resources.pipeline.<Alias>.sourceBranch
resources.pipeline.<Alias>.sourceCommit
resources.pipeline.<Alias>.sourceProvider
resources.pipeline.<Alias>.requestedFor
resources.pipeline.<Alias>.requestedForID
I checked it again and simple
- download: pipelineA
works as code above.

Combine results of several pipeline jobs into azure web app package

We're in the process of moving our product to an azure web app. We have an extensive existing pipeline containing multiple parallel jobs. One of these jobs compiles a asp.net web application. Some others compile a vue.js website. Currently, the results of the web application and the vue projects are combined in a separate stage, using a powershell script.
Now I can convert the publish step of the web application to generate a deployment package for azure. But what is the best way of also adding the vue outputs into this package so I can deploy it to azure correctly, without losing the parallel jobs? I cannot include the is output files in my project, because they don't exist within the web application build job
You can use publish build artifact task to upload the build results of the web application and vue projects to azure devops server as #Krzysztof mentioned. And you can add a new job to download the artifacts.
Please check below simple example in yaml.
In order to combine the build results, you can use extract file task to extract the zipped artifacts and published the unpacked artifacts in Build_web job. And in the Combine job you can use copy file task to copy the results of vue artifacts to the web artifacts folder. And then you can use archive file task to pack the artifacts which now contains the results of web and vue application.
Combine job should dependsOn Build_web and Build_vue jobs
jobs:
- job: Build_Web
pool:
vmImage: "windows-latest"
steps:
- task: ExtractFiles#1
inputs:
archiveFilePatterns: '*.zip'
destinationFolder: '$(Build.ArtifactStagingDirectory)\unzip'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)\unzip'
artifactName: webapp
- job: Build_Vue
pool:
vmImage: "windows-latest"
steps:
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: 'path to the build results'
artifactName: vueapp
- job: Combine
dependsOn:
- Build_Web
- Build_Vue
pool:
vmImage: "windows-latest"
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
artifactName: webapp
downloadPath: "$(System.ArtifactsDirectory)"
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
artifactName: vueapp
downloadPath: "$(System.ArtifactsDirectory)"
- task: CopyFiles#2
inputs:
SourceFolder: '$(System.ArtifactsDirectory)\vueapp'
TargetFolder: 'path to web application result folder' #eg. $(System.ArtifactsDirectory)\webapp\Content\d_C\a\1\s\AboutSite\AboutSite\obj\Release\netcoreapp2.0\PubTmp\Out\
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: $(System.ArtifactsDirectory)\webapp
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/webapplication.zip'
Above example only shows a general idea. You can aslo move the ExtractFile task to Combine job. In either way, you will have to use extract file, copy file and archive file task.
For TargetFolder parameter in copy file task, you can check the download build artifacts log for webapp artifact to get the full path. For example as below screenshot shows.
You can use PublishPipelineArtifact#1 to create artifacts for your projects and then in a separate job DownloadPipelineArtifact#2. By defining path you may compose your final artifact (if this mixing many projects is not more complicated than putting one inside another). And publish your artifact as build or pipeline artifact depending how you have roganized your release.
# Download an artifact named 'WebApp' to 'bin' in $(Build.SourcesDirectory)
- task: DownloadPipelineArtifact#2
inputs:
artifact: 'WebApp'
path: $(Build.SourcesDirectory)/bin
Here you have more info about publishing and downloding artifacts.