Is it possible to copy files from one build agent to another and kick it off as a part of the pipeline task?
One build agent is Linux but I need to continue my work on Windows agent.
Following Hanna's solution, attached more detailed working solution:
Agent-1 and Agent-2 are two different machines from a different agent pool.
Agent-1 does 2 steps:
CopyFiles - writes the file 'livne.txt' (can be any pattern) from the default working directory into the artifact's staging directory
PublishPipelineArtifact - publishes an artifact called: 'PROJECT_NAME' contains all the files copied into the default working directory.
Agent-2 does one main task:
DownloadPipelineArtifact - downloads the file 'livne.txt' (can be any pattern) from the 'PROJECT_NAME' artifact into the current agent's working directory.
simple bash script to make sure 'livne.txt' indeed exists on the working directory.
pool:
name: Agent-1
- task: CopyFiles#2
displayName: 'Copy txt file'
inputs:
SourceFolder: '$(system.DefaultWorkingDirectory)'
Contents: livne.txt
TargetFolder: '$(build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(build.ArtifactStagingDirectory)'
artifact: 'PROJECT_NAME'
dependsOn: Job_1
pool:
name: Self Hosted Ubuntu for Docker Multiplatform
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact'
inputs:
artifactName: 'PROJECT_NAME'
itemPattern: livne.txt
targetPath: '$(build.ArtifactStagingDirectory)'
- bash: |
ls $(build.ArtifactStagingDirectory)
cat $(build.ArtifactStagingDirectory)/livne.txt
displayName: 'Bash Script'
I think the best way to do this is typically to publish the files as artefacts of the pipeline and then download those artefacts again on the second agent. I have done this in projects before when one machine is consuming test results from the test agent to build reports.
You might imagine your pipeline would look something like this:
- job: Build
displayName: Build on Linux
steps:
...
- task: PublishPipelineArtifact#1
displayName: Publish Built binaries from Linux
inputs:
path: $(Build.SourcesDirectory)/bin/
artifact: Binaries
- job: Additional
displayName: Do something with the binaries on windows
steps:
- task: DownloadPipelineArtifact#2
inputs:
artifact: Binaries
targetPath: $(Pipeline.Workspace)/Binaries
...
I hope this helps! :)
Related
I have CI and CD pipelines using Azure DevOps for a frontend angular project. Both are separate pipelines.
Here goes the YAML file for the CI pipeline which produces published artifact: output_final.zip. The below pipeline leverages Azure Pipelines for generating the published artifact.
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- integration
pool:
vmImage: ubuntu-latest
steps:
- task: NodeTool#0
inputs:
versionSpec: '14.x'
displayName: 'Install Node.js'
- powershell: |
$buildNumber="$(Build.BuildNumber)"
echo $buildNumber > src/version.txt
- script: |
npm install -g #angular/cli
npm install
ng build --prod
displayName: 'npm install and build'
- task: CopyFiles#2
displayName: 'Copy Files of UI'
inputs:
SourceFolder: 'dist/source'
TargetFolder: '$(Build.ArtifactStagingDirectory)/output'
OverWrite: true
- task: ArchiveFiles#2
displayName: 'Archive'
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/output/'
includeRootFolder: false
archiveFile: '$(Build.ArtifactStagingDirectory)/output/output_final.zip'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/output/output_final.zip'
ArtifactName: drop
Now I have a separate CD pipeline which leverages self hosted private agent. In this CD pipeline, I want to consume artifacts published by the CI pipeline in the CD pipeline
Can anyone help me to know how to consume artifacts published by the CI pipeline and use it in the CD pipeline with some sample YAML example.
My suggestion would be to use the Publish Pipeline Artifact task to publish the CI artifact. Then, in your CD pipeline, use the Download Pipeline Artifact to consume it.
You can configure the download task to get the artifact from the current run or from a specific run. In your case, it sounds like you want a specific run since the CI pipeline is separate.
Here's an example of what the yaml tasks might look like:
#CI Task
- task: PublishPipelineArtifact#1
displayName: 'Publish pipeline Artifact'
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: '<Some Artifact Name>'
publishLocation: 'pipeline'
#CD Task example
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: ''
definition: ''
specificBuildWithTriggering: true
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
I am doing a simple Azure DevOps CICD deployment. I am following all the steps. First up, here is my YAML file. I have kept the comments as it is, just in case, I am making more mistakes than I am aware of.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
# Set variables
variables:
directory: ReactJSRecipeApp
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.13.0'
displayName: 'Install Node.js'
- script:
npm install
displayName: 'npm install'
- script:
npm run build
displayName: 'npm build'
- task: CopyFiles#2
displayName: 'Copy files'
inputs:
sourceFolder: 'build'
Contents: '**/*'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
cleanTargetFolder: true
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- task: PublishBuildArtifacts#1
displayName: 'Publish Build Artifacts'
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
ArtifactName: 'www' # output artifact named www
- task: AzureWebApp#1
displayName: 'Deploy to App Service'
inputs:
azureSubscription: 'ReactJSRecipeAppConnection'
appName: 'reactjsrecipeappwebapp2'
appType: 'webApp'
package: '$(System.ArtifactsDirectory)/$(Build.BuildId).zip'
customWebConfig: '-Handler iisnode -NodeStartFile server.js -appType node'
So, when I run this, I get no errors in the Pipeline.
Further, I want to point out the following.
If I download the artifacts zip folder, I am able to run that folder locally and get my react app running in a localhost server just fine.
I check my Azure web app via Kudo Tools, I see all the files inside wwwroot, with timestamps that match the zip file from the artifact folder. So, I am assuming that the files are indeed getting pushed and to the correct spot in the web server.
Before I run the CICD trigger, these azure web apps were created brand new, and I get the standard azure welcome/landing page. So, the web apps themselves are fine.
After all this, the website itself does not serve the pages. I get a 404. I have tried two different web apps on Azure but the same results.
Any advise, where I am going wrong?
Update 1
I decided to manually check the files on Filezilla. But, its empty!!!
But, KUDO shows files. I dont understand!
Update 2
So, I did a direct deploy from visual studio code with the artifact publish folder. the web app runs fine. So, did this step to make sure that the web app is configured correctly.
Alright, so, it looks like my YAML file was not correct. I finally got it to work.
I am posting it here if someone comes around looking for a ready to use React YAML file (because the Azure DevOps Documentation is not that useful in its current form)
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
- task: CopyFiles#2
inputs:
Contents: 'build/**' # Pull the build directory (React)
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
# Default value: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/build/'
includeRootFolder: false
- task: AzureWebApp#1
inputs:
azureSubscription: 'ReactJSRecipeAppConnection'
appName: 'ReactJSRecipeApp4'
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
The full repository with simple react code including the YAML is here.
https://github.com/Jay-study-nildana/ReactJSRecipeApp
There is a duplication of a dist folder in the artifacts produced by the AzureDevOps -> Pipelines, the duplication is the /dist folder and also /drop/dist folder. EDIT: Full azure-pipeline.yml file
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
# Major modification referencing
# https://dev.to/thisdotmedia/continuously-integrating-angular-with-azure-devops-2k9l
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
# Build angular app area
- script: npm install
displayName: 'npm install'
- script: npx ng build --prod
displayName: 'npm build'
# Testing area
- script: npm install puppeteer --save-dev
displayName: 'Installing puppeteer (Headless browser for testing)'
- script: npx ng test --watch=false --codeCoverage=true
displayName: 'Running Tests'
- task: PublishTestResults#2
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TEST-*.xml'
displayName: 'Publish Test Results'
# Publishing items
# deploy.psl (Powershell script to deploy)
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: 'deploy.ps1'
ArtifactName: 'drop'
publishLocation: 'Container'
# Firebase.json
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: Firebase.json'
inputs:
PathtoPublish: 'firebase.json'
ArtifactName: 'drop'
publishLocation: 'Container'
# App
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: 'dist'
ArtifactName: 'drop/dist'
publishLocation: 'Container'
displayName: 'Publish Artifacts'
# Code Coverage Results
- task: PublishCodeCoverageResults#1
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)/coverage/ng-azure-devops/cobertura-coverage.xml'
displayName: 'Publish Code Coverage Results'
- script: npx ng lint
displayName: 'Code Analysis'
I've tried the using 'drop' as the ArtifactName, which will NOT produce a duplicate folder artifact anywhere. I am very confused on why 'drop/dist' will produce another '/dist' artifact
AzureDevOps Duplicate dist folder in pipelines build ? Why?
I could reproduce this issue on my side.
When we use the publish the artifacts dist folder with ArtifactName: drop/dist, Azure Devops will create a new folder drop first, then publish the artifacts dist folder to that folder drop.
You can get this message from the build log:
Upload '/home/vsts/work/1/s/dist' to file container:
'#/3620698/drop/dist'
However, the drop folder is already present by default. When we publish the dist folder with with ArtifactName: drop/dist, there are two drop folder, then Azure devops will publish dist folder to those two drop folders:
In order to understand this problem more clearly, you could disable the Multi-stage pipelines in the Preview features, then you will get the output:
Obviously, there are two drop folders here, that is the reason why you get the Duplicate dist folder in pipelines build.
So, to resolve this issue, we could change the ArtifactName: drop/dist to ArtifactName: dropTest/dist:
Now, the duplicate dist folder disappears.
Hope this helps.
I am using Azure DevOps build pipeline to send a artifact over to a release pipeline. My build pipeline is as follows:
Install Node JS with NodeTool#0
Run a Script Command that runs npm install and then runs the test suite.
Post the code coverage results via PublishCodeCoverageResults#1
Publish artifact with PublishPipelineArtifact#1 and targetPath: '$(Pipeline.Workspace)' as the artifact directory.
When I originally did this without using the code coverage part, IE not installing node, the files sent over were 257 and was a little over 25MB. Completely workable and quick.
However when I installed node and ran code coverage the files exploded to 750MB, which is understandable - but is way too much to transfer as an artifact IMHO.
So, looking at a solution I found this: https://learn.microsoft.com/en-us/azure/devops/artifacts/reference/artifactignore?view=azure-devops
And, I added this into the root of the repo.
However, this file seems to have not made a difference using the PublishPipelineArtifact#1 at all. No matter what I did, all the files from the Workspace were used as the artifact.
What did work though, was physically deleting the files in the build process.
I realize that is a lot of detail for a question, but I would like to know if the .artifactignore file actually works with this build pipeline?
Is there a way I can use the artifact ignore file properly, and not worry about deleting files before creating the artifact?
Tried to change the target directory in the PublishPipelineArtifact#1 task.
Here is the current working YAML:
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test:devops
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: DeleteFiles#1
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: |
node_modules
coverage
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
The expected results would be a smaller artifact file that still posts code coverage and installs node.
The .artifactignore should be in the same directory as defined as target path of the publish pipelines artifact task (not only in the root of the repo like azure-pipelines.yml).
Is not in the docs and there is an issue about it here. also check here.
Ok, so I used the information from Shayki Abramczyk - which is the correct answer - but wanted to put a working yaml together for everyone. I had to copy my .artifactignore from the repo, which is: $(Pipeline.Workspace)/s to the root which is $(Pipeline.Workspace)
As soon as I did this - it showed results. As soon as I adjusted the artifact ignore file, the output reduced from 750+MB to 2.3MB and processed in 8 seconds.
Working YAML:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: CopyFiles#2
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: '**/.artifactignore'
TargetFolder: '$(Pipeline.Workspace)'
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
FYI - this is being built for angular, so this is the artifact ignore file I used. Your mileage will vary based on your project.
**/dist
**/node_modules
**/coverage
**/.git
Converting a pipeline from classic to YAML (so I know it works).
I build and my publish artifact task looks like this:
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)/MyArtifact'
artifact: MyArtifact
The result when build looks like this:
I then want to deploy this to a webapp
YAML:
- stage: Dev
jobs:
- job: DeployApp
displayName: Deploy Web App
pool:
name: myPrivate
demands: msbuild
steps:
- task: DownloadPipelineArtifact#2
inputs:
artifact: MyArtifact
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy App'
inputs:
WebAppKind: 'Web App On Windows'
azureSubscription: 'edited out...'
WebAppName: webAppName
package: '$(System.ArtifactsDirectory)'
enableXmlTransform: true
Which results in this output in Azure DevOps:
A difference in chunks but since size match very exactly I'm happy.
I then get error on the deployment so I started look in Kudo and can see that in SitePackages all my packages are 1 kb (or empty...). I expected 61.2 MB.
I took my working classic deployment, and there i had this row:
packageForLinux:'$(System.DefaultWorkingDirectory)/_MyArtifact/MyArtifact/MyApplication.zip'
But that path wasn't even found. And packageForLinux felt wrong but doesn't matter as I understand it.
So can someone please me in the right direction how to deploy a web app (on windows) using YAML?
The folder that's used is $(Pipeline.Workspace), not $(System.DefaultWorkingDirectory), the default folder is different for each task type, so you can't always trust they extract to the default working directory.
I've wasted a lot of time deciphering these folder paths. I'm adding this info to help anyone else with this issue. Obviously my artifact was empty because my staging folder was empty.
After a few hours of digging I discovered that $(Build.SourcesDirectory) represents the root of where my dacpac files are built to
In my case I'm using the CopyFiles task as it supports wildcards. So whatever dacpac files are generated under the bin subfolder will be picked up and copied to the artifacts staging directory
# Build and publish the database solution
trigger:
- master
# this builds on a self hosted pool rather than the MS cloud agent
pool:
name: MySelfHostedPool
demands:
- MSBuild
- SqlPackage
steps:
- task: MSBuild#1
displayName: Build DB solution
inputs:
solution: MemberDB.sln
logFileVerbosity: diagnostic
# Copy build output to artifact staging directory so they can be published as artifacts in the next step
- task: CopyFiles#2
displayName: Copy dacpac to artifacts staging
inputs:
SourceFolder: '$(Build.SourcesDirectory)'
flattenFolders: true
Contents: '**\bin\**\*.dacpac'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact#1
displayName: Publish dacpac
inputs:
artifactname: 'drop'
targetPath: $(Build.ArtifactStagingDirectory)