I am using Azure DevOps build pipeline to send a artifact over to a release pipeline. My build pipeline is as follows:
Install Node JS with NodeTool#0
Run a Script Command that runs npm install and then runs the test suite.
Post the code coverage results via PublishCodeCoverageResults#1
Publish artifact with PublishPipelineArtifact#1 and targetPath: '$(Pipeline.Workspace)' as the artifact directory.
When I originally did this without using the code coverage part, IE not installing node, the files sent over were 257 and was a little over 25MB. Completely workable and quick.
However when I installed node and ran code coverage the files exploded to 750MB, which is understandable - but is way too much to transfer as an artifact IMHO.
So, looking at a solution I found this: https://learn.microsoft.com/en-us/azure/devops/artifacts/reference/artifactignore?view=azure-devops
And, I added this into the root of the repo.
However, this file seems to have not made a difference using the PublishPipelineArtifact#1 at all. No matter what I did, all the files from the Workspace were used as the artifact.
What did work though, was physically deleting the files in the build process.
I realize that is a lot of detail for a question, but I would like to know if the .artifactignore file actually works with this build pipeline?
Is there a way I can use the artifact ignore file properly, and not worry about deleting files before creating the artifact?
Tried to change the target directory in the PublishPipelineArtifact#1 task.
Here is the current working YAML:
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test:devops
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: DeleteFiles#1
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: |
node_modules
coverage
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
The expected results would be a smaller artifact file that still posts code coverage and installs node.
The .artifactignore should be in the same directory as defined as target path of the publish pipelines artifact task (not only in the root of the repo like azure-pipelines.yml).
Is not in the docs and there is an issue about it here. also check here.
Ok, so I used the information from Shayki Abramczyk - which is the correct answer - but wanted to put a working yaml together for everyone. I had to copy my .artifactignore from the repo, which is: $(Pipeline.Workspace)/s to the root which is $(Pipeline.Workspace)
As soon as I did this - it showed results. As soon as I adjusted the artifact ignore file, the output reduced from 750+MB to 2.3MB and processed in 8 seconds.
Working YAML:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: CopyFiles#2
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: '**/.artifactignore'
TargetFolder: '$(Pipeline.Workspace)'
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
FYI - this is being built for angular, so this is the artifact ignore file I used. Your mileage will vary based on your project.
**/dist
**/node_modules
**/coverage
**/.git
Related
I am doing a simple Azure DevOps CICD deployment. I am following all the steps. First up, here is my YAML file. I have kept the comments as it is, just in case, I am making more mistakes than I am aware of.
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
# Set variables
variables:
directory: ReactJSRecipeApp
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.13.0'
displayName: 'Install Node.js'
- script:
npm install
displayName: 'npm install'
- script:
npm run build
displayName: 'npm build'
- task: CopyFiles#2
displayName: 'Copy files'
inputs:
sourceFolder: 'build'
Contents: '**/*'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
cleanTargetFolder: true
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- task: PublishBuildArtifacts#1
displayName: 'Publish Build Artifacts'
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
ArtifactName: 'www' # output artifact named www
- task: AzureWebApp#1
displayName: 'Deploy to App Service'
inputs:
azureSubscription: 'ReactJSRecipeAppConnection'
appName: 'reactjsrecipeappwebapp2'
appType: 'webApp'
package: '$(System.ArtifactsDirectory)/$(Build.BuildId).zip'
customWebConfig: '-Handler iisnode -NodeStartFile server.js -appType node'
So, when I run this, I get no errors in the Pipeline.
Further, I want to point out the following.
If I download the artifacts zip folder, I am able to run that folder locally and get my react app running in a localhost server just fine.
I check my Azure web app via Kudo Tools, I see all the files inside wwwroot, with timestamps that match the zip file from the artifact folder. So, I am assuming that the files are indeed getting pushed and to the correct spot in the web server.
Before I run the CICD trigger, these azure web apps were created brand new, and I get the standard azure welcome/landing page. So, the web apps themselves are fine.
After all this, the website itself does not serve the pages. I get a 404. I have tried two different web apps on Azure but the same results.
Any advise, where I am going wrong?
Update 1
I decided to manually check the files on Filezilla. But, its empty!!!
But, KUDO shows files. I dont understand!
Update 2
So, I did a direct deploy from visual studio code with the artifact publish folder. the web app runs fine. So, did this step to make sure that the web app is configured correctly.
Alright, so, it looks like my YAML file was not correct. I finally got it to work.
I am posting it here if someone comes around looking for a ready to use React YAML file (because the Azure DevOps Documentation is not that useful in its current form)
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
- task: CopyFiles#2
inputs:
Contents: 'build/**' # Pull the build directory (React)
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
# Default value: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/build/'
includeRootFolder: false
- task: AzureWebApp#1
inputs:
azureSubscription: 'ReactJSRecipeAppConnection'
appName: 'ReactJSRecipeApp4'
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
The full repository with simple react code including the YAML is here.
https://github.com/Jay-study-nildana/ReactJSRecipeApp
There is a duplication of a dist folder in the artifacts produced by the AzureDevOps -> Pipelines, the duplication is the /dist folder and also /drop/dist folder. EDIT: Full azure-pipeline.yml file
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
# Major modification referencing
# https://dev.to/thisdotmedia/continuously-integrating-angular-with-azure-devops-2k9l
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
# Build angular app area
- script: npm install
displayName: 'npm install'
- script: npx ng build --prod
displayName: 'npm build'
# Testing area
- script: npm install puppeteer --save-dev
displayName: 'Installing puppeteer (Headless browser for testing)'
- script: npx ng test --watch=false --codeCoverage=true
displayName: 'Running Tests'
- task: PublishTestResults#2
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TEST-*.xml'
displayName: 'Publish Test Results'
# Publishing items
# deploy.psl (Powershell script to deploy)
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: 'deploy.ps1'
ArtifactName: 'drop'
publishLocation: 'Container'
# Firebase.json
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: Firebase.json'
inputs:
PathtoPublish: 'firebase.json'
ArtifactName: 'drop'
publishLocation: 'Container'
# App
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: 'dist'
ArtifactName: 'drop/dist'
publishLocation: 'Container'
displayName: 'Publish Artifacts'
# Code Coverage Results
- task: PublishCodeCoverageResults#1
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)/coverage/ng-azure-devops/cobertura-coverage.xml'
displayName: 'Publish Code Coverage Results'
- script: npx ng lint
displayName: 'Code Analysis'
I've tried the using 'drop' as the ArtifactName, which will NOT produce a duplicate folder artifact anywhere. I am very confused on why 'drop/dist' will produce another '/dist' artifact
AzureDevOps Duplicate dist folder in pipelines build ? Why?
I could reproduce this issue on my side.
When we use the publish the artifacts dist folder with ArtifactName: drop/dist, Azure Devops will create a new folder drop first, then publish the artifacts dist folder to that folder drop.
You can get this message from the build log:
Upload '/home/vsts/work/1/s/dist' to file container:
'#/3620698/drop/dist'
However, the drop folder is already present by default. When we publish the dist folder with with ArtifactName: drop/dist, there are two drop folder, then Azure devops will publish dist folder to those two drop folders:
In order to understand this problem more clearly, you could disable the Multi-stage pipelines in the Preview features, then you will get the output:
Obviously, there are two drop folders here, that is the reason why you get the Duplicate dist folder in pipelines build.
So, to resolve this issue, we could change the ArtifactName: drop/dist to ArtifactName: dropTest/dist:
Now, the duplicate dist folder disappears.
Hope this helps.
Is it possible to copy files from one build agent to another and kick it off as a part of the pipeline task?
One build agent is Linux but I need to continue my work on Windows agent.
Following Hanna's solution, attached more detailed working solution:
Agent-1 and Agent-2 are two different machines from a different agent pool.
Agent-1 does 2 steps:
CopyFiles - writes the file 'livne.txt' (can be any pattern) from the default working directory into the artifact's staging directory
PublishPipelineArtifact - publishes an artifact called: 'PROJECT_NAME' contains all the files copied into the default working directory.
Agent-2 does one main task:
DownloadPipelineArtifact - downloads the file 'livne.txt' (can be any pattern) from the 'PROJECT_NAME' artifact into the current agent's working directory.
simple bash script to make sure 'livne.txt' indeed exists on the working directory.
pool:
name: Agent-1
- task: CopyFiles#2
displayName: 'Copy txt file'
inputs:
SourceFolder: '$(system.DefaultWorkingDirectory)'
Contents: livne.txt
TargetFolder: '$(build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(build.ArtifactStagingDirectory)'
artifact: 'PROJECT_NAME'
dependsOn: Job_1
pool:
name: Self Hosted Ubuntu for Docker Multiplatform
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact'
inputs:
artifactName: 'PROJECT_NAME'
itemPattern: livne.txt
targetPath: '$(build.ArtifactStagingDirectory)'
- bash: |
ls $(build.ArtifactStagingDirectory)
cat $(build.ArtifactStagingDirectory)/livne.txt
displayName: 'Bash Script'
I think the best way to do this is typically to publish the files as artefacts of the pipeline and then download those artefacts again on the second agent. I have done this in projects before when one machine is consuming test results from the test agent to build reports.
You might imagine your pipeline would look something like this:
- job: Build
displayName: Build on Linux
steps:
...
- task: PublishPipelineArtifact#1
displayName: Publish Built binaries from Linux
inputs:
path: $(Build.SourcesDirectory)/bin/
artifact: Binaries
- job: Additional
displayName: Do something with the binaries on windows
steps:
- task: DownloadPipelineArtifact#2
inputs:
artifact: Binaries
targetPath: $(Pipeline.Workspace)/Binaries
...
I hope this helps! :)
Converting a pipeline from classic to YAML (so I know it works).
I build and my publish artifact task looks like this:
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)/MyArtifact'
artifact: MyArtifact
The result when build looks like this:
I then want to deploy this to a webapp
YAML:
- stage: Dev
jobs:
- job: DeployApp
displayName: Deploy Web App
pool:
name: myPrivate
demands: msbuild
steps:
- task: DownloadPipelineArtifact#2
inputs:
artifact: MyArtifact
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy App'
inputs:
WebAppKind: 'Web App On Windows'
azureSubscription: 'edited out...'
WebAppName: webAppName
package: '$(System.ArtifactsDirectory)'
enableXmlTransform: true
Which results in this output in Azure DevOps:
A difference in chunks but since size match very exactly I'm happy.
I then get error on the deployment so I started look in Kudo and can see that in SitePackages all my packages are 1 kb (or empty...). I expected 61.2 MB.
I took my working classic deployment, and there i had this row:
packageForLinux:'$(System.DefaultWorkingDirectory)/_MyArtifact/MyArtifact/MyApplication.zip'
But that path wasn't even found. And packageForLinux felt wrong but doesn't matter as I understand it.
So can someone please me in the right direction how to deploy a web app (on windows) using YAML?
The folder that's used is $(Pipeline.Workspace), not $(System.DefaultWorkingDirectory), the default folder is different for each task type, so you can't always trust they extract to the default working directory.
I've wasted a lot of time deciphering these folder paths. I'm adding this info to help anyone else with this issue. Obviously my artifact was empty because my staging folder was empty.
After a few hours of digging I discovered that $(Build.SourcesDirectory) represents the root of where my dacpac files are built to
In my case I'm using the CopyFiles task as it supports wildcards. So whatever dacpac files are generated under the bin subfolder will be picked up and copied to the artifacts staging directory
# Build and publish the database solution
trigger:
- master
# this builds on a self hosted pool rather than the MS cloud agent
pool:
name: MySelfHostedPool
demands:
- MSBuild
- SqlPackage
steps:
- task: MSBuild#1
displayName: Build DB solution
inputs:
solution: MemberDB.sln
logFileVerbosity: diagnostic
# Copy build output to artifact staging directory so they can be published as artifacts in the next step
- task: CopyFiles#2
displayName: Copy dacpac to artifacts staging
inputs:
SourceFolder: '$(Build.SourcesDirectory)'
flattenFolders: true
Contents: '**\bin\**\*.dacpac'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact#1
displayName: Publish dacpac
inputs:
artifactname: 'drop'
targetPath: $(Build.ArtifactStagingDirectory)
I am struggling to get a simple build and deploy working and was hoping for some assistance. Could anyone review the steps and also why the Publish Artifacts does not work? It's a simple Angular 7 project.
Error:
[section]Starting: Publish Artifact: dist
========================================================================== Task: Publish Build Artifacts Description: Publish build
artifacts to Azure Pipelines/TFS or a file share Version:
1.142.2 Author : Microsoft Corporation Help : More Information
[warning]Directory 'D:\a\1\s\dist' is empty. Nothing will be added to build artifact 'dist'.
[section]Finishing: Publish Artifact: dist
YAML:
pool:
vmImage: Hosted VS2017
demands: npm
steps:
- script: |
echo Write your commands here
mkdir dist
echo Use the environment variables input below to pass secret variables to this script
displayName: 'Command - mkdir dist'
- task: Npm#1
displayName: 'npm install'
inputs:
verbose: false
- task: Npm#1
displayName: 'npm build'
inputs:
command: custom
verbose: false
customCommand: 'build --prod'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: dist'
inputs:
PathtoPublish: dist
ArtifactName: dist
- task: FtpUpload#1
displayName: 'FTP Upload: dist'
inputs:
credentialsOption: inputs
serverUrl: ‘xxx’
username: Tester2
password: 'Tester$2'
rootDirectory: dist
filePatterns: '*'
remoteDirectory: /
trustSSL: true
Azure DevOps Pipeline issue
The Publish Build Artifacts task is used to publish build artifacts to Azure Pipelines, TFS, or a file share.
But, just like Daniel and Andrey said, although you add the npm build, you did not set the installed folder to be dist. So the result of npm build will not be saved in the dist folder. In this case, the folder dist is empty.
Besides, to save the build result to the dist folder, you can try to use the option -- -op like following:
run ng build --prod -- -op ..\..\dist
The ..\..\dist should use relative path based on the project.json file.
Check the document JavaScript frameworks: AngularJS for some more details info.
Hope this helps.