I'm trying to cache the cypress installation in my build pipeline.
I have this task setup:
- task: Cache#2
inputs:
key: 'cypress | $(Agent.OS) | package-lock.json'
path: 'C:\npm\prefix\node_modules\cypress'
I've run the build pipeline multiple times but I'm always getting the same error:
There is a cache miss
This is the previous build:
As you can see it's the same fingerprint, so why is the caching not working?
One more thing - In order that cache task works as expected all task must succeed.
Set variable system.debug to true to get more information.
You would check the path to see whether it is correct on the agent machine (you are using self-hosted agent, correct?)
Usually, on the first run after the task is added, the cache step will report a "cache miss" since the cache identified by this key does not exist. As you always got "cache miss" issue, I suspect the cache is not created or uploaded correctly. You may try to do some modification for package-lock.json to recache and re-generated one new key, to see how's the result.
Check the "cache post-job task" results and see if the keys are different.
In my case I had to use npm install --no-save so that the package-lock.json file wasn't regenerated during the pipeline. This ensured the "cache post-job task" was using the same cache key when caching node_modules.
edit
Here is an exported example of what we currently use in our pipeline to cache npm modules.
(You must make sure that your package-lock.json is checked in to your code repository)
steps:
- task: Cache#2
displayName: 'Npm Install Cache'
inputs:
key: '"npm" | "$(Agent.OS)" | my-project/package-lock.json'
path: 'my-project/node_modules'
cacheHitVar: NpmInstallCached
- task: Npm#1
displayName: 'Npm Install'
inputs:
command: custom
workingDir: my-project/
verbose: false
customCommand: 'install --no-save'
condition: ne(variables['NpmInstallCached'], 'true')
Related
This post will be a bit longer, as I not only describe my problem, but also show my different attempts to solve the problem.
I have a solution contaning .Net-6-Web-Api-Project (csproj) and a C++/CLI-Wrapper-Project (vcxproj). I have a reference from the C#-Project to the c++-Project. I use DevOps 2019 and VS22 on my local building agent.
I'm not able to successfully run this solution through an Azure DevOps Pipeline using the task DotNetCoreCLI#2, VSBuild#1 or a custom script as a workaround for the MSBuild#1 to publish.
VSBuild
My initial approach was to simply use the VSBuild#1 task. Using this task does not allow the pipeline to start, with the following error:
##[Error 1]
No agent found in pool My_Pool which satisfies the specified demands:
agent.name -equals My_Agend_Unity_1
Cmd
msbuild
visualstudio
Agent.Version -gtVersion 2.153.1
The cause is the compatibility issue between DevOps 2019 and VS2022. The agent does not recognize VS2022 and therefore does not create system capabilities for it. Its the same issue for the MSBuild#1 and why I tried a custom script to work around, because it couldn't find MSBuild.
DotNetCoreCLI
The first error I got was:
error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
So I fixed that by adding the env variable to the task:
env:
PATH: 'C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170'
The resulting further error was:
##[error]Error: Unable to locate executable file: 'dotnet'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.
So I tried to fix it by using the Task UseDotNet#2, even though it doesn't make sense to me. But at the end I still get an error similar to the first error.:
MSBuild version 17.3.2+561848881 for .NET
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj : warning NU1503: Skipping restore for project "C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj". The project file may be invalid or missing targets required for restore. [C:\agent\_work\2\s\XXX\YYY\MySolution.sln]
Determining projects to restore...
"C:\agent\_work\2\s\XXX\YYY\DotNet6Project\MyProject.csproj" restored (in "2,4 sec").
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj(21,3):error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj(21,3): error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
##[error]Error: The process 'C:\agent\_work\_tool\dotnet\dotnet.exe' failed with exit code 1
##[error]Dotnet command failed with non-zero exit code on the following projects : C:\agent\_work\2\s\XXX\YYY\MySolution.sln
##[section]Finishing: Build & Publish XXX Service - DotNetCoreCLI#2
MSBuild
My last hope then was my custom script that I already use in another pipeline that accesses the same agent and uses MSBuild from VS22. This is the approach I've come furthest with, as it looks like the project builds fine, but then fails because of this error.
(ResolvePackageAssets Target) -> C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5):
error NETSDK1064: Package "Microsoft.EntityFrameworkCore.Analyzers",
Version 6.0.4, not found. It may have been deleted after the NuGet restore.
Otherwise, the NuGet restore may have been only partially completed due to limitations on the maximum path length.
[C:\agent\_work\2\s\XXX\YYY\DotNet6Project\MyProject.csproj]
How to proceed with it, I do not know right now. I enabled already long paths via Group Policy Editor→Administrative templates→All Settings→Enable Win32 long paths.
My yaml file:
pool:
name: 'My_Pool'
demands:
- agent.name -equals My_Agent
variables:
buildPlatform: 'x64'
buildConfiguration: 'Release'
solution: '$(System.DefaultWorkingDirectory)/XXX/YYY/MySolution.sln'
DotNet6Project: '$(System.DefaultWorkingDirectory)/XXX/YYY/DotNet6Project/MyProject.csproj'
CPPWrapper: '$(System.DefaultWorkingDirectory)/XXX/YYY/CPPWrapper/MyProject.vcxproj'
steps:
- task: NuGetToolInstaller#0
displayName: 'NuGet Tool Installer - NuGetToolInstaller#0'
name: 'NuGetToolInstaller'
inputs:
versionSpec: '>=6.1.0'
- task: NuGetCommand#2
displayName: 'NuGet Restore - NuGetCommand#2'
inputs:
command: 'restore'
restoreSolution: '$(solution)'
noCache: true
- task: BatchScript#1
displayName: 'Run BatchScript to create DLLs, Libs & Header - BatchScript#1'
inputs:
filename: '$(System.DefaultWorkingDirectory)/ICP/ZZZ/build_release.bat'
env:
PATH: 'C:\Program Files\CMake\bin'
- task: PowerShell#2
displayName: 'Run Powershell Script to unpack Packages from BatchScript for ZZZWrapper - PowerShell#2'
inputs:
filePath: '$(System.DefaultWorkingDirectory)/XXX/YYY/CPPWrapper/install_ZZZ_package.ps1'
# Workaround for MSBuild#1
- script: |
#echo off
setlocal enabledelayedexpansion
for /f "usebackq tokens=*" %%i in (`"!ProgramFiles(x86)!\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe`) do (set msbuild_exe=%%i)
"!msbuild_exe!" "$(solution)" /p:Configuration="$(buildConfiguration)" /p:Platform="$(buildPlatform)" /p:PackageLocation="$(build.artifactStagingDirectory)" /t:rebuild
displayName: 'Build - Script'
# ---------- VSBuild ----------------
#- task: VSBuild#1
# inputs:
# solution: '$(solution)'
# msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
# platform: '$(buildPlatform)'
# configuration: '$(buildConfiguration)'
# ---------- DotNetCoreCLI ----------
#- task: UseDotNet#2
# inputs:
# packageType: 'sdk'
# version: '6.x'
#- task: DotNetCoreCLI#2
# displayName: 'Build & Publish - DotNetCoreCLI#2'
# inputs:
# command: 'publish'
# publishWebProjects: false
# projects: '$(solution)'
# arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
# zipAfterPublish: false
# env:
# PATH: 'C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170'
- task: PublishBuildArtifacts#1
displayName: 'Publish Build Artifacts - PublishBuildArtifacts#1'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'XXXArtifact'
publishLocation: 'Container'
From your yaml file ,I understand you want to build a solution and then publish build artifact. Accroding the error of your described task, I would like to provide suggestions that you can check.
1 VSBuild#1
taskError information: No agent found in pool My_Pool which satisfies the specified demandsThis indicates there are no agents on your machine that meet the demand requirements. You should check whether exists the agent which name is My_Agend_Unity_1 and exists check for Cmd,msbuild,visualstudio,Agent.Version -gtVersion 2.153.1.
See more information refer to doc:pool definition
2 UseDotNet#2
There’s a warning warning NU1503: Skipping restore for project that indicates the packages required for the project MyProject are not restored correctly. You should edit the affected project to add targets for restore.
Please refer to doc:NuGet Warning NU1503
About the error MSB4019,you should check whether the project path "C:\Microsoft.Cpp.Default.props" exists. Here’s a ticekt similar to your issue.You can try to this workaround and see if it works.
3 MSBuild
MSBuildAbout error NETSDK1064, this error occurs when the build tools can't find a NuGet package that's needed to build a project. This is typically due to a package restore issue related to warning NU1503 inTask UseDotNet#2`. You can refer this doc:NETSDK1064: Package not found to take some actions provided to resolve this error.
I'm using the Cache#2 DevOps task to cache nuget packages from multiple projects:
variables:
NUGET_PACKAGES: $(Pipeline.Workspace)/.nuget/packages
- task: Cache#2
displayName: 'NuGet cache'
inputs:
key: 'nuget | "$(Agent.OS)" | **/packages.lock.json,!**/bin/**,!**/obj/**'
restoreKeys: |
nuget | "$(Agent.OS)"
nuget
path: $(NUGET_PACKAGES)
cacheHitVar: 'CACHE_RESTORED'
- task: NuGetCommand#2
displayName: 'NuGet restore'
condition: ne(variables.CACHE_RESTORED, true)
inputs:
command: 'restore'
restoreSolution: '$(solution)'
- task: VSBuild#1
displayName: 'Build solution'
...
I'm following the documentation here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/caching-nuget?view=azure-devops
In the 'NuGet cache' step, if there is a cache, it is restored:
Resolved to: nuget|"Windows_NT"|Gor2Y1OZWvAeaan3RC3GH9D0ldp6z17wm6JB2YUxrS0=
There is a cache hit: `nuget|"Windows_NT"|Gor2Y1OZWvAeaan3RC3GH9D0ldp6z17wm6JB2YUxrS0=`
Path =
Type = tar
Code Page = UTF-8
Characteristics = ASCII
Everything is Ok
Folders: 6022
Files: 8938
Size: 1894324465
Compressed: 7660544
Process exit code: 0
Cache restored.
If there is a cache hit, the 'NuGet restore' task is skipped:
Evaluating: ne(variables['CACHE_RESTORED'], True)
Expanded: ne('true', True)
Result: False
Then comes my problem. The 'Build solution' task fails with thousands of errors like these
##[error]NHO.Core\IdentityServerClientStartup.cs(32,12): Error CS0246: The type or namespace name 'OwinStartupAttribute' could not be found (are you missing a using directive or an assembly reference?)
It cannot resolve references to classes in NuGet packages.
Any idea why this fails? I'm following the documentation exactly, but I don't know if I'm still missing something?
Update: Possible solution/source of error
It seems to be an error in the documentation.
If I run the NuGet restore step normally, without the condition
- task: NuGetCommand#2
displayName: 'NuGet restore'
inputs:
command: 'restore'
restoreSolution: '$(solution)'
it is a lot quicker than normal (20 seconds instead of 2 minutes), so I'm guessing it uses the cache.
The documentation provides only a basic case that all packages and references are stored in .nuget/packages and no assets files are required. In this case, the users can skip the restore task and just perform the cache task.
However, in many other cases, performing a restore task is necessary if using Microsoft-hosted agent. Since most packages are already cached by cache task, the running time of nuget restore task will be significantly reduced.
We are trying to use the SonarQube v5.5 extension in our Azure DevOps pipelines. The build runs fine as long as we do not include the pipeline task SonarQubePrepare in our azure-pipelines.yml file. As soon as we include the mentioned task, the MSBuild seems to end in a loop with the error error MSB6006: "csc.dll" exited with code 137 and then keeps trying to build the same project.
We did a little research and found that this could be related to a memory issue. We have allocated 16gb of ram to the build AgentPool. The only effect seems to be that the loop runs faster than before.
Has anyone encountered a similar problem?
Pipeline:
jobs:
- job: 'BackendBuild'
pool:
name: 'xyz'
workspace:
clean: all
steps:
- checkout: git://$(repository) # Variable for repository name to checkout
persistCredentials: true
clean: true
# Build runs fine without this task
- task: SonarQubePrepare#5
displayName: Prepare analysis with SonarQube (MSBuild)
inputs:
SonarQube: 'Sonarqube' # Service connection name
scannerMode: 'MSBuild'
projectKey: $(repository)
projectName: $(repository)
- task: DotNetCoreCLI#2
displayName: 'Build projects'
inputs:
projects: '**/*.csproj'
arguments: '--configuration Release'
Log:
##[warning]myproj.Domain.Email/Services/ViewRenderingService.cs(31,35): Warning S4457: Split this method into two, one handling parameters
check and the other handling the asynchronous code.
Services/ViewRenderingService.cs(31,35): warning S4457: Split this
method into two, one handling parameters check and the other handling
the asynchronous code.
[/azp/agent/_work/1/s/myproj.Domain.Email/myproj.Domain.Email.csproj]
Sonar: (myproj.Domain.Email.csproj) Project processed successfully
Sonar: Preparing for Razor compilation, moved files
(/azp/agent/_work/1/.sonarqube/out/9/Issues.json;/azp/agent/_work/1/.sonarqube/out/9/output-cs/encoding.pb;/azp/agent/_work/1/.sonarqube/out/9/output-cs/file-metadata.pb;/azp/agent/_work/1/.sonarqube/out/9/output-cs/metrics.pb;/azp/agent/_work/1/.sonarqube/out/9/output-cs/symrefs.pb;/azp/agent/_work/1/.sonarqube/out/9/output-cs/token-cpd.pb;/azp/agent/_work/1/.sonarqube/out/9/output-cs/token-type.pb;/azp/agent/_work/1/.sonarqube/out/9/ProjectInfo.xml) to /azp/agent/_work/1/.sonarqube/out/9.tmp. Sonar: After Razor
compilation, moved files
(/azp/agent/_work/1/.sonarqube/out/9/Issues.Views.json;/azp/agent/_work/1/.sonarqube/out/9/output-cs/file-metadata.pb)
to /azp/agent/_work/1/.sonarqube/out/9.Razor. Sonar: After Razor
compilation, moved files
(/azp/agent/_work/1/.sonarqube/out/9.tmp/Issues.json;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/encoding.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/file-metadata.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/metrics.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/symrefs.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/token-cpd.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/output-cs/token-type.pb;/azp/agent/_work/1/.sonarqube/out/9.tmp/ProjectInfo.xml)
to /azp/agent/_work/1/.sonarqube/out/9 and will remove the temporary
folder.
##[error]/usr/share/dotnet/sdk/3.1.420/Roslyn/Microsoft.CSharp.Core.targets(70,5): Error MSB6006: "csc.dll" exited with code 137.
/usr/share/dotnet/sdk/3.1.420/Roslyn/Microsoft.CSharp.Core.targets(70,5):
error MSB6006: "csc.dll" exited with code 137.
[/azp/agent/_work/1/s/myproj.Data/myproj.Data.csproj]
myproj.Domain.Messages ->
/azp/agent/_work/1/s/myproj.Domain.Messages/bin/Release/netstandard2.1/myproj.Domain.Messages.dll
myproj.Domain.Email ->
/azp/agent/_work/1/s/myproj.Domain.Email/bin/Release/netcoreapp3.1/myproj.Domain.Email.dll
Sonar: (myproj.Domain.Messages.csproj) Project processed successfully
myproj.Domain.Email ->
/azp/agent/_work/1/s/myproj.Domain.Email/bin/Release/netcoreapp3.1/myproj.Domain.Email.Views.dll
myproj.backoffice.sts ->
/azp/agent/_work/1/s/myproj.backoffice.sts/bin/Release/netcoreapp3.1/myproj.backoffice.sts.dll
Sonar: (myproj.backoffice.sts.csproj) Project processed successfully
Build FAILED.
I have an Environment called 'Dev' that has a resource, which is a VM. As part of the 'Dev' pipeline I want to copy files from a specific folder on the develop branch of a specific repo to a specific folder on the VM that's on the Environment.
I've not worked with Environments before or yaml pipelines much but I gather I need to use the CopyFiles#2 task.
So I've got an azure pipeline yaml file something like this:
variables:
isDev: $[eq(variables['Build.SourceBranch'], 'refs/heads/develop')]
stages:
- stage: Build
jobs:
- job: Build
pool:
vmImage: 'windows-latest'
steps:
- task: CopyFiles#2
displayName: 'Copy Files'
inputs:
contents: 'myFolder\**'
Overwrite: true
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)
artifactName: myArtifact
- stage: Deployment
dependsOn: Build
condition: and(succeeded(), eq(variables.isDev, true))
jobs:
- deployment: Deploy
displayName: Deploy to Dev
pool:
vmImage: 'windows-latest'
environment: Dev
strategy:
runOnce:
deploy:
steps:
- script: echo Foo Bar
The first question is how to I get this to copy the files to a specific path on the Dev environment?
Is the PublishBuildArtifacts really needed? The reason I ask is that I want this to copy files every time the pipeline is run and not error if the artifact already exists.
It also feels a bit dirty to have to check the branch is the correct branch this way. Is there a better way to do it?
The deployment strategy you're using relies on specifying an agent pool, which means it doesn't run on the machines in the environment. If you use a strategy such as rolling, it will run the specified steps on those machines automatically, including any download steps to download artifacts.
Ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops#deployment-strategies
You need to publish artifacts as part of the pipeline if you want them to be automatically available to down-stream jobs. Each run will get a different set of artifacts, even if the actual artifact contents are the same.
That said, based on the YAML you posted, you probably don't need to. In fact, you don't need the "build" stage at all. You could just add a checkout step during your rolling deployment, and the repo would be cloned on each of the target machines.
Ok, worked this out with help from this article: https://dev.to/kenakamu/azure-devops-yaml-release-pipeline-trigger-when-build-pipeline-completed-54d5.
I've taken the advice from Daniel Mann regarding the strategy being 'rolling'. I then split my pipeline into 2 pipelines; 1 for building the artifacts and 1 for releasing (copying them).
If you want just download the particular folders instead of all the source files from the repository, you can try using the REST API "Items - Get" to download each particular folder individually.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/items?path={path}&download=true&$format=zip&versionDescriptor.version={versionDescriptor.version}&resolveLfs=true&api-version=6.0
For example:
Have the repository like as below.
Now, in the YAML pipeline, I just want to download the 'TestFolder01' folder from the main branch.
jobs:
- job: build
. . .
steps:
- checkout: none # Do not check out all the source files.
- task: Bash#3
displayName: 'Download particular folder'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
inputs:
targetType: inline
script: |
curl -X GET \
-o TestFolder01.zip \
-u :$SYSTEM_ACCESSTOKEN 'https://dev.azure.com/MyOrg/MyProj/_apis/git/repositories/ShellScripts/items?path=/res/TestFolder01&download=true&$format=zip&versionDescriptor.version=main&resolveLfs=true&api-version=6.0'
This will download the 'TestFolder01' folder as a ZIP file (TestFolder01.zip) into the current working directory. You can use the unzip command to decompress it.
[UPDATE]
If you want to download the particular folders in the deploy job which target to your VM environment, yes, the folders will be download into the pipeline working directory on the VM.
Actually, you can consider a VM type environment resource is a self-hosted agent installed on the VM. So, when your deploy job is targeting to the VM environment resource, it is running on the self-hosted agent on the VM.
The pipeline working directory is under the directory where you install the VM environment resource (self-hosted agent). Normally, you can use the variable $(Pipeline.Workspace) to get value of this path (see here).
stages:
- stage: Deployment
jobs:
- deployment: Deploy
displayName: 'Deploy to Dev'
environment: 'Dev.VM-01'
strategy:
runOnce:
deploy:
steps:
- task: Bash#3
displayName: 'Download particular folder'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
inputs:
targetType: inline
script: |
echo "Current working directory: $PWD"
curl -X GET \
-o TestFolder01.zip \
-u :$SYSTEM_ACCESSTOKEN 'https://dev.azure.com/MyOrg/MyProj/_apis/git/repositories/ShellScripts/items?path=/res/TestFolder01&download=true&$format=zip&versionDescriptor.version=main&resolveLfs=true&api-version=6.0'
I have a YAML build script in an Azure hosted git repository which gets triggered across 7 build agents running on a local VM. Every time this runs, the build performs a git clean which takes a significant amount of time due to a large node_modules folder which takes a long time to clean up.
The MSDN page here seems to suggest this is configurable but shows no detail of how to configure it. I can't tell whether this is a setting that should be specified on the agent, the YAML script, within DevOps on the pipeline, or where.
Is there any other documentation I'm missing or is this not possible?
Update:
The start of the YAML file is here:
variables:
BUILD_VERSION: 1.0.0.$(Build.BuildId)
buildConfiguration: 'Release'
process.clean: false
jobs:
###### ######################################################
###### 1 - Build and publish .NET
#############################################################
- job: net_build_publish
displayName: .NET build and publish
pool:
name: default
steps:
- script: echo $(BUILD_VERSION)
- task: DotNetCoreCLI#2
displayName: dotnet build $(buildConfiguration)
inputs:
command: 'build'
projects: |
myrepo/**/API/*.csproj
arguments: '-c $(buildConfiguration) /p:Version=$(BUILD_VERSION)'
The complete yaml is a lot longer, but the output from the first job includes this output in a Checkout task
Checkout myrepo#master to s
View raw log
Starting: Checkout myrepo#master to s
==============================================================================
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: myrepo (Git)
Prepending Path environment variable with directory containing 'git.exe'.
git version
git version 2.26.2.windows.1
git lfs version
git-lfs/2.11.0 (GitHub; windows amd64; go 1.14.2; git 48b28d97)
git config --get remote.origin.url
git clean -ffdx
Removing myrepo/Data/Core/API/bin/
Removing myrepo/Data/Core/API/customersettings.json
Removing myrepo/Data/Core/API/obj/
Removing myrepo/Data/Core/Shared/bin/
Removing myrepo/Data/Core/Shared/obj/
....
We have another job further down which runs npm install and npm build for an Angular project, and every build in the pipeline is taking 5 minutes to perform the npm install step, possibly because of this git clean when retrieving the repository?
Click on your pipeline to show the run history
Click Edit
Click the 3 dot kebab menu
Click Triggers
Click YAML
Click Get Sources
Set Clean to False and Save
To say this is obfuscated is an understatement!
I can't say what affect this will have though, I think the agent reuses the same folder each time a pipeline runs and I'm not Node.js developer so I don't know what leaving old node_modules hanging around will do!
P.S. what people were saying about pipeline caching I don't think is what you were asking, also pipeline caching zips up the cached folder and uploads it to your artifacts storage, it then downloads it each time, if you only have 1 build agent then actually not doing a git clean might be more efficent I'm not 100%
As I mentioned below. You need to calculate hash before you run npm install. If hash is the same as the one kept close to node_modules you can skip installing dependencies. This may help you achieve this:
steps:
- task: PowerShell#2
displayName: 'Calculate and save packages.config hash'
inputs:
targetType: 'inline'
pwsh: true
script: |
# generates a hash of package-lock.json
$newHash = Get-FileHash -Algorithm MD5 -Path (Get-ChildItem package-lock.json)
$hashPath = "$(System.DefaultWorkingDirectory)/cache-npm/hash.txt"
if(Test-Path -path $hashPath) {
if(Compare-Object -ReferenceObject $(Get-Content $hashPath) -DifferenceObject $newHash) {
Write-Host "##vso[task.setvariable variable=NodeModulesAreUpToDate;]true"
$newHash > $hashPath
Write-Host ("Hash File saved to " + $hashPath)
} else {
# files are the same
Write-Host "no need to install node_modules"
}
} else {
$newHash > $hashPath
Write-Host ("Hash File saved to " + $hashPath)
}
$storedHash = Get-Content $hashPath
Write-Host $storedHash
workingDirectory: '$(System.DefaultWorkingDirectory)/cache-npm'
- script: npm install
workingDirectory: '$(Build.SourcesDirectory)/cache-npm'
condition: ne(variables['NodeModulesAreUpToDate'], true)
git clean -ffdx will clean any change untracked by source control in the source. You may try Pipeline caching, which can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again. Check the following link:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#nodejsnpm
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
steps:
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
path: $(npm_config_cache)
displayName: Cache npm
In the checkout step, it allows us to set the boolean option clean to true or false. The default is true so it runs git clean by default.
Below is a minimal example with clean set to false.
jobs:
- job: Build_Job
timeoutInMinutes: 0
pool: 'PoolOne'
steps:
- checkout: self
clean: false
submodules: recursive
- task: PowerShell#2
displayName: Make build
inputs:
targetType: 'inline'
script: |
bash -c 'make'
More documentation and related options can be found here