TLDR;
I have an Azure Devops Pipeline that uses chocolatey to install dependencies. I would like to cache downloaded chocolatey packages similar to how node dependencies are cached (see below) or use a caching proxy server.
Is it possible to cache downloaded packages from chocolatey to be available at the next run?
If that is not currently easily possible, then is it possible to run a caching proxy server similar to AptCacherNg for chocolatey packages?
Current Setup
I currently have a pipeline setup in Azure Devops that requires packages from chocolatey community. The step runs the equivalent of:
choco install nasm --confirm --no-progress
I am caching node dependencies using the following:
steps:
- task: Cache#2
displayName: 'Cache npm packages'
inputs:
key: '**/package-lock.json, !**/node_modules/**/package-lock.json, !**/.*/**/package-lock.json'
path: '$(System.DefaultWorkingDirectory)/node_modules'
I have considered whether it was possible to modify the keys for this to check the choco packages or use a duplicate step that uses this plugin but I don't know specifically how to do this.
Backgound
Recently, one of the modules website went offline for a few hours. While it was offline I noticed that in the logs it stated that licensed users likely didn't have the issue as they cache packages. I checked into license pricing and it seems that there is a reasonable $96/year cost for a single user license of up to 8 machines but in the license it states that using this would violate the terms for business usage. The business license is $16/year/machine with a minimum of 100 machines. $1600/year is a bit more than I want to pay at this time for such a small dev team that only needs a few packages installed. They suggested community edition.
Choco has a handy --cache option that lets you specify the cache location. Use this option together with a dedicated Cache step:
- task: Cache#2
displayName: 'Cache choco'
inputs:
key: 'path_to_a_file_or_just_a_string_that_you_update_manually'
path: '$(System.DefaultWorkingDirectory)/choco_cache'
- script: choco install nasm --confirm --no-progress --cache $(System.DefaultWorkingDirectory)/choco_cache
displayName: 'install choco packages'
Related
I would like to understand how the tasks work in Azure pipeline, and where can I find the documentation for it.
I have a self-hosted agent where I have installed Terraform, Azure CLI and Azure Powershell.
There are some tasks defined for them, I actually have difficulties to find them in a documentation,for example with terraform, I have to go to azure release and simulate an action of adding task, then convert that in yaml and there I have the task name:
steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV3#3
displayName: 'Terraform : azurerm'
inputs:
command: custom
customCommand: import
environmentServiceNameAzureRM: 'something'
Also, some tasks asks to install some resources in order to use them, will it affect the resources of my self-hosted agent? for example if we have to install the task before using it ,will it affect somehow the version of my self-hosted terraform?
I have azure powershell installed in the self-hosted agent with an old version, but there is a task that in the version parameter has the latest version, but when executing the task it uses my old version.
How versions for tasks work? Can I use the latest version or az cli without installing it in the self-hosted agent but only with the tasks for the deployments only?
- task: AzurePowerShell#5
displayName: 'DevCode'
inputs:
#azureSubscription: 'something'
azureSubscription: 'something'
ScriptPath: 'something'
azurePowerShellVersion: LatestVersion
The same for az cli.
For the commonly used pipeline tasks published by Microsoft (such as Azure CLI task and Azure PowerShell task), you can find the related documents here.
All these tasks are open-source on the GitHub repository "microsoft/azure-pipelines-tasks".
For the Terraform task, you can find it on Marketplace here and this GitHub repository.
Normally, the tasks would not change the tools have installed on your self-hosted agent machine, also not automatically install the required tools. They just call the required tools have installed on the agent machine.
For example, when you run the Azure CLI task on your self-hosted agent, if you have installed the Azure CLI tool on the agent machine, the task will call the tool to run the specified scripts. If the Azure CLI tool has not installed on the machine, normally the task will go to failure since it cannot find the required Azure CLI tool on the machine.
You do not need to manage the versions of the tasks, you just use the tasks in your pipelines and select the available released versions of the tasks (latest versions are recommended). If a new version of a task is released, you can go to edit your pipelines to use the new version.
I have uploaded a package that is being used by multiple pipelines. Some pipelines in some projects get the latest version and some pipelines in some projects do not. The key here is they all share the same pipeline template so I'm not quite sure where the problem may be or where to start looking. The call being made is:
install ${{ parameters.NugetPckDefinition }} -ExcludeVersion -NoCache -OutputDirectory "$(Build.SourcesDirectory)"
The versions are the following:
1.0.3169
1.0.3009
Some pipelines are getting 1.0.3169 and some are downloading 1.0.3009. Currently, in order for those pipelines that cannot get the latest version, I call the REST API to get the specific package so that it will show up in Azure Artifacts.
I suspect there are some permissions issues or possibly some caching issues, but the pipelines share the same Build Service account. I checked the permissions on the Agent Pool and they all have the same settings.
I have a legacy solution that we've moved into an Azure Devops git repo and I am trying to setup a build pipeline for it. The solution is a mixture of v141 (2017) and v140 (2015) projects, which on my local machine I can just build in Visual Studio 2017 as long as I've installed the v140 toolset.
Ideally, I'd like to use a Microsoft provided agent but it seems like the vs2017-win2016 image does not include the v140 toolset by default. As this is not something we are planning to build very often, I attempted to install the v140 toolset using the 2017 installer:
pool:
vmImage: 'vs2017-win2016'
steps:
- task: PowerShell#2
displayName: 'Install Visual Studio v140 Toolset'
inputs:
targetType: 'inline'
script: |
Write-Output "Starting the installation process. This will take some time"
$installProcess = Start-Process -FilePath $(Build.SourcesDirectory)/External/VisualStudioBuildTools/installer.exe -ArgumentList "--add Microsoft.VisualStudio.Component.VC.140", "--quiet", "--wait", "--norestart" -Wait -PassThru
Write-Output "Install Completed with code $($process.ExitCode)"
- task: VSBuild#1
displayName: 'Build Debug [.sln]'
inputs:
solution: '$(Build.SourcesDirectory)/LegacySolution.sln'
vsVersion: '15.0'
configuration: 'Debug'
platform: 'x64'
When I run this in Azure Devops, the install process exits with code 0 (success) after about a minute. However, when it then tries to build the solution it fails with:
##[error]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.Cpp.Platform.targets(67,5): Error MSB8020: The build tools for v140 (Platform Toolset = 'v140') cannot be found. To build using the v140 build tools, please install v140 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".
Has anyone tried this before with any luck? The only other thing I can think of is to try to check in a copy of the v140 toolset and play with adding it correctly to the path, but I can see this being a major pain to get right!
Installing the Visual Studio 2015 toolkit (v140) on a Microsoft hosted Azure Devops Build Pipeline Agent
In generally you can't. If something requires admin access and you're using the hosted agent, you can't do that thing. I have test this command line in my local, I need to run the powershell with Administrator, otherwise, I will get a confirmation prompt.
Besides, MS replied:
VS has grown to a point, where installing multiple versions on the
same agent is not feasible any longer for us. Also, we notice problems
in side-by-side VS installations for certain project types. For these
reasons, we have decided to keep a single tool set going forward. If
you need multiple versions, then you would have to setup custom
agents. We are sorry that we cannot meet all of our customers'
requirements using a common hosted agent image.
So, to resolve this issue, we have to setup our private agent.
Hope this helps.
We have a ClickOnce application which we're trying to get running with CI/CD on Azure DevOps.
Currently the ClickOnce Prerequisites are set to 'Download prerequisites from the same location as my application' which is the behaviour we want ideally (this app is installed on a Customer's server and the client PCs download the .NET framework and various custom bootstrappers from that server):
When VS builds on my local development PC it picks up those bootstrapper files (which includes some custom bootstrappers we've written) from the local machine and outputs them to the ClickOnce Publishing folder:
However, this doesn't work on Azure DevOps I get this error about not being able to find the .NET bootstrapper instead:
[error]C:\Program Files (x86)\Microsoft Visual
Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(5390,5):
Error MSB3152: To enable 'Download prerequisites from the same
location as my application' in the Prerequisites dialog box, you must
download file 'DotNetFX462\NDP462-KB3151800-x86-x64-AllOS-ENU.exe' for
item 'Microsoft .NET Framework 4.6.2 (x86 and x64)' to your local
machine. For more information, see
http://go.microsoft.com/fwlink/?LinkId=616018.
As you'd expect, if I untick .NET 4.6.2 as I prerequisite I no longer get an error about .NET, but weirdly I don't then get an error for our custom bootstrappers, even though those don't exist on Azure either. The difference there seems to be that those custom bootstrappers don't currently exist on my local PC either, so they are shown with yellow exclamation triangles on my local machine.
So, is the .NET bootstrapper definitely not in the Azure build server image that gets spun up, or is it perhaps just a different path?
Or failing that, is there any way to tell DevOps to ignore this issue and just carry on building the actual ClickOnce application files and complete the build? The bootstrappers are already installed on our customer's server, so I don't actually need Azure to bundle them in the output.
Although not the best solution, the following worked for me:
I created an artifacts feed
I used Azure Artifacts with Universal packages to post my bootstrap packages
In the pipeline I used the "Universal packages" task to download the packages in "$ (System.DefaultWorkingDirectory)\bootstrapper"
I added the argument "/p:GenerateBootstrapperSdkPath=$(System.DefaultWorkingDirectory)\bootstrapper" to msbuild
The biggest drawback with this is the space used.
To publish the packages you will need:
Azure CLI
Azure DevOps extension for the Azure CLI: Run "az extension add --name azure-devops" when you have Azure CLI installed
Prepare the packages
In addition to your packages you need to upload the "Engine" directory which is where the setup.bin file is located. In my case I took it from "C:\Program Files (x86)\Microsoft SDKs\ClickOnce Bootstrapper"
I copied the "Engine" and "Schemas" directories to a temporary directory and then executed:
az login
az artifacts universal publish ^
--organization https://dev.azure.com/<MY-ORGANIZATION>/ ^
--project = "<MY PROJECT>" ^
--scope project ^
--feed <MY-FEED> ^
--name clickonce ^
--version 16.0.28315 ^
--description "ClickOnce Bootstrapper" ^
--path "D:\temp\ClickOnce Bootstrapper"
In the parameters "name", "version" and "description" you can put what you want, the important thing is to identify the package.
If the feed you created is associated with the organization, the command should not include the parameters "project" or "scope", you can see an example on how to publish by clicking on the "Connect to feed" button in your artifacts feed.
Then I ran the same command for each package I require, for example:
az artifacts universal publish ^
--organization https://dev.azure.com/my-organization/ ^
--project = "my project" ^
--scope project ^
--feed my-feed ^
--name dotnet ^
--version 4.6.2 ^
--description "Microsoft .Net Framework 4.6.2" ^
--path "C:\Program Files (x86)\Microsoft SDKs\ClickOnce Bootstrapper\Packages\DotNetFX462"
The pipeline
Before the compilation task add the download of the packages:
First download the engine, in the case of this example it would be the clickonce package 16.0.28315
- task: UniversalPackages#0
inputs:
command: 'download'
downloadDirectory: '$(System.DefaultWorkingDirectory)\bootstrapper'
feedsToUse: 'internal'
vstsFeed: '<MY FEED ID>'
vstsFeedPackage: '<MY PACKAGE ID>'
vstsPackageVersion: '16.0.28315'
Of course, with the wizard it is much easier, find the "Universal packages" task, select the "Download" command and fill in the rest of the parameters.
For your packages it is similar, just change the download path to "$(System.DefaultWorkingDirectory)\bootstrapper\Packages<PACKAGE NAME>"
The important thing at this point is that if you already have the packages published on a website, the "Package name" is the same name of the directory where the package is currently published, since the setup will try to download it from that path.
Lastly, my compilation task looks something like this:
- task: VSBuild#1
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
msbuildArgs: '/target:publish /p:GenerateBootstrapperSdkPath=$(System.DefaultWorkingDirectory)\bootstrapper'
Of course, this makes a lot more sense with your own packages. It works with the .Net Framework, but it will use some valuable MB. An alternative could be to create a script, in PowerShell for example, that downloads and installs the .Net Framework. Another option, if you have one available, is to copy the files from an FTP or fileserver, the latter applies to both the .Net Framework and its own packages, the only thing relevant is to redirect the Bootstrappers path with the parameter "GenerateBootstrapperSdkPath", about how packets get into that directory, you have several options.
We are building CD pipeline using VSTS hosted build servers. It takes more than 3 minutes to restore Nuget. This is too much time.
How can I make it run faster? Is there any sort of caching system we can use?
UPDATE: Caching is now generally available (docs)
Caching is currently on the feature pipeline with a TBD date. In the mean time you can use the Upload Pipeline Artifact/Download Pipeline Artifact tasks to store results in your Azure DevOps account to speed up up/downloads.
The Work-in-progress can be tracked here.
In the mean time, the Microsoft 1ES (one engineering system, internal organization) has released their internal solution that uses Universal Packages to store arbitrary packages in your Azure DevOps account. It's very fast because it can sync the delta between previous packages. There is a sample on how to configure your Azure Pipeline to store the NuGet package cache in your Sources Directory in order for the task to cache them.
variables:
NUGET_PACKAGES: $(Build.SourcesDirectory)/packages
keyfile: '**/*.csproj, **/packages.config, salt.txt'
vstsFeed: 'feed name'
steps:
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreCache#1
displayName: 'Restore artifact'
inputs:
keyfile: $(keyfile)
targetfolder: $(NUGET_PACKAGES)
vstsFeed: $(vstsFeed)
In my scenario, Nuget restore ran quickly when run interactively, but very slowly when run through CD pipeline (Jenkins). Setting revocation check mode to offline reduced my Nuget restore times from 13+ minutes to under 30 seconds (I found this solution here)
I set an environment variable in my build script prior to running Nuget restore:
SET NUGET_CERT_REVOCATION_MODE=offline
Disclaimer: Turning off certificate revocation has implications - see this link.