Azure DevOps 2019 - Adding Script leads to change of the building agent - unity3d

I have a local building agent for a Unity|HoloLens-Project. Today I had to switch from VS2019 to VS2022 on my building agent and I came across the issue, that the MSBuild-Task-Version of DevOps 2019 is not compatible with VS2022.
So I found a workaround. That workaround is to use the following script instead of the MSBuild task. But using that script I ran into the following misbehavior:
If I use the MSBuild#1-task my pipeline crashes because of mentioned compatibility problem. But the pipeline uses the right building agent.
If I use the Script the pipeline crashes because it uses the wrong building agent. That agent has no unity installed and is not mentioned in the YAML.
Yaml:
- task: MSBuild#1
displayName: 'Build Appx Package - MSBuild#1'
inputs:
solution: '$(Build.BinariesDirectory)$(unity.outputPath)/*.sln'
msbuildVersion: 'latest'
platform: 'ARM64'
configuration: 'Release'
msbuildArguments: '/p:AppxBundle=Always /p:AppxPackageDir=$(Build.ArtifactStagingDirectory)/AppPackages'
# ----- or ------
- script: |
#echo off
setlocal enabledelayedexpansion
for /f "usebackq tokens=*" %%i in (`"!ProgramFiles(x86)!\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe`) do (set msbuild_exe=%%i)
"!msbuild_exe!" "$(Build.BinariesDirectory)$(unity.outputPath)/*.sln" /p:Configuration=Release /p:Platform="ARM64" /p:AppxBundle=Always /p:AppxPackageDir=$(Build.ArtifactStagingDirectory)/AppPackages /t:rebuild
At the beginng of my YAML I set of course the pool and the name of the building agent:
- job: unity
pool:
name: 'My_Pool'
demand: Agent.name -equals My_Agent_Unity_1
I don't understand how removing the MSBuild task and adding the script can cause a different building agent to suddenly be taken.

I don't understand how removing the MSBuild task and adding the script
can cause a different building agent to suddenly be taken.
Because you entered the wrong keyword "demand" in your YAML. It should be demands.
- job: unity
pool:
name: 'My_Pool'
demands: Agent.name -equals My_Agent_Unity_1
Please see Manually entered demands and pool definition for details.

Related

Azure Pipeline - DevOps19 + VS22: Building Solution with two different projects

This post will be a bit longer, as I not only describe my problem, but also show my different attempts to solve the problem.
I have a solution contaning .Net-6-Web-Api-Project (csproj) and a C++/CLI-Wrapper-Project (vcxproj). I have a reference from the C#-Project to the c++-Project. I use DevOps 2019 and VS22 on my local building agent.
I'm not able to successfully run this solution through an Azure DevOps Pipeline using the task DotNetCoreCLI#2, VSBuild#1 or a custom script as a workaround for the MSBuild#1 to publish.
VSBuild
My initial approach was to simply use the VSBuild#1 task. Using this task does not allow the pipeline to start, with the following error:
##[Error 1]
No agent found in pool My_Pool which satisfies the specified demands:
agent.name -equals My_Agend_Unity_1
Cmd
msbuild
visualstudio
Agent.Version -gtVersion 2.153.1
The cause is the compatibility issue between DevOps 2019 and VS2022. The agent does not recognize VS2022 and therefore does not create system capabilities for it. Its the same issue for the MSBuild#1 and why I tried a custom script to work around, because it couldn't find MSBuild.
DotNetCoreCLI
The first error I got was:
error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
So I fixed that by adding the env variable to the task:
env:
PATH: 'C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170'
The resulting further error was:
##[error]Error: Unable to locate executable file: 'dotnet'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.
So I tried to fix it by using the Task UseDotNet#2, even though it doesn't make sense to me. But at the end I still get an error similar to the first error.:
MSBuild version 17.3.2+561848881 for .NET
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj : warning NU1503: Skipping restore for project "C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj". The project file may be invalid or missing targets required for restore. [C:\agent\_work\2\s\XXX\YYY\MySolution.sln]
Determining projects to restore...
"C:\agent\_work\2\s\XXX\YYY\DotNet6Project\MyProject.csproj" restored (in "2,4 sec").
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj(21,3):error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
C:\agent\_work\2\s\XXX\YYY\CPPWrapper\MyProject.vcxproj(21,3): error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the expression in the Import declaration "\Microsoft.Cpp.Default.props" is correct, and that the file exists on disk.
##[error]Error: The process 'C:\agent\_work\_tool\dotnet\dotnet.exe' failed with exit code 1
##[error]Dotnet command failed with non-zero exit code on the following projects : C:\agent\_work\2\s\XXX\YYY\MySolution.sln
##[section]Finishing: Build & Publish XXX Service - DotNetCoreCLI#2
MSBuild
My last hope then was my custom script that I already use in another pipeline that accesses the same agent and uses MSBuild from VS22. This is the approach I've come furthest with, as it looks like the project builds fine, but then fails because of this error.
(ResolvePackageAssets Target) -> C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5):
error NETSDK1064: Package "Microsoft.EntityFrameworkCore.Analyzers",
Version 6.0.4, not found. It may have been deleted after the NuGet restore.
Otherwise, the NuGet restore may have been only partially completed due to limitations on the maximum path length.
[C:\agent\_work\2\s\XXX\YYY\DotNet6Project\MyProject.csproj]
How to proceed with it, I do not know right now. I enabled already long paths via Group Policy Editor→Administrative templates→All Settings→Enable Win32 long paths.
My yaml file:
pool:
name: 'My_Pool'
demands:
- agent.name -equals My_Agent
variables:
buildPlatform: 'x64'
buildConfiguration: 'Release'
solution: '$(System.DefaultWorkingDirectory)/XXX/YYY/MySolution.sln'
DotNet6Project: '$(System.DefaultWorkingDirectory)/XXX/YYY/DotNet6Project/MyProject.csproj'
CPPWrapper: '$(System.DefaultWorkingDirectory)/XXX/YYY/CPPWrapper/MyProject.vcxproj'
steps:
- task: NuGetToolInstaller#0
displayName: 'NuGet Tool Installer - NuGetToolInstaller#0'
name: 'NuGetToolInstaller'
inputs:
versionSpec: '>=6.1.0'
- task: NuGetCommand#2
displayName: 'NuGet Restore - NuGetCommand#2'
inputs:
command: 'restore'
restoreSolution: '$(solution)'
noCache: true
- task: BatchScript#1
displayName: 'Run BatchScript to create DLLs, Libs & Header - BatchScript#1'
inputs:
filename: '$(System.DefaultWorkingDirectory)/ICP/ZZZ/build_release.bat'
env:
PATH: 'C:\Program Files\CMake\bin'
- task: PowerShell#2
displayName: 'Run Powershell Script to unpack Packages from BatchScript for ZZZWrapper - PowerShell#2'
inputs:
filePath: '$(System.DefaultWorkingDirectory)/XXX/YYY/CPPWrapper/install_ZZZ_package.ps1'
# Workaround for MSBuild#1
- script: |
#echo off
setlocal enabledelayedexpansion
for /f "usebackq tokens=*" %%i in (`"!ProgramFiles(x86)!\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe`) do (set msbuild_exe=%%i)
"!msbuild_exe!" "$(solution)" /p:Configuration="$(buildConfiguration)" /p:Platform="$(buildPlatform)" /p:PackageLocation="$(build.artifactStagingDirectory)" /t:rebuild
displayName: 'Build - Script'
# ---------- VSBuild ----------------
#- task: VSBuild#1
# inputs:
# solution: '$(solution)'
# msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
# platform: '$(buildPlatform)'
# configuration: '$(buildConfiguration)'
# ---------- DotNetCoreCLI ----------
#- task: UseDotNet#2
# inputs:
# packageType: 'sdk'
# version: '6.x'
#- task: DotNetCoreCLI#2
# displayName: 'Build & Publish - DotNetCoreCLI#2'
# inputs:
# command: 'publish'
# publishWebProjects: false
# projects: '$(solution)'
# arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
# zipAfterPublish: false
# env:
# PATH: 'C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170'
- task: PublishBuildArtifacts#1
displayName: 'Publish Build Artifacts - PublishBuildArtifacts#1'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'XXXArtifact'
publishLocation: 'Container'
From your yaml file ,I understand you want to build a solution and then publish build artifact. Accroding the error of your described task, I would like to provide suggestions that you can check.
1 VSBuild#1
taskError information: No agent found in pool My_Pool which satisfies the specified demandsThis indicates there are no agents on your machine that meet the demand requirements. You should check whether exists the agent which name is My_Agend_Unity_1 and exists check for Cmd,msbuild,visualstudio,Agent.Version -gtVersion 2.153.1.
See more information refer to doc:pool definition
2  UseDotNet#2
There’s a warning warning NU1503: Skipping restore for project that indicates  the packages required for the project MyProject are not restored correctly. You should edit the affected project to add targets for restore.
Please refer to doc:NuGet Warning NU1503
About the error MSB4019,you should check whether the project path "C:\Microsoft.Cpp.Default.props" exists. Here’s a ticekt  similar to your issue.You can try to this workaround and see if it works.
3 MSBuild
MSBuildAbout error NETSDK1064, this error occurs when the build tools can't find a NuGet package that's needed to build a project. This is typically due to a package restore issue related to warning NU1503 inTask   UseDotNet#2`. You can refer this doc:NETSDK1064: Package not found to take some actions provided to resolve this error.

How do I determined a User Environment Variable in YAML file for Microsoft Hosted Agent?

As per title above.
I am trying to mimic my automation setup on my local machine to match the automation i have running using Microsoft Hosted Agent machine.
This is how it set up in my local machine. therefore when the automation inserts "%repoic%" in a file explorer and the select folder button is pressed, it will open the desired folder
Now, I need to do the same using the Microsoft Hosted Agent to run my automation. Would anyone give me a clue on how this can be done? Is it simply do the following within my DevOps Pipeline's YAML file?
cheers!
From your requirement, you need to set the Environment variable and use it to navigate to path of the agent machine.
In Azure DevOps Pipeline, you can set the Pipeline variable as the screenshot show in the question.
Then you can use the format: $(variablename) to use the Pipeline variable.
Or you can use the format: %NAME% for batch and $env:NAME in PowerShell to use the Pipeline Environment Variable. Refer to this doc: Environment Variable
Here is an example:
pool:
vmImage: windows-latest
variables:
testpath: ./src/app/head
steps:
- powershell: Get-Location
workingDirectory: $(testpath)
Result:
OR you can use the environment variable in Pipeline.
Example:
pool:
vmImage: windows-latest
variables:
testpath: ./src/app/head
steps:
- task: CmdLine#2
inputs:
script: |
cd %testpath%

Is there a predefined variable for "/home/vsts" in Azure Pipelines?

I need to cache files in /home/vsts/.cache/torch/.
Is there a predefined variable for the home folder?
If by "Predefined variables" you mean one of the variables specified in the documentation then no.
However if you are using Hosted Linux Agents you can simply look at the environment variable $HOME, which works at least if you are in a script:, which may or may not fit your use case.
Example:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo $HOME
The output of the script task is /home/vsts
No, there is no such variable. You can use one of the /home/vsts/work paths and cache $(System.WorkFolder)/../.cache/torch/.
Values of predefined variables for an Ubuntu agent:
Agent.AcceptTeeEula True
Agent.BuildDirectory /home/vsts/work/1
Agent.HomeDirectory /home/vsts/agents/2.181.2
Agent.Id 32
Agent.JobName Job
Agent.JobStatus Succeeded
Agent.MachineName fv-az160-540
Agent.Name Azure Pipelines 2
Agent.OS Linux
Agent.OSArchitecture X64
Agent.RetainDefaultEncoding false
Agent.ReadOnlyVariables true
Agent.RootDirectory /home/vsts/work
Agent.TempDirectory /home/vsts/work/_temp
Agent.ToolsDirectory /opt/hostedtoolcache
Agent.Version 2.181.2
Agent.WorkFolder /home/vsts/work
Build.ArtifactStagingDirectory /home/vsts/work/1/a
Build.BinariesDirectory /home/vsts/work/1/b
Build.DefinitionName FooBar
Build.SourceBranch refs/heads/FOOBAR-123-caching
Build.SourceVersion 2b2c45223722fc7226eb23d752fc722bcdee2c54
Build.SourcesDirectory /home/vsts/work/1/s
Build.StagingDirectory /home/vsts/work/1/a
Common.TestResultsDirectory /home/vsts/work/1/TestResults
Pipeline.Workspace /home/vsts/work/1
System.AccessToken ***
System.ArtifactsDirectory /home/vsts/work/1/a
System.CollectionId c02d3c7b-9d74-4532-aa21-abccdf07c888
System.Culture en-US
System.DefaultWorkingDirectory /home/vsts/work/1/s
System.DefinitionId 26
System.EnableAccessToken SecretVariable
System.HostType build
System.JobAttempt 1
System.JobId 33f11733-54f2-5aa3-20dd-22fc7dcf5912
System.JobName __default
System.PhaseAttempt 1
System.PhaseDisplayName Job
System.PhaseName Job
System.ServerType Hosted
System.StageAttempt 1
System.StageName __default
System.TeamProject Foo Bar
System.TeamProjectId 3337a4f6-1411-40aa-beeb-0c7d86b9ecba
System.WorkFolder /home/vsts/work
Task.DisplayName print predefined variables

Azure CI and CD with NBGV

We're using NB.GV in our CI pipeline like:
- task: DotNetCoreCLI#2
inputs:
command: custom
custom: tool
arguments: install --tool-path . nbgv --ignore-failed-sources
displayName: Install NBGV tool
- script: nbgv cloud -c -a
displayName: Set Version
It sets the version correctly, so in further tasks, we're able to use them (e.g. $(GitBuildVersion))
The problem comes when we're trying to setup a CD pipeline based on this article. There we need to read $(Build.BuildNumber) which has a different value than expected. Based on official documentation it should be e.g. 1.3.1.57621 but we're getting 1.3.1+g15e1898f47.
It seems like AssemblyInformationalVersion and BuildVersion got exchanged.
We have set setVersionVariables: true in version.json.
Thanks for any help in advance
You my add this after running ngbv to update you BuildNumber
- powershell: Write-Host "##vso[build.updatebuildnumber]$(GitBuildVersion)"
displayName: 'Update build number to $(GitBuildVersion)'
According to this you should have then expected value:
You may also check what variables you have by running - bash: env | sort

Azure devops pipeline gives different results on two different self hosted agents of the same version

I have a virtual machine on Azure that I use to run a self hosted DevOps agent(version 2.170.1).
The pipeline consists of some PowerShell tasks that runs build and unit testing.
I use PowerShell core for this(version 7.0.2) and the server OS is Windows Server 2019 Datacenter 1809.
I have tried to run the tasks directly on the server from Powershell core(Version 7.0.2) and everything works as expected.
When it runs in a pipeline with the Powershell#2 task, warnings are reported as errors causing the task to fail even though it should not.
Furthermore other unexpected errors also occurs, that did not happen while running it directly on the server.
As an experiment I installed an agent of the same version on a laptop, and ran the pipeline on that instead. Again Powershell Core(Version 7.0.2) was used and the laptop OS is Windows 10 Enterprise 1909.
All compilers, test tools, and such are also the same version.
This time everything worked as expected while running the pipeline.
This seems to suggest that it is not the Powershell task or the Agent that is the problem.
However I have now arrived at a point where I dont really know where the problem could be located.
Here is an example of two of the tasks the pipeline is running.
- task: PowerShell#2
displayName: Build source code
inputs:
targetType: inline
pwsh: true
script: 'cd build; make all --jobs 4 --output-sync'
- task: PowerShell#2
displayName: Unit test
inputs:
targetType: inline
pwsh: true
script: 'cd test/unit; ceedling gcov:i2c_ha utils:gcov'
Does anyone have an idea of what the problem could be and what a possible solution could be?
UPDATE:
So I have done some further investigation and it seems like problem actually originates in the way errors are logged from the output.
When I run Ceedling with warnings disabled no errors or warnings are reported.
When warnings is enabled all warnings are reported as errors. So it seems like the output from Ceedling is analysed incorrectly.
A piece of the output from running Ceedling
In the link I have attached a piece of the output log of the task running Ceedling. As it can be seen Ceedling only reports warnings, but the logging tool in Azure DevOps actually logs it as an error.
This makes it hard to get an overview of warnings and the actual errors.
When it runs in a pipeline with the Powershell#2 task, warnings are
reported as errors causing the task to fail even though it should not.
1.Normally the PS task won't fail for warning, try adding script below:
ignoreLASTEXITCODE: true
pwsh: true
continueOnError: true
2.If #1 doesn't resolve your issue, you should check if you're picking right machine to run your pipeline. Assuming your agent pool has more than one agents, you can add demands to specifiy the agent.
pool:
name: Default
demands:
- Agent.Name -equals YourAgentName