In one step of a github actions job, I download an artifact that was uploaded in a previous job. I see these messages in the workflow output:
Artifact Executable_file was downloaded to D:\a\pcm_pcmplatinum_u\pcm_pcmplatinum_u\PCMateSetup\PCMPLUS.exe
Artifact download has finished successfully
In a subsequent step of the same job, I run msbuild on a *.wixproj file. Candle runs without error, but Light fails with this error:
error LGHT0103: The system cannot find the file 'D:\a\pcm_pcmplatinum_u\pcm_pcmplatinum_u\PCMateSetup\.\PCMPLUS.exe'.
Here are the relevant sections of the workflow .yml file:
- name: Download executable file
uses: actions/download-artifact#v3
with:
name: Executable_file
path: ./PCMateSetup/PCMPLUS.exe
- name: Build PCMateSetup
run: |
msbuild /m /p:MajorVer="${{env.majVer}}" /p:MinorVer="${{env.minVer}}" /p:BuildVer="${{env.bldVer}}" /p:ProtoVer="0" /p:GithubBld="true" /p:Configuration=${{env.BUILD_CONFIGURATION}} /v:n ${{env.SOLUTION_FILE_PATH}}\PCMateSetup\PCMateSetup.wixproj
From the .wxs files:
<?define PCMPlus.TargetDir="$(var.ProjectDir)." ?>
<File Id="file.instDir.PCMPLUSExecutable" Vital="yes" Source="$(var.PCMPlus.TargetDir)\PCMPLUS.exe" KeyPath="yes" />
And also from the output messages:
ProjectDir: D:\a\pcm_pcmplatinum_u\pcm_pcmplatinum_u\PCMateSetup\
I see no reason why Light should be failing this way. I did try changing the WiX code to eliminate the extra backslash and period in the path to the file, but Light failed with the same error. Does anyone have any idea why Light is failing to find the file? Thanks.
In one path it's PCMateSetup and in the other path it's PCMateSetup.
I have the following steps to copy files to artifact and publish artifact:
- name: copy FA function arm templates
run: Copy 'Service/Project/Hosts/FA/Infrastructure/' 'upload/functions_arm_templates/FA/Infrastructure'
shell: powershell
- name: copy FB function arm templates
run: Copy 'Service/Project/Hosts/FB/Infrastructure/' 'upload/functions_arm_templates/FB/Infrastructure'
shell: powershell
- name: publish artifact
uses: actions/upload-artifact#v2
with:
name: ${{github.run_number}}
path: upload/**
The Build succeeded however after downloading the artifact, I don't see a folder named functions_arm_templates. What am I missing?
You are only copying the folder without its contents.
Use copy 'Service/Project/Hosts/FA/Infrastructure/' 'upload/functions_arm_templates/FA/Infrastructure/' -recurse to copy the folder with its contents.
Does anyone know if it is possible to have a multi-stage Travis build that uses SBT (latest 1.4.3 version) and for each stage to reuse the compiled code from the previous one?
I tried with caching, but it does not seem to work (I hope I am just missing something).
What I have is a first “Compile” stage, and then a second “Tests” stage where two jobs are run in parallel (unit tests and integration tests).
What I want is that the Compile compiles everything (including test code) and then both the next stage just picks up where the compilation left.
I managed to organise the build to do that but:
either the compilation is re-done in each step or
by caching the whole ./ the compilation is saved…but then it ignores every new change I push (obviously).
The caching config I am using comes from the sbt docs:
cache:
directories:
- $HOME/.cache/coursier
- $HOME/.ivy2/cache
- $HOME/.sbt
Any idea?
(alternatively, if you know this is a problem of Travis and it can be made to work properly in Github Actions, that would be ok too)
There is a new Travis feature (in beta) that allows you to share files from one job with subsequent jobs in a build: workspaces
jobs:
include:
- stage: warm_cache
script:
- echo "foo" > foo.txt
workspaces:
create:
name: ws1
paths:
- foo.txt
- stage: use_cache
workspaces:
use: ws1
script:
- cat foo.txt || true
Also, SBT 1.4 allows you to push build artefacts to a Maven server and fetch then again later: https://www.scala-sbt.org/1.x/docs/Remote-Caching.html
I've just started playing around with Azure DevOps as I want to migrate all my personal projects over to it as I really like using it at the office but I've never had to set it up and I'm having my first problem and hope someone can help.
I created a project in DevOps and I pushed my main solution under into its repo. All good. This solution has a dependency library from another project so I created another project in DevOps and pushed this library to its repo as well.
My main solution (.sln) which includes the project for my main project also includes a reference to library and compiling the solution locally works as expected but when I try this in DevOps, I get the following error:
D:\a\1\s\MyProject.sln.metaproj(0,0): Error MSB3202: The project file
"D:\a\1\s\..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj"
was not found.
and as a result of that I get a bunch of other errors similar to the below as it failed to compile the library that was referenced in the solution but is in a completely different project in DevOps:
MyProject\Controllers\MembersController.cs(8,7): Error CS0246: The type or namespace name
'MyLibraryAPI' could not be found (are you missing a using directive or an assembly
reference?)
Clearly can't find the project when in DevOps but how can I resolve this? I still want to make sure that this library gets compiled every time I compile my main project.
I've only read a little bit of YAML and I assume that I may have to change something in there but I'm not sure what to be honest. Anyway, I'll wait for someone's feedback and I hope the above makes sense and that someone can clarify how I can resolve this problem.
Thanks.
UPDATE-1
Thanks for the detailed feedback #MerlinLiang. I have followed the instruction from the link provided by #FrankAlvaro and yours and they are indeed the same but I appreciate the additional information as I did face this problem regarding the error:
The Git repository with name or identifier {repos name}does not exist
or you do not have permissions for the operation you are attempting
But I still haven't resolved the issue. I'm now getting the following error:
##[error]d:\a\1\s\MyProject\MyProject.sln.metaproj(0,0):
Error MSB3202: The project file "d:\a\1\s\MyProject\..\..\..\Libraries\
MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj" was not found.
Remember that MyProject is my main solution which has a reference to my MyLibrary but it's weird to see that it's trying to put MyLibrary\MyLibraryAPI inside the MyProject as locally they are in different location, so while I understand it would different in Azure DevOps, I would have thought that when checking out the project, it would have tried to respect the folder structure used locally, no?
I don't know if this will help solve my problem but this is how the projects are defined in my .sln file:
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyProject",
"MyProject\MyProject.csproj", "{CD02E7AD-97EE-4831-A18D-A57E2CC92E08}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyLibraryAPI",
"..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj",
"{82A7849D-22EF-4889-A7A6-2AE7D3F98F77}"
EndProject
UPDATE-2
I've tried what #MerlinLiang suggested by setting a path but I'm not sure I'm doing right as it still isn't working as expected. Here's what I've done:
Set a path for MyProject
Set a path for MyLibraryAPI
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
MyProjectCheckoutPath: 'Repos\Work\Company\MyProject'
MyLibraryApiCheckoutPath: 'Repos\Libraries\MyLibrary'
steps:
checkout: self
path: '$(MyProjectCheckoutPath)'
checkout: MyLibrary
path: '$(MyLibraryApiCheckoutPath)'
Note: MyLibrary is my solution folder which contains 2 folders, one of them called MyLibraryAPI which contains MyLibraryAPI.csproj while the other is just a console application used to test the library but isn't required. I'm not sure how to specifically specify to use the MyLibraryAPI project only as it is contained within a repo that contains 2 folders. I hope this makes sense.
When I specify the above path in YAML and I run the pipeline, I get the following warnings:
Repository is current at 'D:\a\1\s\My Project', move to
'D:\a\1\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Work\Company\MyProject'.
Repository is current at 'D:\a\1\s\MyLibrary', move to
'D:\a\1\Repos\Libraries\MyLibrary'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Libraries\MyLibrary'.
As you can see, both are giving a warning when I'm trying to use the paths defined in my variables.
As these are only warning, it continues with the build but eventually I get an error in the VSBuild section:
##[error]Solution not found using search pattern 'D:\a\1\s\**\*.sln'.
I've tried changing the solution variable in my YAML to MyProject
variables:
solution: '**/MyProject.sln'
But I still get an error as the path is clearly different:
##[error]Solution not found using search pattern
'D:\a\1\s\**\MyProject.sln'.
I'm not sure if I'm defining the path correctly as I'm getting warnings but assuming, it somehow did work, the solution path is still a problem.
Any ideas?
UPDATE-3
For a second, I thought I had it sorted as it took longer than usual for VSBuild to fail but to no avail. I changed the solution path to use the full path of MyProject to:
but ended up getting the following error:
"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\vswhere.exe"
-version [16.0,17.0) -latest -format json
"C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe"
"D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln" /nologo /nr:false
/dl:CentralLogger,"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.
MSBuild.Logger.dll";"RootDetailId=7d46ac00-083a-434d-9729-
be159b80eea4|SolutionDir=D:\a\1\s\Repos
\Work\Company\MyProject"*ForwardingLogger,"D:\a\_tasks\
VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll"
/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true
/p:PackageLocation="D:\a\1\a" /p:platform="Any CPU"
/p:configuration="Release" /p:VisualStudioVersion="16.0"
/p:_MSDeployUserAgent="VSTS_aa0b7157-846e-49d9-8668-38efaf1e9745_build_5_0"
MSBUILD : error MSB1009: Project file does not exist.
Switch: D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln
##[error]Process 'msbuild.exe' exited with code '1'.
Finishing: VSBuild
UPDATE-4:
I've tried to create the directory via a Powerscript as suggested in
Warning when fetching additional repositories
but this didn't solve the problem. I've also added a script to list the content of the directory where it is supposed to have extracted the files from the repositories and both directories are showing as empty.
Here is my full YAML:
resources:
repositories:
- repository: MyLibrary
type: git
name: MyLibrary/MyLibrary
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: 'Repos\Work\Company\MyProject\MyProject.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
myLibraryCheckoutPath: 'Repos\Libraries\MyLibrary'
myProjectCheckoutPath: 'Repos\Work\Company\MyProject'
steps:
- task: PowerShell#2
displayName: "Create repository folder $(myProjectCheckoutPath)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myProjectCheckoutPath)" -ItemType "directory"
pwsh: true
- checkout: self
path: '$(myProjectCheckoutPath)'
- script: dir
workingDirectory: $(myProjectCheckoutPath)
displayName: List contents of a folder for MyProject
- task: PowerShell#2
displayName: "Create repository folder $(myLibraryCheckoutPath)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myLibraryCheckoutPath)" -ItemType "directory"
pwsh: true
- checkout: MyLibrary
path: '$(myLibraryCheckoutPath)'
- script: dir
workingDirectory: $(myLibraryCheckoutPath)
displayName: List contents of a folder for MyLibrary
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true
/p:SkipInvalidConfigurations=true
/p:PackageLocation="$(build.artifactStagingDirectory)"'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
# - task: VSTest#2
# inputs:
# platform: '$(buildPlatform)'
# configuration: '$(buildConfiguration)'
UPDATE 5
I'm still failing to understand what's going on.
When I have 2 projects being checked out, I get the following:
- Directory created: D:\a\1\s\Repos\Work\Company\MyProject
- Directory created: D:\a\1\s\Repos\Libraries\MyLibrary
And I get the following output in the logs:
- Repository is current at
'D:\a\1\s\MyProject', move to
'D:\a\1\s\Repos\Work\Company\MyProject'
(##[warning]Unable move and reuse existing repository to required location)
- Repository is current at
'D:\a\1\s\MyLibrary', move to
'D:\a\1\s\Repos\Libraries\MyLibrary'
(##[warning]Unable move and reuse existing repository to required location)
and as you can see if still fails to pull the relevant repos and move them to the relevant directory.
Yet, when I just pull a single repo i.e. MyProject, the following is outputted to the log:
Repository is current at
'D:\a\1\s', move to 'D:\a\1\s\Repos\Work\Company\MyProject'.
Repository will be located at
'D:\a\1\s\Repos\Work\Company\MyProject'.
And this works as expected.
In both scenarios, the root is 'D:\a\1\s' but when extracting one, the current directory is the same but when used with 2, it is the root directory and the project name. You would think this should be ok but it clearly isn't since it's failing to move the files.
One thing I don't get as well is according to Check out multiple repositories in your pipeline, there is a Note that states:
If you are using default paths, adding a second repository checkout
step changes the default path of the code for the first repository.
For example, the code for a repository named tools would be checked
out to C:\agent_work\1\s when tools is the only repository, but if a
second repository is added, tools would then be checked out to
C:\agent_work\1\s\tools. If you have any steps that depend on the
source code being in the original location, those steps must be
updated.
What is meant by
"If you have any steps that depend on the source code being in the original
location, those steps must be updated"
What steps and update them to what??
I've read so much documentation, I've been googling for hours and try so many YAML combinations but to no avail. I just don't seem to able to find a solution. Sorry if I'm missing something but I just don't understand why it works with one repo and doesn't work with 2.
I eventually figured out the missing piece!
While the Microsoft Azure DevOps documentation is excellent overall, I feel, it totally failed to explain the multi-checkout requirements, not to mention, a clear flaw in the system.
Note: My answer is based on the path requirements from my question.
In short, when checking out multiple projects, it simply doesn't create the source folders for each of the projects that are being checked out.
2 Issues:
1) Misleading warning message:
Repository is current at 'D:\a\1\s\MyProject', move to
'D:\a\1\s\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\s\Repos\Work\Company\MyProject'.
Why not display what the actual problem is i.e.
The folder 'D:\a\1\s\MyProject' not found.
and instead of creating a warning, it should create an error and stop the process.
2) Does not automatically create the source folder: Maybe there is an underlying reason behind this but I can't think of what it might be, but I personally think it should always create the source directory where the files are going to be checked out before they are moved.
Ideally, it should just create them directly in the specified path. I don't get either why you have to create it in one folder, only to move it to another one.
So to resolve your problem, simply make sure that you create the source folder for each of your checkouts and everything should work as expected.
My final script ended up looking like this:
resources:
repositories:
- repository: MyProject
type: git
name: My Project/My Project
- repository: MyLibrary
type: git
name: MyLibrary/MyLibrary
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
myProjectCheckoutSource: 'My Project'
myProjectCheckoutTarget: 's/Repos/Work/Company/MyProject'
myProjectCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Work\Company\MyProject'
myLibraryCheckoutSource: 'MyLibrary'
myLibraryCheckoutTarget: 's/Repos/Libraries/MyLibrary'
myLibraryCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Libraries\MyLibrary'
steps:
###################
# Pull My Project #
###################
# Create checkout source folder as when using multiple checkouts,
# it doesn't get created automatically.
- task: PowerShell#2
displayName: "Create repository folder: $(myProjectCheckoutSource)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myProjectCheckoutSource)" -ItemType "directory"
pwsh: true
# The target checkout folder defined in the 'path' is created automatically
- checkout: self
displayName: checking out $(myProjectCheckoutSource) to $(myProjectCheckoutTarget)
path: $(myProjectCheckoutTarget)
##################
# Pull MyLibrary #
##################
# Create checkout source folder as when using multiple checkouts,
# it doesn't get created automatically.
- task: PowerShell#2
displayName: "Create repository folder: $(myLibraryCheckoutSource)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myLibraryCheckoutSource)" -ItemType "directory"
pwsh: true
# The target checkout folder defined in the 'path' is created automatically
- checkout: myLibrary
displayName: checking out $(myLibraryCheckoutSource) to $(myLibraryCheckoutTarget)
path: $(myLibraryCheckoutTarget)
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true
/p:PackageLocation="$(build.artifactStagingDirectory)"'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
Same suggestion with Frank.
In your scenario, you stored multi dependency libraries at multi projects. As default design, it only checkout current repo when you did not do any specific configuration in pipeline.
Take a simple example, project A has dependency on project B in its actual build. And, you store project A in Azure devops project A, project B in Azure devops project B. When you configure pipeline to build project A, our system will only checkout repo A into work space. In another word, there's no repo B exists in work space when A is finding B to execute build.
For why you can compile the solution locally works as expected, that's because all of projects are all exists in work space. This is different with Azure devops work logic.
Just follow doc to re-configure your YAML:
resources:
repositories:
- repository: {repos name A }
type: git
name: {project name A }/{repos name A }
- repository: {repos name B }
type: git
name: {project name B }/{repos name B }
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- checkout: git://{project name A }/{repos name A }
- checkout: git://{project name B }/{repos name B }
Checkout all of repos that your project needed.
We have done clear explanation and sample scripts in doc, you can follow that guide to finish YAML configuration.
Since you are very new to Azure devops, here I just share one issue you are most likely to encounter, which also the one you are mostly to be confused with:
remote: TF401019: The Git repository with name or identifier {repos
name}does not exist or you do not have permissions for the operation
you are attempting.
fatal: repository
'https://dev.azure.com/{org}/{project}/_git/{repos}/' not found
You may think it is indicating the incorrect YAML arguments configured. Actually not, this is a different prompt for permission issue: The service account that current pipeline used does not have access permission to target project.
Based on the target project cannot be accessed, the target repo under the target project does not exist for the pipeline, then the service account could know its existence.
To solve that issue, just go Project settings => Settings, and make sure Limit job authorization scope to current project is disabled:
Update on 3/31/2020:
Glad above additional notification has work for you.
I checked your updated contents, great help for me to figure problem! The error you faced is expected now.
Previously, you faced not found error is caused by there's no those files exists in build working directory. Now, they are all checkout into working directory. Though it still said not found, but it just represent the reference path is not correct.
You store those into different repos and projects, instead of one repo. At this time, the system could not know how's your project structure like, and it can only respect the checkout rule: install them into xxx/xxx/.../s folder.
it's weird to see that it's trying to put MyLibrary\MyLibraryAPI
inside the MyProject
You can check this note:
I'm afraid you should specify one argument into your YAML script: path. Re-configure the cache location to make it satisfied your actual project structure. Since I did not know how's your project structure like, you can do that by yourself firstly.
Update on 4/1/2020:
Still in this doc, we mentioned that If a path is specified for a checkout step, that path is used, relative to (Agent.BuildDirectory).
And for (Agent.BuildDirectory), its value is like /home/vsts/work/1. This means, the actual complete checkout path for your Myproject and Mylibrary is
xxx/1/Repos/Libraries/MyLibrary and xxx/1/Repos/Work/Company/MyProject. This is what you saw from the warning, because our original/default checkout path should be xxx/1/s folder.
This will not lead the error, just notify you the path changed here. The place that actually caused the error was in the vsbuild task.
One things you need pay attention about azure devops, the working directory of most of task is Build.SourcesDirectory, which actual path is xxx/1/s. In another word, it is build with finding sln/csproj from xxx/1/s.
Now, issues coming, you store them into xxx/1/repos/xxxx but the build task is looking for solution file from xxx/1/s/xxxx. Absolutely, it failed with nothing found. This also indicated in your build error message, please pay focus on the path.
I'm trying to release new version on a test projet but my artifact is not recognised.
Discovering tests...OK
Deploying using GitHub provider
Creating "a2" release for repository "Phonations/TravisTest" tag "a2" commit "44d044aa7708508f016dfef67fbdd82f08642f8d"...OK
No artifacts were published. Make sure you have specified correct artifacts filter.
Build success
(full output available here)
I checked the artifact artifact.txt is present as you can see when performing the dir command in the build output.
It is specified here
You should setup artifacts packaging in appveyor.yml. For that file in your project this would be:
artifacts:
- path: artifact.txt