Here is my idea: build a static web page template, over time add .md files to a /posts directory and build a CI job to convert the .md files to html (with the showdownjs/showdown package).
Is there a need to install the package on every push? Seems like a waste, but uploading /node-modules is incorrect as well. Is there a way to install the package once, and let github action just work with it (run the md to html converter on newly added files)?
You have 2 options:
Recommened: Use caching action to cache dependencies for your project - using a hash from package-lock.json as a key to make sure it rebuild when depenendencies has changed:
- name: Cache node modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ hashFiles('**/package-lock.json') }}
- name: Install Dependencies
run: npm install
Push your node_modules to Git repository so it's checkout together with everything else
For optimising a need to run your converted you can use this action:
https://github.com/tj-actions/changed-files/
and detect if any files were modified at certain path
I've just started playing around with Azure DevOps as I want to migrate all my personal projects over to it as I really like using it at the office but I've never had to set it up and I'm having my first problem and hope someone can help.
I created a project in DevOps and I pushed my main solution under into its repo. All good. This solution has a dependency library from another project so I created another project in DevOps and pushed this library to its repo as well.
My main solution (.sln) which includes the project for my main project also includes a reference to library and compiling the solution locally works as expected but when I try this in DevOps, I get the following error:
D:\a\1\s\MyProject.sln.metaproj(0,0): Error MSB3202: The project file
"D:\a\1\s\..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj"
was not found.
and as a result of that I get a bunch of other errors similar to the below as it failed to compile the library that was referenced in the solution but is in a completely different project in DevOps:
MyProject\Controllers\MembersController.cs(8,7): Error CS0246: The type or namespace name
'MyLibraryAPI' could not be found (are you missing a using directive or an assembly
reference?)
Clearly can't find the project when in DevOps but how can I resolve this? I still want to make sure that this library gets compiled every time I compile my main project.
I've only read a little bit of YAML and I assume that I may have to change something in there but I'm not sure what to be honest. Anyway, I'll wait for someone's feedback and I hope the above makes sense and that someone can clarify how I can resolve this problem.
Thanks.
UPDATE-1
Thanks for the detailed feedback #MerlinLiang. I have followed the instruction from the link provided by #FrankAlvaro and yours and they are indeed the same but I appreciate the additional information as I did face this problem regarding the error:
The Git repository with name or identifier {repos name}does not exist
or you do not have permissions for the operation you are attempting
But I still haven't resolved the issue. I'm now getting the following error:
##[error]d:\a\1\s\MyProject\MyProject.sln.metaproj(0,0):
Error MSB3202: The project file "d:\a\1\s\MyProject\..\..\..\Libraries\
MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj" was not found.
Remember that MyProject is my main solution which has a reference to my MyLibrary but it's weird to see that it's trying to put MyLibrary\MyLibraryAPI inside the MyProject as locally they are in different location, so while I understand it would different in Azure DevOps, I would have thought that when checking out the project, it would have tried to respect the folder structure used locally, no?
I don't know if this will help solve my problem but this is how the projects are defined in my .sln file:
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyProject",
"MyProject\MyProject.csproj", "{CD02E7AD-97EE-4831-A18D-A57E2CC92E08}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyLibraryAPI",
"..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj",
"{82A7849D-22EF-4889-A7A6-2AE7D3F98F77}"
EndProject
UPDATE-2
I've tried what #MerlinLiang suggested by setting a path but I'm not sure I'm doing right as it still isn't working as expected. Here's what I've done:
Set a path for MyProject
Set a path for MyLibraryAPI
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
MyProjectCheckoutPath: 'Repos\Work\Company\MyProject'
MyLibraryApiCheckoutPath: 'Repos\Libraries\MyLibrary'
steps:
checkout: self
path: '$(MyProjectCheckoutPath)'
checkout: MyLibrary
path: '$(MyLibraryApiCheckoutPath)'
Note: MyLibrary is my solution folder which contains 2 folders, one of them called MyLibraryAPI which contains MyLibraryAPI.csproj while the other is just a console application used to test the library but isn't required. I'm not sure how to specifically specify to use the MyLibraryAPI project only as it is contained within a repo that contains 2 folders. I hope this makes sense.
When I specify the above path in YAML and I run the pipeline, I get the following warnings:
Repository is current at 'D:\a\1\s\My Project', move to
'D:\a\1\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Work\Company\MyProject'.
Repository is current at 'D:\a\1\s\MyLibrary', move to
'D:\a\1\Repos\Libraries\MyLibrary'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Libraries\MyLibrary'.
As you can see, both are giving a warning when I'm trying to use the paths defined in my variables.
As these are only warning, it continues with the build but eventually I get an error in the VSBuild section:
##[error]Solution not found using search pattern 'D:\a\1\s\**\*.sln'.
I've tried changing the solution variable in my YAML to MyProject
variables:
solution: '**/MyProject.sln'
But I still get an error as the path is clearly different:
##[error]Solution not found using search pattern
'D:\a\1\s\**\MyProject.sln'.
I'm not sure if I'm defining the path correctly as I'm getting warnings but assuming, it somehow did work, the solution path is still a problem.
Any ideas?
UPDATE-3
For a second, I thought I had it sorted as it took longer than usual for VSBuild to fail but to no avail. I changed the solution path to use the full path of MyProject to:
but ended up getting the following error:
"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\vswhere.exe"
-version [16.0,17.0) -latest -format json
"C:\Program Files (x86)\Microsoft Visual
Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe"
"D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln" /nologo /nr:false
/dl:CentralLogger,"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.
MSBuild.Logger.dll";"RootDetailId=7d46ac00-083a-434d-9729-
be159b80eea4|SolutionDir=D:\a\1\s\Repos
\Work\Company\MyProject"*ForwardingLogger,"D:\a\_tasks\
VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll"
/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true
/p:PackageLocation="D:\a\1\a" /p:platform="Any CPU"
/p:configuration="Release" /p:VisualStudioVersion="16.0"
/p:_MSDeployUserAgent="VSTS_aa0b7157-846e-49d9-8668-38efaf1e9745_build_5_0"
MSBUILD : error MSB1009: Project file does not exist.
Switch: D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln
##[error]Process 'msbuild.exe' exited with code '1'.
Finishing: VSBuild
UPDATE-4:
I've tried to create the directory via a Powerscript as suggested in
Warning when fetching additional repositories
but this didn't solve the problem. I've also added a script to list the content of the directory where it is supposed to have extracted the files from the repositories and both directories are showing as empty.
Here is my full YAML:
resources:
repositories:
- repository: MyLibrary
type: git
name: MyLibrary/MyLibrary
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: 'Repos\Work\Company\MyProject\MyProject.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
myLibraryCheckoutPath: 'Repos\Libraries\MyLibrary'
myProjectCheckoutPath: 'Repos\Work\Company\MyProject'
steps:
- task: PowerShell#2
displayName: "Create repository folder $(myProjectCheckoutPath)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myProjectCheckoutPath)" -ItemType "directory"
pwsh: true
- checkout: self
path: '$(myProjectCheckoutPath)'
- script: dir
workingDirectory: $(myProjectCheckoutPath)
displayName: List contents of a folder for MyProject
- task: PowerShell#2
displayName: "Create repository folder $(myLibraryCheckoutPath)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myLibraryCheckoutPath)" -ItemType "directory"
pwsh: true
- checkout: MyLibrary
path: '$(myLibraryCheckoutPath)'
- script: dir
workingDirectory: $(myLibraryCheckoutPath)
displayName: List contents of a folder for MyLibrary
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true
/p:SkipInvalidConfigurations=true
/p:PackageLocation="$(build.artifactStagingDirectory)"'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
# - task: VSTest#2
# inputs:
# platform: '$(buildPlatform)'
# configuration: '$(buildConfiguration)'
UPDATE 5
I'm still failing to understand what's going on.
When I have 2 projects being checked out, I get the following:
- Directory created: D:\a\1\s\Repos\Work\Company\MyProject
- Directory created: D:\a\1\s\Repos\Libraries\MyLibrary
And I get the following output in the logs:
- Repository is current at
'D:\a\1\s\MyProject', move to
'D:\a\1\s\Repos\Work\Company\MyProject'
(##[warning]Unable move and reuse existing repository to required location)
- Repository is current at
'D:\a\1\s\MyLibrary', move to
'D:\a\1\s\Repos\Libraries\MyLibrary'
(##[warning]Unable move and reuse existing repository to required location)
and as you can see if still fails to pull the relevant repos and move them to the relevant directory.
Yet, when I just pull a single repo i.e. MyProject, the following is outputted to the log:
Repository is current at
'D:\a\1\s', move to 'D:\a\1\s\Repos\Work\Company\MyProject'.
Repository will be located at
'D:\a\1\s\Repos\Work\Company\MyProject'.
And this works as expected.
In both scenarios, the root is 'D:\a\1\s' but when extracting one, the current directory is the same but when used with 2, it is the root directory and the project name. You would think this should be ok but it clearly isn't since it's failing to move the files.
One thing I don't get as well is according to Check out multiple repositories in your pipeline, there is a Note that states:
If you are using default paths, adding a second repository checkout
step changes the default path of the code for the first repository.
For example, the code for a repository named tools would be checked
out to C:\agent_work\1\s when tools is the only repository, but if a
second repository is added, tools would then be checked out to
C:\agent_work\1\s\tools. If you have any steps that depend on the
source code being in the original location, those steps must be
updated.
What is meant by
"If you have any steps that depend on the source code being in the original
location, those steps must be updated"
What steps and update them to what??
I've read so much documentation, I've been googling for hours and try so many YAML combinations but to no avail. I just don't seem to able to find a solution. Sorry if I'm missing something but I just don't understand why it works with one repo and doesn't work with 2.
I eventually figured out the missing piece!
While the Microsoft Azure DevOps documentation is excellent overall, I feel, it totally failed to explain the multi-checkout requirements, not to mention, a clear flaw in the system.
Note: My answer is based on the path requirements from my question.
In short, when checking out multiple projects, it simply doesn't create the source folders for each of the projects that are being checked out.
2 Issues:
1) Misleading warning message:
Repository is current at 'D:\a\1\s\MyProject', move to
'D:\a\1\s\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\s\Repos\Work\Company\MyProject'.
Why not display what the actual problem is i.e.
The folder 'D:\a\1\s\MyProject' not found.
and instead of creating a warning, it should create an error and stop the process.
2) Does not automatically create the source folder: Maybe there is an underlying reason behind this but I can't think of what it might be, but I personally think it should always create the source directory where the files are going to be checked out before they are moved.
Ideally, it should just create them directly in the specified path. I don't get either why you have to create it in one folder, only to move it to another one.
So to resolve your problem, simply make sure that you create the source folder for each of your checkouts and everything should work as expected.
My final script ended up looking like this:
resources:
repositories:
- repository: MyProject
type: git
name: My Project/My Project
- repository: MyLibrary
type: git
name: MyLibrary/MyLibrary
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
myProjectCheckoutSource: 'My Project'
myProjectCheckoutTarget: 's/Repos/Work/Company/MyProject'
myProjectCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Work\Company\MyProject'
myLibraryCheckoutSource: 'MyLibrary'
myLibraryCheckoutTarget: 's/Repos/Libraries/MyLibrary'
myLibraryCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Libraries\MyLibrary'
steps:
###################
# Pull My Project #
###################
# Create checkout source folder as when using multiple checkouts,
# it doesn't get created automatically.
- task: PowerShell#2
displayName: "Create repository folder: $(myProjectCheckoutSource)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myProjectCheckoutSource)" -ItemType "directory"
pwsh: true
# The target checkout folder defined in the 'path' is created automatically
- checkout: self
displayName: checking out $(myProjectCheckoutSource) to $(myProjectCheckoutTarget)
path: $(myProjectCheckoutTarget)
##################
# Pull MyLibrary #
##################
# Create checkout source folder as when using multiple checkouts,
# it doesn't get created automatically.
- task: PowerShell#2
displayName: "Create repository folder: $(myLibraryCheckoutSource)"
inputs:
targetType: inline
script: |
New-Item -Path . -Name "$(myLibraryCheckoutSource)" -ItemType "directory"
pwsh: true
# The target checkout folder defined in the 'path' is created automatically
- checkout: myLibrary
displayName: checking out $(myLibraryCheckoutSource) to $(myLibraryCheckoutTarget)
path: $(myLibraryCheckoutTarget)
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true
/p:PackageLocation="$(build.artifactStagingDirectory)"'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
Same suggestion with Frank.
In your scenario, you stored multi dependency libraries at multi projects. As default design, it only checkout current repo when you did not do any specific configuration in pipeline.
Take a simple example, project A has dependency on project B in its actual build. And, you store project A in Azure devops project A, project B in Azure devops project B. When you configure pipeline to build project A, our system will only checkout repo A into work space. In another word, there's no repo B exists in work space when A is finding B to execute build.
For why you can compile the solution locally works as expected, that's because all of projects are all exists in work space. This is different with Azure devops work logic.
Just follow doc to re-configure your YAML:
resources:
repositories:
- repository: {repos name A }
type: git
name: {project name A }/{repos name A }
- repository: {repos name B }
type: git
name: {project name B }/{repos name B }
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- checkout: git://{project name A }/{repos name A }
- checkout: git://{project name B }/{repos name B }
Checkout all of repos that your project needed.
We have done clear explanation and sample scripts in doc, you can follow that guide to finish YAML configuration.
Since you are very new to Azure devops, here I just share one issue you are most likely to encounter, which also the one you are mostly to be confused with:
remote: TF401019: The Git repository with name or identifier {repos
name}does not exist or you do not have permissions for the operation
you are attempting.
fatal: repository
'https://dev.azure.com/{org}/{project}/_git/{repos}/' not found
You may think it is indicating the incorrect YAML arguments configured. Actually not, this is a different prompt for permission issue: The service account that current pipeline used does not have access permission to target project.
Based on the target project cannot be accessed, the target repo under the target project does not exist for the pipeline, then the service account could know its existence.
To solve that issue, just go Project settings => Settings, and make sure Limit job authorization scope to current project is disabled:
Update on 3/31/2020:
Glad above additional notification has work for you.
I checked your updated contents, great help for me to figure problem! The error you faced is expected now.
Previously, you faced not found error is caused by there's no those files exists in build working directory. Now, they are all checkout into working directory. Though it still said not found, but it just represent the reference path is not correct.
You store those into different repos and projects, instead of one repo. At this time, the system could not know how's your project structure like, and it can only respect the checkout rule: install them into xxx/xxx/.../s folder.
it's weird to see that it's trying to put MyLibrary\MyLibraryAPI
inside the MyProject
You can check this note:
I'm afraid you should specify one argument into your YAML script: path. Re-configure the cache location to make it satisfied your actual project structure. Since I did not know how's your project structure like, you can do that by yourself firstly.
Update on 4/1/2020:
Still in this doc, we mentioned that If a path is specified for a checkout step, that path is used, relative to (Agent.BuildDirectory).
And for (Agent.BuildDirectory), its value is like /home/vsts/work/1. This means, the actual complete checkout path for your Myproject and Mylibrary is
xxx/1/Repos/Libraries/MyLibrary and xxx/1/Repos/Work/Company/MyProject. This is what you saw from the warning, because our original/default checkout path should be xxx/1/s folder.
This will not lead the error, just notify you the path changed here. The place that actually caused the error was in the vsbuild task.
One things you need pay attention about azure devops, the working directory of most of task is Build.SourcesDirectory, which actual path is xxx/1/s. In another word, it is build with finding sln/csproj from xxx/1/s.
Now, issues coming, you store them into xxx/1/repos/xxxx but the build task is looking for solution file from xxx/1/s/xxxx. Absolutely, it failed with nothing found. This also indicated in your build error message, please pay focus on the path.
Is there a way in Github to include md files in for example the README.md?
# Headline
Text
[include](File:load_another_md_file_here.md)
It should not link to the file, it should load the contents from it, like PHP include / file_get_contents.
That does not seem to be possible, especially when considering github/markup#346 and github/markup#172.
No include directive is supported.
This is not the correct answer but a workaround for others who really want this.
It's possible to use Gulp and Gulp Concat to merge the files into one before they are sent to Github..
Since it is not possible I just ended up placing a link as
[MY-LINK](../../SOME-OTHER-README.MD)
migrate your readme to a different file then construct your actual README however you like as a github action
EDIT: Here's a demo that you can build off of. This repo has a single github action that runs a script that dynamically builds the README.md based on the contents of the repository (to build a site map for the repo in the form of a table of contents): https://github.com/dmarx/bench-warmers
the workflow config:
name: update-readme
on:
push:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- uses: actions/setup-python#v2
- name: Run the script
run: python scripts/update_readme.py
- name: Commit files
run: |
git config --local user.name "dmarx"
git add README.md
git commit -m "Updated TOC"
- name: Push changes
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
force: true
Here's the chunk of my update script that's relevant to you:
... # code that builds the object `toc_str`
# template readme
with open('README.stub') as f:
readme_stub = f.read()
# simple replacement, use whatever stand-in value is useful for you.
readme = readme_stub.replace('{TOC}',toc_str)
with open('README.md','w') as f:
f.write(readme)
Which assumes you have a file named README.stub which might look something like this:
# Title
some text
{TOC}
more text
Where {TOC} is the substitution target for our dynamic content.
Ruby gem markdown_helper implements include files for GitHub flavored markdown (GFM).
Disclosure: I wrote the gem.
I have a small question that hopefully can get a quick answer. For Travis-CI repos set up for a C language project, are the variables defined in a ./configure script exported to travis build environment / subsystem? I just want to know if I can do the following with my .travis.yml:
deploy:
provider: releases
# ...
file:
- distribution-$VERSION.tar.gz
- distribution-$VERSION.zip
- distribution-windows-$VERSION.zip
Refs: http://docs.travis-ci.com/user/deployment/ and http://docs.travis-ci.com/user/deployment/releases/
I'm tring to deploy my JAR file to Github, but I don't want to create a tag every time.
I know it is not the correct behaviour, but I want to do this way if possible.
Building process is running fine, but when I use this travis.yml file:
language: java
deploy:
provider: releases
api-key: "<my_api_key>"
file: "teapot-1.2.5-beta.jar"
skip_cleanup: true
on:
branch: master
I got this error:
Installing deploy dependencies
Fetching: addressable-2.3.6.gem (100%)
Successfully installed addressable-2.3.6
Fetching: multipart-post-2.0.0.gem (100%)
Successfully installed multipart-post-2.0.0
Fetching: faraday-0.9.1.gem (100%)
Successfully installed faraday-0.9.1
Fetching: sawyer-0.6.0.gem (100%)
Successfully installed sawyer-0.6.0
Fetching: octokit-3.7.0.gem (100%)
Successfully installed octokit-3.7.0
5 gems installed
Fetching: mime-types-2.4.3.gem (100%)
Successfully installed mime-types-2.4.3
1 gem installed dpl.2
Preparing deploy
Logged in as Carlos Magno Oliveira de Abreu
Deploying to repo: icemagno/teapot
Current tag is: dpl.3
Deploying application
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-3.7.0/lib/octokit/response/raise_error.rb:16:in `on_complete': POST https://api.github.com/repos/icemagno/teapot/releases: 422 - Validation Failed (Octokit::UnprocessableEntity)
Error summary:
resource: Release
code: missing_field
field: tag_name
resource: Release
code: custom
field: tag_name
message: tag_name is not well-formed
resource: Release
code: custom
message: Published releases must have a valid tag // See: https://developer.github.com/v3/repos/releases/#create-a-release
...
failed to deploy
This is the basic deploy config:
language: java
deploy:
provider: releases
api-key: "<my_key_again>"
file: "teapot-1.2.5-beta.jar"
skip_cleanup: true
on:
tags: true
all_branches: true
But I have not created any tag because I don't want to do this for now.
EDIT
I've created a tag, now I need to change .travis.yml file to test some configurations and GitHub is not allowing me to change anything on tag files (ok, must be this way) cr#p !
If I may, I believe what you are looking for is the same as I was.
Basically add something like the following to your .travis.yml:
before_deploy:
- git config --global user.email "builds#travis-ci.com"
- git config --global user.name "Travis CI"
- export GIT_TAG=$TRAVIS_BRANCH-v0.1.$TRAVIS_BUILD_NUMBER
- git tag $GIT_TAG -a -m "Generated tag from TravisCI for build $TRAVIS_BUILD_NUMBER"
- git push -q https://<your-api-key>#github.com/<your_name>/<project_name> --tags
deploy:
skip_cleanup: true
provider: releases
api_key:
secure: [redacted]
file:
- "your_file"
on:
tags: false
all_branches: true
branches:
except:
- /^*-v[0-9]/
Careful doing this though, as messing up while working with these particular config options might lead to an infinite Travis-CI build loop, which will pollute your build and tag history, and not be easy to cleanup.